Operator calculation method, device and medium

Abstract

The invention provides an operator calculation method and device and a medium, and the method comprises the steps: segmenting a calculation task of an operator based on the output layout of the operator, and obtaining a plurality of subtasks; based on similarities and differences between input data arrangement and output data arrangement required by operator output, determining a loading mode of the input data of the operator, and loading the input data based on the loading mode; running the plurality of sub-tasks based on the loaded input data to obtain running results of the plurality of sub-tasks; and merging the operation results of the plurality of sub-tasks, and taking the merged operation results as output data of the operator. According to the method, the device and the medium, adaptation of operator calculation and layout design is guaranteed, operator calculation oriented to various layouts can be achieved only through one set of execution logic, the situation that kernel function codes are incrementally developed to adapt to different layouts is avoided, the workload of code development is greatly reduced, and the development efficiency is improved. And the function support rate of the operator is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an operator calculation method, device and medium. Background Art

[0002] As the underlying arrangement of data, layout design can be used to accelerate the data read and write bandwidth of heterogeneous operators. In order to match the continuous improvement of computing performance requirements in the field of deep learning, more and more new layouts are being designed and applied.

[0003] However, different layout designs will complicate operator implementation. A kernel function used to implement a specific operator usually only supports one layout. In order to apply more layouts to operator implementation, it is usually necessary to incrementally develop corresponding kernel functions for different layouts, which makes the cost of implementing different layouts on operators high. Summary of the invention

[0004] The present invention provides an operator calculation method, device and medium, which are used to solve the defect in the related art that the implementation cost of the operator remains high due to adaptation to different layouts.

[0005] The present invention provides an operator calculation method, comprising: Based on the output layout of the operator, the computing task of the operator is divided into multiple subtasks; Determine a loading method for input data of the operator based on similarities and differences between input data arrangement required for output of the operator and output data arrangement, and load the input data based on the loading method; Based on the loaded input data, the plurality of subtasks are run to obtain the running results of the plurality of subtasks; The running results of the multiple subtasks are merged, and the merged result is used as the output data of the operator.

[0006] According to an operator calculation method provided by the present invention, the method of determining a loading method for input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0007] According to an operator calculation method provided by the present invention, the method of determining a loading method for input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0008] An operator calculation method provided by the present invention further includes: If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, then after the input data is loaded, the input data is rearranged.

[0009] According to an operator calculation method provided by the present invention, the read and write granularity of the first loading mode is a thread warp, and the read and write granularity of the second loading mode is a single thread.

[0010] According to an operator calculation method provided by the present invention, the dividing of the calculation tasks of the operator based on the output layout of the operator includes: Determining a vectorization granularity corresponding to the operator based on a calculation mode of the operator; The computing tasks of the operator are divided based on the output layout of the operator and the vectorization granularity corresponding to the operator.

[0011] According to an operator calculation method provided by the present invention, the calculation task of the operator is divided based on the output layout of the operator and the vectorization granularity corresponding to the operator, including: The computing tasks of the operator are divided based on a dimensional division order corresponding to the output layout and a division granularity corresponding to the vectorization granularity.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the operator calculation methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program implements any of the operator calculation methods described above.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned operator calculation methods.

[0015] The operator calculation method, device and medium provided by the present invention divide the calculation task based on the output layout of the operator to realize the parallel operation of multiple subtasks; in addition, based on the similarities and differences between the input data arrangement and the output data arrangement of the operator, the input data loading method is determined, thereby realizing the loading of the input data and the operator calculation based on the loaded data, ensuring the adaptation of the operator calculation and the layout design, only needing one set of execution logic to realize the operator calculation for various layouts, avoiding the situation of incrementally developing kernel function code to adapt to different layouts, greatly reducing the workload of code development, and improving the function support rate of the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is one of the flow charts of the operator calculation method provided by the present invention; Figure 2 This is the second flow chart of the operator calculation method provided by the present invention; Figure 3 It is a structural schematic diagram of an operator computing device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the field of deep learning, data usually exists in the form of multidimensional arrays (such as tensors). The arrangement of this data (i.e., layout) has an important impact on the implementation and performance of operators. Layout is the arrangement of data in memory, including the order in which data is arranged in memory and the mapping relationship between data and memory addresses. As the requirements for computing performance in the field of deep learning continue to increase, more and more layouts are being designed and applied to achieve higher read and write bandwidth.

[0020] As a function that performs specific mathematical or logical operations, the design of an operator is closely related to the layout of the data. Specifically, different layouts may require different algorithm implementations to optimize their performance. Therefore, a large number of layout designs will complicate the implementation of operators.

[0021] For the above situation, the main solution at present is to enter different operators through layout judgment and incrementally develop operators required for different layouts.

[0022] Entering different operators through layout determination means dynamically selecting the corresponding operator according to the layout of the data. This means that for each operator, a supported version needs to be provided for different layouts.

[0023] Specifically, operators are usually implemented through functions executed on the underlying hardware, namely kernel functions. Since kernel functions correspond to specific hardware architectures and data layouts, a kernel function usually only supports one layout. If operators need to be executed on different layouts, it is necessary to write corresponding kernel functions for each layout, that is, incrementally develop operators required for different layouts.

[0024] However, incrementally developing operators required for different layouts will introduce new problems.

[0025] First, a corresponding kernel function needs to be written for each layout, which makes the implementation cost of operators adapted to various layouts extremely high. In addition, the large-scale kernel function code leads to poor maintainability. If there is a new layout, a new kernel function still needs to be written, which makes the above method not scalable.

[0026] Second, in actual applications, there may be data broadcasting and different data shapes. Data broadcasting is a mechanism that enables tensors of different shapes to automatically adjust their shapes to match when performing element-by-element operations. This mechanism simplifies the process of operating on tensors of different shapes but with a certain degree of compatibility without explicitly expanding the dimensions of the tensor or copying the data. However, data broadcasting and different data shapes mean that the operator needs to support multiple layout combinations. That is, a corresponding kernel function needs to be written for each possible combination, which directly leads to an explosive growth in the amount of code.

[0027] Third, the performance of different layouts varies, and the performance of the kernel functions corresponding to different layouts also varies. This makes the performance differences between different operators large, and the layouts that different operators can support are also different, and the generalization support of operators is uneven.

[0028] In view of the above problems, an embodiment of the present invention provides an operator calculation method. Figure 1 is one of the flow charts of the operator calculation method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110 , based on the output layout of the operator, divide the computing task of the operator into multiple subtasks.

[0029] The operator here is an operator used for computing operations. In the embodiment of the present invention, the operator can support various layouts.

[0030] The output layout of an operator refers to the layout of the output data of the operator. After obtaining the output layout of the operator, the computing task of the operator can be divided based on the output layout, thereby obtaining multiple subtasks corresponding to the output layout of the operator.

[0031] Here, splitting the computing task of the operator refers to splitting the overall computing task into multiple smaller subtasks so that each subtask can be executed in parallel on different computing devices or different threads of the same computing device, so as to maximize the utilization of computing resources and reduce the time required for operator calculation.

[0032] It is understandable that the computing tasks of the operator are divided based on the output layout. Specifically, different task division methods can be set for different output layouts in advance, so that the task division method corresponding to the output layout can be directly applied to achieve the division of the computing tasks of the operator. For example, according to the arrangement order of each element in the output layout, a dimensional division method that matches it can be set to achieve the division of the computing tasks. In some embodiments, the division of computing tasks can be designed with the goals of task balancing of thread bundles, load balancing of multiple computing units, maximizing the merging of reads and writes, and maximizing the parallelism of reads, writes and calculations. The embodiments of the present invention do not limit the specific division method.

[0033] Step 120: Determine a method for loading input data for the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement, and load the input data based on the loading method.

[0034] Specifically, data arrangement is the arrangement of data, which can be understood as a combination of data layout and data shape. Data shape is used to describe the size of data in each dimension. Data layout determines the arrangement of data in memory, and data shape determines the shape structure of data. The data arrangement composed of the two determines the storage structure and access method of data.

[0035] In an embodiment of the present invention, input data arrangement, i.e., data arrangement of input data, may include the layout of input data of an operator and the shape of input data, and it should be noted that the input data arrangement here is specifically the data arrangement of input data required to obtain output data of an operator; output data arrangement, i.e., data arrangement of output data, may include the layout of output data of an operator and the shape of output data.

[0036] Before performing calculations on an operator, the similarities and differences between the input data arrangement and the output data arrangement of the operator can be compared, that is, to determine whether the data arrangement of the input data and the output data is consistent. The similarities and differences here can be expressed as the same or different, or can also be expressed as the same, similar, and dissimilar, which is not specifically limited in the embodiments of the present invention.

[0037] The similarity of the data arrangement of the input data and the output data may refer to that the input data arrangement and the output data arrangement are partially the same, and the input data can be stored continuously. The dissimilarity of the input data arrangement and the output data arrangement may include the case where the input data arrangement and the output data arrangement are partially the same, but the input data is not stored continuously, and the case where the input data arrangement and the output data arrangement are completely different.

[0038] After determining the similarities and differences between the input data arrangement and the output data arrangement, the loading mode corresponding to the similarities and differences between the input data arrangement and the output data arrangement can be determined. That is, corresponding loading modes can be set in advance for various possible similarities and differences, thereby determining the corresponding loading mode according to the actual similarities and differences.

[0039] The loading mode here refers to the mode of loading the input data of the operator. Different loading modes can be manifested as different read and write granularities when loading data, for example, thread bundles can be used as the read and write granularity, or a single thread can be used as the read and write granularity, which is not specifically limited in the embodiment of the present invention.

[0040] It is understandable that when the input data arrangement and the output data arrangement are the same, there is no need to perform data misalignment arrangement when loading the input data. In addition, when the input data arrangement and the output data arrangement are similar, there are fewer cases where data misalignment arrangement is required when loading the input data, and the continuity of input data loading is stronger. At this time, a loading method with a larger read / write granularity can be selected to improve the loading efficiency of the input data. When the input data arrangement and the output data arrangement are different or dissimilar, there are more cases where data error arrangement is required when loading the input data, and the continuity of input data loading is poor. At this time, a loading method with a smaller read / write granularity can be selected to improve the flexibility of input data loading and adapt to the data arrangement of the output data.

[0041] After the loading method is determined, the input data required for operator calculation can be loaded based on the loading method that can adapt to the layout and shape of the input data, thereby ensuring the rationality and reliability of the input data loading.

[0042] Step 130: Run the multiple subtasks based on the loaded input data to obtain the running results of the multiple subtasks.

[0043] Step 140: merge the running results of the multiple subtasks and use the merged results as the output data of the operator.

[0044] Specifically, after the input data loading is completed, the loaded input data can be used as the input of the subtask running, so that multiple subtasks are run in parallel to obtain the running results of the multiple subtasks.

[0045] After obtaining the running results of each subtask, the running results of all subtasks can be merged. The merged result here is the calculation result of the operator, that is, the output data of the operator. It can be understood that since each subtask is divided based on the output layout of the operator, after obtaining the running results of each subtask, the running results of each subtask can be merged into the output data that conforms to the output layout of the operator. For example, after completing the merging of the running results, the merged results can be written back to the output data of the operator.

[0046] Furthermore, the calculation results of the operators obtained by merging here can be expressed as a larger read-write granularity, for example, the read-write granularity can be a thread warp. Writing the calculation results obtained by merging back to a continuous storage of the output data of the operator with the thread warp as the read-write granularity can effectively improve the efficiency of writing back the output data.

[0047] In the method provided in the embodiment of the present invention, the computing task is divided based on the output layout of the operator to realize the parallel operation of multiple subtasks; in addition, based on the similarities and differences between the input data arrangement and the output data arrangement of the operator, the input data loading method is determined, thereby realizing the loading of the input data and the operator calculation based on the loaded data, ensuring the adaptation of the operator calculation and the layout design, only one set of execution logic is needed to realize the operator calculation for various layouts, avoiding the situation of incrementally developing kernel function code to adapt to different layouts, greatly reducing the workload of code development, and improving the function support rate of the operator.

[0048] In addition, in the method provided in the embodiment of the present invention, the parameterization of the layout is also implemented, which is specifically reflected in that parameters such as operator task segmentation and input data loading can be implemented for different layouts, so that operator calculations can adapt to various layouts.

[0049] In addition, in terms of the function support rate of operators, especially for operators supporting data broadcasting, the function support rate is greatly improved by switching from an incremental development solution to the generalized universal prevention provided by the embodiment of the present invention, and the function coverage rate can reach 100%.

[0050] Based on the above embodiment, in step 120, determining the loading method for the input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0051] Specifically, when determining the differences between the input data arrangement and the output data arrangement, two situations can be classified, namely, the input data arrangement and the output data arrangement are the same, or the input data arrangement and the output data arrangement are different.

[0052] Among them, when the input data arrangement and the output data arrangement are the same, there is no need to arrange the data in a staggered manner when loading the input data, and the input data can be loaded continuously. At this time, a loading method with a larger read / write granularity can be selected, which is recorded as the first loading method here. By loading the input data through the first loading method with a larger read / write granularity, it is possible to achieve merged loading of finer-grained data, thereby improving the loading efficiency of the input data.

[0053] When the input data arrangement is different from the output data arrangement, the input data needs to be staggered when loading, so as to adapt to the output data arrangement. At this time, the continuity of loading the input data is poor, and a loading method with a smaller read / write granularity can be selected, which is recorded as the second loading method. Loading the input data through the second loading method with a smaller read / write granularity can effectively improve the flexibility of loading the input data.

[0054] Therefore, two types of loading methods with different read and write granularities can be pre-set, namely, a first loading method with larger read and write granularity and a second loading method with smaller read and write granularity, and the first loading method is applied to load input data when it is determined that the input data arrangement and the output data arrangement are the same, and the second loading method is applied to load input data when it is determined that the input data arrangement and the output data arrangement are different, thereby being compatible with operator implementations under different layout designs.

[0055] Based on any of the above embodiments, in step 120, determining a loading method for input data of the operator based on similarities and differences between input data arrangement required for output of the operator and output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0056] Specifically, when judging the similarities and differences between the input data arrangement and the output data arrangement, it can be divided into three situations, namely, the input data arrangement and the output data arrangement are the same, the input data arrangement and the output data arrangement are similar, or the input data arrangement and the output data arrangement are not similar.

[0057] The input data arrangement and the output data arrangement being the same means that the input data arrangement and the output data arrangement are completely consistent.

[0058] When the input data arrangement and output data arrangement are the same, there is no need to perform data misalignment arrangement when loading the input data. The input data can be loaded continuously. At this time, a loading method with a larger read / write granularity can be selected, which is referred to as the first loading method. By loading the input data using the first loading method with a larger read / write granularity, it is possible to achieve merged loading of data with finer granularity, thereby improving the loading efficiency of the input data.

[0059] The input data arrangement and the output data arrangement are similar, which means that the input data arrangement and the output data arrangement are partially the same, and the input data can be stored continuously. Here, the input data arrangement and the output data arrangement are partially the same means that the input data arrangement and the output data arrangement are not completely the same, but there are parts of the same arrangement. The data input can be stored continuously, specifically, the input data required under a larger read-write granularity can be arranged on a continuous video memory. The larger read-write granularity referred to here can refer to the read-write granularity corresponding to the first loading method.

[0060] When the input data arrangement and output data arrangement are similar, there is a need for partial data misalignment in the input data loading, and the loading of the input data may be discontinuous. At this time, in order to improve the loading efficiency of the input data, you can also choose a loading method with a larger read / write granularity, that is, the first loading method. By loading the input data through the first loading method with a larger read / write granularity, you can achieve merged loading of finer-grained data, thereby improving the loading efficiency of the input data.

[0061] The input data arrangement and the output data arrangement are not similar, including the case where the input data arrangement and the output data arrangement are partially the same, but the input data is not stored continuously, and the case where the input data arrangement and the output data arrangement are completely different. That is, the input data arrangement and the output data arrangement are not similar, which can cover all cases except the above-mentioned cases where the input data arrangement and the output data arrangement are the same and similar.

[0062] When the input data arrangement is not similar to the output data arrangement, the input data needs to be staggered when loading, so as to adapt to the output data arrangement. At this time, the continuity of loading the input data is poor, and a loading method with a smaller read / write granularity can be selected, which is recorded as the second loading method. Loading the input data through the second loading method with a smaller read / write granularity can effectively improve the flexibility of loading the input data.

[0063] Therefore, two types of loading methods with different read and write granularities can be pre-set, namely, a first loading method with larger read and write granularity and a second loading method with smaller read and write granularity, and the first loading method is applied to load input data when it is determined that the input data arrangement and the output data arrangement are the same or similar, and the second loading method is applied to load input data when it is determined that the input data arrangement and the output data arrangement are not similar, thereby being compatible with operator implementations under different layout designs.

[0064] Based on any of the above embodiments, the method further includes: If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, then after the input data is loaded, the input data is rearranged.

[0065] Specifically, when the input data arrangement and the output data arrangement are similar, the first loading method with a larger read / write granularity is used to load the input data. Although this can improve the loading efficiency, the data loaded in this way cannot all be adapted to the output data arrangement. Therefore, after the data loading is completed, it is necessary to rearrange the loaded data, and use the data that can adapt to the output data arrangement after the data rearrangement is completed as the input data for the calculation of multiple subtasks.

[0066] The data reordering here may be performed in a data block or inside a thread warp, and may be implemented by way of shared memory or by a shfl class function, which is not specifically limited in the embodiment of the present invention.

[0067] Based on any of the above embodiments, the read and write granularity of the first loading mode is a warp, and the read and write granularity of the second loading mode is a single thread.

[0068] Specifically, based on the similarities and differences between the input data arrangement and the output data arrangement of the operator, the first loading method or the second loading method can be selected as the input data loading method.

[0069] The first loading method is to use warps as the read / write granularity. Using warps as the read / write granularity can support read / write merging, thereby accelerating read / write performance, and this method can adapt to the read / write modes of different hardware.

[0070] The second loading method is to use a single thread as the read / write granularity. Using a single thread as the read / write granularity can support flexible read / write scenarios such as broadcasting, and implement data reading and writing through the first address pointer and address offset to meet various data misalignment arrangement requirements.

[0071] In the method provided by the embodiment of the present invention, a more generalized high-performance read-write interface is implemented to support the reading of input data vectorized at different granularities through the input data loading methods of thread bundles and single threads. Among them, the input data loading method based on thread bundles, that is, the first loading method, can realize merged reading and writing, and its read-write bandwidth can reach the hardware upper limit. In addition, for the case of adopting the second loading method, that is, the case of not performing merged reading, merged writing is implemented in step 140, and a bandwidth of 60% or more can also be achieved.

[0072] Based on any of the above embodiments, in step 110, dividing the computing tasks of the operator based on the output layout of the operator includes: Determining a vectorization granularity corresponding to the operator based on a calculation mode of the operator; The computing tasks of the operator are divided based on the output layout of the operator and the vectorization granularity corresponding to the operator.

[0073] Specifically, for any operator, the calculation mode of the operator, that is, the algorithm applied when the operator is calculated, such as addition, subtraction, multiplication, and division, and also such as power pow and inverse tangent function atan.

[0074] It is understandable that different computing modes can adapt to calculations at different vectorization granularities, thereby adapting to the requirements of operators of different complexities for registers. The selection of vectorization granularity is a trade-off process that requires consideration of multiple factors, including the complexity of the computing mode, register requirements, processor architecture, etc. Larger vectorization granularity can improve computing performance, but may increase the risk of register overflow; smaller vectorization granularity can reduce register requirements, but may reduce computing performance. For example, computing modes such as pow and atan may consume a large number of registers, and the vectorization granularity needs to be reduced to prevent register overflow. For relatively simple computing modes such as addition, subtraction, multiplication and division, the vectorization granularity can be configured according to the optimal bandwidth for data loading.

[0075] Thus, the correspondence between different calculation modes and vectorization granularities can be preset, so that for an operator to be calculated, the vectorization granularity corresponding to the calculation mode can be determined according to the calculation mode of the operator.

[0076] Therefore, the computing task can be split in combination with the output layout and vectorization granularity of the operator. For example, different task splitting methods can be set in advance for different output layouts and vectorization granularities, so that the task splitting methods corresponding to the output layout and vectorization granularity can be directly applied to achieve the splitting of the computing task of the operator. For another example, the computing task can be split based on the output layout first, and then the subtasks that have been split can be further split based on the vectorization granularity. The embodiment of the present invention does not specifically limit this.

[0077] In the method provided in the embodiment of the present invention, parameterization of the computing mode is also implemented, that is, different vectorization granularities can be matched for different computing modes, thereby adapting to the requirements of operators of different complexities for registers.

[0078] Based on any of the above embodiments, in step 110, the dividing the computing tasks of the operator based on the output layout of the operator and the vectorization granularity corresponding to the operator includes: The computing tasks of the operator are divided based on a dimensional division order corresponding to the output layout and a division granularity corresponding to the vectorization granularity.

[0079] Specifically, in the process of splitting computing tasks according to the output layout and vectorization granularity, the output layout can be used to determine the dimensional splitting order when splitting tasks, and the vectorization granularity can be used to determine the splitting granularity when splitting tasks. Among them, the dimensional splitting order specifically refers to the execution order when splitting the corresponding dimensions in the output layout. The splitting granularity refers to the minimum granularity for task splitting, that is, the amount of data of the subtasks allocated to each thread for processing. Combining these two, the splitting of computing tasks can be achieved, thereby obtaining multiple subtasks that adapt to the output layout.

[0080] Based on any of the above embodiments, Figure 2 This is the second flow chart of the operator calculation method provided by the present invention, such as Figure 2 As shown, the operator calculation method may include the following steps: First, you can configure the corresponding vectorization granularity based on the operator's calculation mode; In addition, a generalized representation of the input data and output data of the operator can be constructed, where the input data and output data have their own layouts, that is, there are input layouts and output layouts.

[0081] Secondly, the computing tasks of the operator can be divided based on the output layout and vectorization granularity, thereby obtaining multiple subtasks.

[0082] Then, you can compare the similarities and differences between the input data arrangement and the output data arrangement of the operator. The data arrangement here includes the layout and shape. The similarities and differences thus determined are one of the following three situations: In the case where the input data arrangement and the output data arrangement are the same, the input data loading may be performed based on the first loading mode in which the thread warp is a read-write granularity; In the case where the arrangement of input data is similar to that of output data, the input data may be loaded based on the first loading mode in which the thread warp is a read-write granularity, and the input data may be rearranged after the loading is completed; In the case where the input data arrangement and the output data arrangement are not similar, the input data can be loaded based on the second loading mode with a single thread as the read-write granularity.

[0083] After the input data is loaded, vectorized calculations can be performed on the loaded input data based on the vectorized granularity, thereby implementing the running of multiple subtasks to obtain the running results of the multiple subtasks.

[0084] Subsequently, the running results of multiple subtasks can be merged and the output data can be written back after merging.

[0085] Based on any of the above embodiments, if the operator is a pointwise operator and the input data arrangement and output data arrangement of the operator are the same, the calculation steps of the operator may include: The high-performance computing task splitting interface elementwise__helper is applied to split the computing task of the operator based on the output layout of the operator and the vectorization granularity burst_mode of the operator, thereby obtaining multiple subtasks of the operator output by elementwise__helper. The multiple subtasks here can be specifically manifested as assigning each subtask to the coordinates of the first thread of the thread bundle warp.

[0086] In the case where the input data arrangement and the output data arrangement are the same, the input data of each input parameter can be merged and loaded through the high-performance read-write interface loadm, and then each subtask is run based on the input data to obtain the running results of each subtask.

[0087] Finally, based on the high-performance read-write structure storem, the running results of each subtask are merged and written, and the results are written back to the output data.

[0088] By performing calculations on pointwise operators based on the method provided in the embodiment of the present invention, compared with a method of writing kernel functions adapted to various layouts for pointwise operators respectively, the amount of code opened can be reduced by more than 6,000 lines.

[0089] Based on any of the above embodiments, if the operator is a layout conversion operator and the input data arrangement and output data arrangement of the operator are different, the calculation steps of the operator may include: The high-performance computing task splitting interface elementwise__helper is applied to split the computing task of the operator based on the output layout of the operator and the vectorization granularity burst_mode of the operator, thereby obtaining multiple subtasks of the operator output by elementwise__helper. The multiple subtasks here can be specifically manifested as assigning each subtask to the coordinates of the first thread of the thread bundle warp.

[0090] In the case of different input data arrangements and output data arrangements, the high-performance read-write interface load can be used to calculate the address of the input data required for output for each thread based on the local coordinates coord_array of each thread and read the corresponding data, and then each subtask is run based on the input data to obtain the running results of each subtask.

[0091] Finally, based on the high-performance read-write structure storem, the running results of each subtask are merged and written, and the results are written back to the output data.

[0092] The operator calculation device provided by the present invention is described below. The operator calculation device described below and the operator calculation method described above can be referenced to each other.

[0093] Figure 3 is a schematic diagram of the structure of the operator computing device provided by the present invention, such as Figure 3 As shown, the device comprises: A splitting unit 310, configured to split the computing task of the operator based on the output layout of the operator to obtain a plurality of subtasks; A loading unit 320, configured to determine a loading method for the input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement, and load the input data based on the loading method; An operating unit 330 is used to operate the plurality of subtasks based on the loaded input data to obtain the operating results of the plurality of subtasks; The output unit 340 is used to merge the running results of the multiple subtasks and use the merged results as the output data of the operator.

[0094] In the device provided by the embodiment of the present invention, the computing task is divided based on the output layout of the operator to realize the parallel operation of multiple subtasks; in addition, based on the similarities and differences between the input data arrangement and the output data arrangement of the operator, the input data loading method is determined, thereby realizing the loading of the input data and the operator calculation based on the loaded data, ensuring the adaptation of the operator calculation and the layout design, only one set of execution logic is needed to realize the operator calculation for various layouts, avoiding the situation of incrementally developing kernel function code to adapt to different layouts, greatly reducing the workload of code development, and improving the function support rate of the operator.

[0095] Based on any of the above embodiments, the loading unit is specifically used for: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0096] Based on any of the above embodiments, the loading unit is specifically used for: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

[0097] Based on any of the above embodiments, the loading unit is further used for: If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, then after the input data is loaded, the input data is rearranged.

[0098] Based on any of the above embodiments, the read and write granularity of the first loading mode is a warp, and the read and write granularity of the second loading mode is a single thread.

[0099] Based on any of the above embodiments, the segmentation unit is specifically used for: Determining a vectorization granularity corresponding to the operator based on a calculation mode of the operator; The computing tasks of the operator are divided based on the output layout of the operator and the vectorization granularity corresponding to the operator.

[0100] Based on any of the above embodiments, the segmentation unit is specifically used for: The computing tasks of the operator are divided based on a dimensional division order corresponding to the output layout and a division granularity corresponding to the vectorization granularity.

[0101] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the operator calculation method, which includes: Based on the output layout of the operator, the computing task of the operator is divided into multiple subtasks; Determine a loading method for input data of the operator based on similarities and differences between input data arrangement required for output of the operator and output data arrangement, and load the input data based on the loading method; Based on the loaded input data, the plurality of subtasks are run to obtain the running results of the plurality of subtasks; The running results of the multiple subtasks are merged, and the merged result is used as the output data of the operator.

[0102] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0103] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the operator calculation method provided by the above methods, the method comprising: Based on the output layout of the operator, the computing task of the operator is divided into multiple subtasks; Determine a loading method for input data of the operator based on similarities and differences between input data arrangement required for output of the operator and output data arrangement, and load the input data based on the loading method; Based on the loaded input data, the plurality of subtasks are run to obtain the running results of the plurality of subtasks; The running results of the multiple subtasks are merged, and the merged result is used as the output data of the operator.

[0104] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the operator calculation method provided by the above methods, the method comprising: Based on the output layout of the operator, the computing task of the operator is divided into multiple subtasks; Determine a loading method for input data of the operator based on similarities and differences between input data arrangement and output data arrangement of the operator, and load the input data based on the loading method; Based on the loaded input data, the plurality of subtasks are run to obtain the running results of the plurality of subtasks; The running results of the multiple subtasks are merged, and the merged result is used as the output data of the operator.

[0105] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0106] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator calculation method, characterized in that: include: Based on the output layout of the operator, the computing task of the operator is divided into multiple subtasks; Determine a loading method for input data of the operator based on similarities and differences between input data arrangement required for output of the operator and output data arrangement, and load the input data based on the loading method; Based on the loaded input data, the plurality of subtasks are run to obtain the running results of the plurality of subtasks; The running results of the multiple subtasks are merged, and the merged result is used as the output data of the operator.

2. The operator calculation method according to claim 1, characterized in that: The determining a loading method for the input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

3. The operator calculation method according to claim 1, characterized in that: The determining a loading method for the input data of the operator based on the similarities and differences between the input data arrangement required for the output of the operator and the output data arrangement includes: If the input data arrangement is the same as the output data arrangement, determining that the loading mode is the first loading mode; If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, determining that the loading mode is the first loading mode; Otherwise, determining that the loading mode is the second loading mode; The read and write granularity of the first loading mode is greater than the read and write granularity of the second loading mode.

4. The operator calculation method according to claim 3, characterized in that: Also includes: If the arrangement of the input data is partially identical to the arrangement of the output data, and the input data is stored continuously, then after the input data is loaded, the input data is rearranged.

5. The operator calculation method according to any one of claims 2 to 4, characterized in that: The read and write granularity of the first loading mode is a thread warp, and the read and write granularity of the second loading mode is a single thread.

6. The operator calculation method according to claim 1, characterized in that: The dividing the computing tasks of the operator based on the output layout of the operator includes: Determining a vectorization granularity corresponding to the operator based on a calculation mode of the operator; The computing tasks of the operator are divided based on the output layout of the operator and the vectorization granularity corresponding to the operator.

7. The operator calculation method according to claim 6, characterized in that: The dividing the computing tasks of the operator based on the output layout of the operator and the vectorization granularity corresponding to the operator includes: The computing tasks of the operator are divided based on a dimensional division order corresponding to the output layout and a division granularity corresponding to the vectorization granularity.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the operator calculation method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the operator calculation method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operator calculation method according to any one of claims 1 to 7 is implemented.

Images