Operator fusion method and system for a compiler facing deep learning inference tasks

Abstract

The application provides an operator fusion method for a deep learning inference task compiler, comprising the following steps: converting a standard model of a deep learning inference task model into an initial graph-level IR; traversing the initial graph-level IR, and identifying fusible operators in the initial graph-level IR as homogenous operators or heterogeneous operators; converting the initial graph-level IR into a tensor expression, and fusing the homogenous operators in the tensor expression; compiling the tensor expression into kernel source code, and fusing the heterogeneous operators in the kernel source code; and compiling the kernel source code into executable code and deploying and running the executable code. The application also provides an operator fusion system for a deep learning inference task compiler, and a data processing device for operator fusion of a deep learning inference task compiler.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an operator fusion method and system for compilers of deep learning inference tasks. Background Technology

[0002] Deep learning is widely used in artificial intelligence technologies such as image classification and recognition, and natural language understanding. With advancements in semiconductor technology and compiler software, the workloads of data centers and supercomputer systems are increasingly coming from artificial intelligence technologies, particularly deep learning neural networks.

[0003] To further improve the speed of computational tasks based on deep learning neural network models on devices, many deep learning compilers focus on optimizing individual operators within the neural network model. They often achieve this optimization by automatically adjusting scheduling parameters and calling operator libraries provided by hardware vendors. However, for many artificial intelligence (AI) computational tasks, especially deep learning inference tasks, the sequentially running individual operators within the neural network model often struggle to fully utilize the parallel computing units and other resources of the entire accelerator.

[0004] Current deep learning convolutional neural network designs increasingly favor multi-branch convolutions over single-branch ones, resulting in wider and deeper network models with a gradually decreasing number of operations per operator. Recent research shows that in the early image classification network VGG, the average number of floating-point operations per convolution operator was 2230 MFlops. However, in the image classification network NASNet, which emerged after 2018, the average number of floating-point operations for convolution operations dropped to 82 MFlops. This trend, coupled with further advancements in hardware technology, will exacerbate the problem of wasted computing power.

[0005] Existing deep learning compilers use operator fusion techniques to increase the computational scale of individual operators, reduce the number of data transfers between operators, and reduce the overhead of kernel function loading, thereby improving overall computational efficiency. This operator fusion technique can sequentially fuse specific combinations of operators along the execution path of a neural network model. However, this vertical fusion approach relies on operator libraries discontinued by vendors, limiting the number of operator combinations that can be fused, and it can only be fused along the computational path, failing to completely solve the problem of low computational resource utilization during system runtime.

[0006] By leveraging multi-stream or resource isolation mechanisms provided by hardware vendors, some optimization methods can allocate parallel operators to isolated hardware streams for concurrent execution, thereby improving the utilization of system computing resources and enhancing overall system performance. However, this approach is limited by the hardware resource isolation mechanism, making dynamic resource allocation and adjustment difficult and posing challenges to kernel function optimization. Furthermore, the scheduling overhead introduced by this method cannot be ignored, impacting overall execution efficiency.

[0007] Based on the acceleration libraries provided by equipment manufacturers, some methods can perform computation graph-level parallel operator fusion optimization. This optimization method can transform identical, parallelizable small operators into larger operators using corresponding batch processing functions in the acceleration library, thereby effectively improving accelerator utilization. However, this approach is limited by the manufacturer's acceleration library, supporting a limited number of operators, and cannot fuse parallel operators of different types and sizes. Furthermore, this method requires concatenation and decomposition of operator inputs and outputs before and after fusion, which also increases computational overhead significantly.

[0008] Recent work has proposed parallel operator fusion mechanisms that do not rely on vendor-accelerated libraries, such as Rammer and HFUSE. These methods can fuse multiple sequential or parallel Element-Wise operators into a single large operator, thereby making better use of hardware computing units and reducing end-to-end model execution time. However, these existing parallel operator fusion mechanisms cannot handle the fusion of common complex operators, such as convolution and matrix multiplication. Moreover, they do not provide optimization schemes for the fused operator, limiting the space for further optimization.

[0009] To address the problem that single-operator optimization in deep learning compilers cannot fully utilize accelerator computing resources, and the shortcomings of existing operator fusion solutions, this invention proposes an operator fusion method for deep learning inference task compilers. Summary of the Invention

[0010] To address the aforementioned problems, this invention proposes an operator fusion method for deep learning inference task compilers, comprising: transforming a standard model of a deep learning inference task model into an initial graph-level IR; traversing the initial graph-level IR and identifying fusionable operators in the initial graph-level IR as similar or dissimilar operators; transforming the initial graph-level IR into a tensor expression and fusing similar operators in the tensor expression; compiling the tensor expression into kernel source code and fusing dissimilar operators in the kernel source code; and compiling the kernel source code into executable code and deploying it for execution.

[0011] The operator fusion method of the present invention includes the following steps for identifying fusionable operators: performing a breadth-first search on the abstract syntax tree of the initial graph-level IR, and identifying operators at the same layer in the standard model in parallel to obtain multiple sets of fusionable operators; traversing each set of fusionable operators, identifying fusionable operators with the same operator type and input / output within the same set as similar operators in the same set; and identifying fusionable operators with different operator types and / or input / output within the same set, and whose maximum number of active thread blocks during deployment and runtime is within a certain range, as dissimilar operators in the same set.

[0012] The operator fusion method of the present invention includes the following steps for fusing similar operators: for the same group of similar operators, simultaneously adding the same batch dimension to the output and input parts of their corresponding tensor expressions to generate new tensor expressions, wherein the size of the batch dimension is equal to the number of similar operators in the group; replacing the position of the group of similar operators on the syntax tree of the standard model with the new tensor expression, and connecting the input and output of the new tensor expression to the syntax tree. The steps for merging the heterogeneous operators include: for the same group of heterogeneous operators, finding the operator kernel of the heterogeneous operator from the kernel source code; aligning the thread blocks of the operator kernel to a uniform size; merging the aligned operator kernels from source to source into a single new operator kernel, and setting the thread grid of the kernel function of the new operator kernel to a one-dimensional structure, remapping the thread index of the one-dimensional structure to ensure the correctness of the computation logic and input / output of the new operator kernel; remapping and adding all inputs and outputs of the heterogeneous operators in this group to the new operator kernel; and uniformly allocating storage space within the kernel function of the new operator kernel to replace the storage space requests within the individual kernel functions of the heterogeneous operators in this group.

[0013] The operator fusion method of the present invention includes the step of aligning the thread block as follows: for an operator kernel, reduce the number of its thread blocks and increase the size of its thread blocks, or retain the number of its thread blocks and add empty threads to its thread blocks to increase the size of the thread blocks, so that the aligned thread block is consistent with the maximum thread block size in the operator kernels of the same group.

[0014] The operator fusion method of the present invention further includes, before the step of identifying the fusionable operator, optimizing the initial graph-level IR using constant folding and / or dead code elimination and / or operator layout adjustment.

[0015] This invention also proposes an operator fusion system for deep learning inference task compilers, comprising: an identification module for identifying fusionable operators; converting the standard model of the deep learning inference task model into an initial graph-level IR; traversing the initial graph-level IR and identifying fusionable operators in the initial graph-level IR as similar or dissimilar operators; a similar operator fusion module for converting the initial graph-level IR into a tensor expression and fusing similar operators in the tensor expression; a dissimilar operator fusion module for compiling the tensor expression into kernel source code and fusing dissimilar operators in the kernel source code; and a running module for compiling the kernel source code into executable code and deploying it for execution.

[0016] The operator fusion system of the present invention includes an identification module comprising: a search submodule for performing a breadth-first search on the abstract syntax tree of the initial graph-level IR, and marking operators at the same layer in the standard model in parallel to obtain multiple sets of fusionable operators; an identification submodule for identifying the fusionable operators as similar or dissimilar operators; traversing each set of fusionable operators, marking fusionable operators with the same operator type and input / output within the same set as similar operators within the same set; and marking fusionable operators with different operator types and / or input / output within the same set, and whose maximum active thread block count is within a certain range during deployment and runtime, as dissimilar operators within the same set.

[0017] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the operator fusion method for a deep learning inference task compiler as described above is implemented.

[0018] The present invention also proposes a data processing apparatus, including the computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes the computer-executable instructions in the computer-readable storage medium, it implements an operator fusion method for a deep learning inference task compiler.

[0019] The operator fusion method provided by this invention is for parallelizable operators in a computation graph. Its advantage is that these operators to be fused can be located on different computation paths in the computation graph, and the operators to be fused can be non-specific operators. Attached Figure Description

[0020] Figure 1 This is a block diagram of the compilation system module structure for deep learning inference tasks according to the present invention.

[0021] Figure 2 This is a flowchart of the compilation system for deep learning inference tasks according to the present invention.

[0022] Figure 3 This is a flowchart of the operator fusion analysis and identification process of the present invention.

[0023] Figure 4 This is a flowchart of the fusion process of parallel operators of the same type in this invention.

[0024] Figure 5 This is a flowchart of the heterogeneous parallel operator fusion process of the present invention.

[0025] Figure 6 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0027] This invention provides a compiler operator fusion method and system for deep learning inference tasks. The aim is to fuse parallel, similar or dissimilar operators on a larger scale when compiling neural network models for deep learning, enabling the generated kernel function to more effectively utilize the target device's computing resources and further improving the throughput of the computing device when processing deep learning inference tasks.

[0028] To address the problems existing in the above-mentioned technical solutions, the present invention provides an operator fusion method and system for deep learning inference task compilers.

[0029] This invention provides an operator fusion method for compilers designed for deep learning tasks, comprising:

[0030] 1. Based on neural network structure information, operator types and operator resource request characteristics, generate the dependency relationships between operators and operator fusion identifiers.

[0031] 2. For fusionable parallel operators of the same type, operator fusion is achieved by expanding the dimension based on the tensor expression form of the operator.

[0032] 3. After merging parallel operators of the same type, the block size of the merged operator is increased a priori to enhance the efficiency of automatic tuning of the new operator.

[0033] 4. For heterogeneous parallelizable operators in deep learning models, existing deep learning compilers are first used to optimize and generate code for each operator separately. Based on the generated kernel function source code, a thread block alignment method is used to ensure that kernel functions from different operators have thread blocks of the same size, thus enabling the fusion of heterogeneous operators.

[0034] 5. After fusing heterogeneous parallel operator kernel functions, a thread block sorting rule based on computational complexity is adopted. Thread blocks from operators with higher computational complexity receive smaller block index values.

[0035] 6. After fusing the kernel functions of heterogeneous operators, the thread blocks of the fused kernel function contain the computational logic from the different operators, processing the original inputs and outputs from those operators. By re-establishing the thread indices within the thread blocks, it is ensured that each thread block activates the correct computational logic and matches the correct inputs and outputs.

[0036] 7. Before the fusion of heterogeneous operator kernel functions, each kernel function contains its own shared memory space allocation requests. After fusion, a unified memory resource request replaces the scattered resource requests within the fused operator kernel functions. The size of the unified resource request is consistent with the maximum resource request in the fused operator kernel functions.

[0037] Based on the operator fusion method with the aforementioned features, this invention provides a compiler system for deep learning inference tasks, such as... Figure 1 As shown, it includes:

[0038] 1. Deep Learning Standard Model Generation Module. This module is responsible for converting deep learning neural network models from different frameworks into standard models. This model will serve as input to the IR generation and optimization module. Common deep learning frameworks include TensorFlow, PyTorch, and Coffee. This invention uses ONNX (Open Neural Network Exchange) as the standard description for deep learning inference task models. ONNX is an open standard for representing deep learning models, enabling the conversion between neural network models built from different frameworks and providing a unified representation method for describing the structure and operations of neural networks.

[0039] 2. IR Generation and Optimization Module. This module is responsible for generating and optimizing intermediate representations (IRs) of the standard deep learning model. Based on IR, this module also performs operator parallel fusion analysis to identify and label suitable operators for parallel fusion. This module consists of three sub-modules: an IR generator module, which compiles the standard deep learning model into a basic intermediate representation; an IR optimizer module, which further optimizes the model at the graph and tensor levels based on the generated basic IR; and an operator fusion analyzer module, which performs parallelism and fusion analysis on the operators in the model based on the model's IR and labels the analysis results.

[0040] 3. Target Code Generation and Optimization Module. This module is responsible for compiling and optimizing the IR (In-Reference Engine) into executable code for the target device. It consists of four sub-modules: a target device kernel source code generator, which converts the optimized low-level IR into initial kernel source code matching the target device; a parallel operator kernel fusion generator, which fuses identified parallel operator kernels and generates the fused operator kernel source code; a target device executable code compiler, which further compiles the fused and optimized kernel source code into executable code matching the target device and runs it on the target device; and a fused kernel automatic tuner, which reads sampled data of code execution performance and uses heuristic algorithms to adjust operator optimization parameters and kernel fusion parameters to assist in generating optimal target device code.

[0041] Figure 2 This describes the workflow of the compiler system in this invention when using the method provided by this invention. Specifically, the operator fusion method of the compiler for deep learning inference tasks in this invention includes:

[0042] Step 1. Convert the deep learning inference task model from the user into an ONNX standard model.

[0043] Step 2. The IR generator receives the ONNX standard model from the user and calls the TVM's RelayIR generation function to convert it into an initial graph-level IR.

[0044] Step 3. The IR optimizer reads the initial graph-level IR and calls the TVMRelayIRPasses function to complete the optimization of the graph-level IR. The optimization method varies depending on the model and the types of operators involved. In the embodiments of this invention, it includes one or more of the following methods: constant folding, dead code elimination, operator layout adjustment, etc.

[0045] Step 4. The operator fusion analyzer reads and traverses the optimized graph-level IR, identifying fusionable parallel operators, including fusionable similar operators and fusionable dissimilar operators.

[0046] For the operator fusion analyzer module, this invention proposes a process for parallel operator fusion analysis and identification, the specific implementation details of which are as follows: Figure 3 As shown, the detailed steps are as follows:

[0047] Step 41. Perform a breadth-first search on the Abstract Syntax Tree (AST) of the graph-level IR and label the operators of the same layer in the deep learning network in parallel.

[0048] Step 42. Traverse each group of parallelizable operators. For operators that are of the same type and have the same input and output, mark them as parallelizable operators of the same type.

[0049] Step 43. Iterate through each group of parallelizable operators. For operators of different types or with different inputs and outputs, call the device runtime or driver API to obtain the "maximum number of active thread blocks per parallel unit" for the operator. Operators with similar numbers are identified as heterogeneous parallelizable operators. The judgment criteria can be derived from prior data.

[0050] Step 5. The IR generator calls the TVM lowering function to convert the optimized graph-level IR into a tensor expression (TE).

[0051] Step 6. Based on the tensor expression TE of the user model and the identification information of the parallelizable operators of the same type that can be fused, the IR optimizer completes the fusion of parallelizable operators of the same type by expanding the TE dimension.

[0052] Regarding the method for fusing similar parallel operators in step 6, this invention proposes a process for fusing similar parallel operators, the specific implementation details of which are as follows: Figure 4 As shown, the detailed steps are as follows:

[0053] Step 61. Based on the fusion markers of operators in the computation graph from the operator fusion analysis module, find a group of parallel operators of the same type that can be fused. Then, simultaneously add the same batch dimension to both the output and input parts of their tensor expressions. The size of the batch dimension is equal to the number of similar operators to be fused.

[0054] Step 62. Replace the positions of all previous operators of the same type on the syntax tree with the newly generated tensor operator expression, and make corresponding modifications to the input and output in the new fusion operator, connecting it to the input and output of the original operator.

[0055] Step 63. Check if all identified operators of the same type have been fused. If there are still operators to be fused, repeat steps 61-62 until all fused parallel operators of the same type have been fused.

[0056] Step 7. The IR generator calls the TVMTE downcompilation function to convert the tensor expression after the fusion of similar operators into a tensor-level IR.

[0057] Step 8. The target device kernel source code generator (CodeGenerator) reads the optimized tensor IR from Step 7 and compiles it into the corresponding kernel source code according to the device and backend compilation chain specified by the user.

[0058] Step 9. The parallel operator kernel fusion unit first reads the identification information of the heterogeneous parallel operators that can be fused. It then retrieves one set of operators to be fused. If all identified operators have been fused, proceed to step 11.

[0059] Step 10. The parallel operator kernel fusion processor fuses a set of heterogeneous parallel operator kernels. The main fusion methods and processes are described in steps 1001-1007 and... Figure 5 As described in the text. After fusion is complete, return to step 10 to prepare for the next set of operator kernel fusion.

[0060] Regarding the heterogeneous parallel operator kernel fusion method in step 10, this invention proposes a heterogeneous parallel operator kernel fusion process, the specific implementation details of which are as follows: Figure 5 As shown, the detailed steps are as follows:

[0061] Step 1001. Based on the fusion markers of operators in the computation graph by the operator fusion analysis module, find a set of fusionable heterogeneous parallel operators, and find the corresponding operator kernel in the operator kernel source code generated in step 9.

[0062] Step 1002. Heterogeneous operator kernels typically have different thread block size settings. The merged single operator kernels should have a consistent thread block size. Therefore, in this step, the merging unit will adjust the thread block size of operator kernels within the group to align thread block sizes from different operator kernels. For operators with relatively simple computational logic (such as matrix multiplication), the merging unit directly reduces the number of thread blocks and increases their thread block size to match the maximum thread block size of operator kernels within the group. For other operator kernels where the number of thread blocks is not easily adjusted directly, the merging unit will retain the number of thread blocks but add empty threads to their thread blocks to increase the thread block size, aligning it with the maximum thread block size within the group.

[0063] Step 1003. Rewrite the operator kernel source with the same thread block size as in Step 1002 into a single new operator kernel.

[0064] Step 1004. The merging unit sets the thread grid of the merged kernel function to a one-dimensional structure, meaning that all thread blocks from the heterogeneous operators will be arranged in a one-dimensional manner in the new kernel function. This simplifies the subsequent thread indexing (ThreadIndex).

[0065] Step 1005. The merging unit remaps the thread index based on the size of the new operator, the position of the original operator kernel in the new operator kernel after merging, and the correspondence between the input and output of the new operator and the input and output of the original operator, to ensure the correctness of the computation logic and input and output after the merging of each operator.

[0066] Step 1006. Remap and add all inputs and outputs of the original operators to the new operator kernel.

[0067] Step 1007. The merging unit allocates unified storage space within the merged operator kernel functions, replacing the individual storage space requests within each operator kernel function before merging. When accessing the unified shared storage, the thread's index value will also be mapped and modified accordingly in the previous step.

[0068] Step 1008. Check if all identified heterogeneous operators have been fused. If there are still remaining operators to be fused, repeat steps 1001-1007 until all heterogeneous parallel operators that can be fused have been fused.

[0069] Step 11. The target device executable code compiler will call the user-specified backend compilation chain to compile the fused operator kernel source code into executable code for the target computing device.

[0070] Step 12. Deploy and run binary code on the computing device.

[0071] Step 13. After the code execution is complete, the fusion kernel auto-tuner automatically adjusts the optimization compilation parameters of the system's multi-operators. The auto-tuner uses the current operator optimization configuration parameters as input to the evolutionary algorithm for this round and the configuration values ​​of the configuration search space. It uses the obtained code sampling execution time as the configuration gain for this set of operators and performs a new round of evolutionary iteration. This yields new operator optimization configuration parameters. These parameters are fed back to the IR optimizer and the target device kernel source code generator to produce new target device executable code. After repeating this process several times until performance convergence is achieved, or after reaching the predetermined optimization goal, the auto-tuning is complete.

[0072] The implementation methods for fusing similar parallel operators and fusing heterogeneous parallel operator kernel functions involved in steps 6 and 10 above include the following 7 characteristics:

[0073] 1) Operator parallel fusion: Existing deep learning model optimization compilers can only perform fusion optimization on specific combinations of operators in the sequential execution path of the neural network. However, the fusionable operators involved in steps 6 and 10 can be operators that are executed in parallel in the deep learning network, thus expanding the scope of fusionable operators.

[0074] 2) Extensive Fusion of Similar and Dissimilar Operators: Specific equipment manufacturers provide operator acceleration libraries that support the fusion of similar, parallelizable small operators by converting them into larger operators using corresponding batch processing functions within the acceleration library. This method is limited by the acceleration libraries provided by the manufacturers, supporting a limited number of operators. Furthermore, operators of different types and sizes cannot be fused. The fusion method proposed in this invention (steps 6 and 10) can theoretically fuse most common types of operators in parallel, further expanding the range of fusible operators.

[0075] 3) Low operator fusion overhead: The fusion method proposed in this invention (steps 6 and 10) does not require calling existing vendor acceleration library functions, thus avoiding connection and separation operations during fusion and further reducing the overhead of operator fusion.

[0076] 4) Fusion of parallel operators of the same kind based on tensor expressions: When fusing parallel operators of the same kind, the fusion is completed by dimensional expansion based on the tensor expressions (TE) of the operators.

[0077] 5) Heterogeneous Parallel Operator Fusion via Thread Block Size Alignment and Input-Output Aggregation: By aligning the operator kernel thread block size and aggregating inputs and outputs, a general automatic fusion of heterogeneous parallel operators is achieved. Adjusting the operator kernel thread block size can be done by directly changing the operator thread block size and its corresponding number, or by adding inactive threads. This fusion implementation method supports the automatic fusion of most operators without requiring vendor-provided custom operator fusion templates and functions.

[0078] 6) Reconstructing thread indexes to ensure the correctness of the fused computation logic: After fusion of heterogeneous operator kernels by aligning thread block sizes and aggregating input and output, the correctness of the original operator computation logic is ensured by reconstructing the original operator thread indexes.

[0079] 7) Unified resource allocation requests after operator fusion: Since the operator fusion methods provided in steps 6 and 10 can greatly expand the sources and scale of operator fusion, the new operator after operator kernel fusion will contain all the storage resource requests of the original operator. The parallel operator kernel fusion unit will merge these requests to form a unified and shared storage resource pool to meet the computing needs of all fused operators.

[0080] The operator fusion method of the compiler for deep learning inference tasks of the present invention is mainly used for the optimization and deployment of artificial intelligence applications, and can also be applied to basic software services of computer clusters, etc., but the present invention is not limited thereto.

[0081] Figure 6 This is a schematic diagram of the data processing apparatus of the present invention. Figure 6As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, they implement the operator fusion method described above for a deep learning inference task compiler. Those skilled in the art will understand that all or part of the steps in the above method can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by an integrated circuit to implement its corresponding function, or it can be implemented as a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.

[0082] The method proposed in this invention has the following advantages: 1) Wide range of fusionable operators: It can efficiently fuse operators of the same type and size, as well as operators of different types. This makes the algorithm more adaptable, and the fused operators can make better use of hardware resources; 2) Many types of fusionable operators: Theoretically, this method can fuse most common types of operators; 3) Operator fusion is not limited by the operator library provided by the vendor: Operator fusion does not require calling existing vendor library functions, thus avoiding connection and separation operations during fusion, further reducing the overhead of operator fusion; 4) No need for resource isolation to achieve high utilization of computing resources: The method proposed in this invention does not require the computing resource isolation mechanism provided by the vendor to improve resource utilization. This avoids the potential resource waste and additional overhead caused by resource isolation; 5) Improved system resource utilization and reduced execution time of deep learning inference tasks: For multiple common network models, compared with vendor function operator libraries, the method provided in this invention can achieve a speed improvement of 1.64 to 4.94 times; compared with other latest operator fusion technologies, the method provided in this invention has an efficiency improvement of 3.37 to 14.68 times.

[0083] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. An operator fusion method for compilers targeting deep learning inference tasks, characterized in that, include: Transform the standard model of deep learning inference task model into an initial graph-level IR; A breadth-first search is performed on the abstract syntax tree of the initial graph-level IR. Operators at the same level in the standard model are marked in parallel to obtain multiple sets of fusionable operators. Each set of fusionable operators is traversed. Fusionable operators with the same operator type and input / output within the same set are marked as similar operators in the same set. Fusionable operators with different operator types and / or input / output within the same set, and whose maximum number of active thread blocks during deployment and runtime are within a certain range, are marked as dissimilar operators in the same set. The initial graph-level IR is transformed into a tensor expression, and similar operators in the tensor expression are fused. Compile the tensor expression into kernel source code and integrate the heterogeneous operators in the kernel source code; Compile the kernel source code into executable code and deploy and run it.

2. The operator fusion method as described in claim 1, characterized in that, The steps for fusing operators of the same class include: For the same group of operators, add the same batch dimension to both the output and input of their corresponding tensor expressions to generate new tensor expressions. The size of this batch dimension is equal to the number of operators in the same group. Replace the position of the same group of operators in the syntax tree of the standard model with this new tensor expression, and connect the input and output of the new tensor expression to the syntax tree.

3. The operator fusion method as described in claim 1, characterized in that, The steps for fusing this heterogeneous operator include: For the same group of heterogeneous operators, find the operator kernel of that heterogeneous operator from the kernel source code; align the thread blocks of the operator kernel to a uniform size; merge the aligned operator kernels from source to source into a single new operator kernel, and set the thread grid of the kernel function of the new operator kernel to a one-dimensional structure, remapping the thread index of the one-dimensional structure to ensure the correctness of the computation logic and input / output of the new operator kernel; remap and add all inputs and outputs of the heterogeneous operators in this group to the new operator kernel; uniformly allocate storage space within the kernel function of the new operator kernel to replace the storage space requests within the individual kernel functions of the heterogeneous operators in this group.

4. The operator fusion method as described in claim 3, characterized in that, The steps to align this thread block include: For a given operator kernel, reduce the number of its thread blocks and increase the size of its thread blocks, or retain the number of its thread blocks and add empty threads to its thread blocks to increase the size of the thread blocks, so that the aligned thread blocks are the same as the largest thread block size in the operator kernels of this group.

5. The operator fusion method as described in claim 1, characterized in that, Before the step of identifying the fusionable operator, the following steps are also included: The initial graph-level IR is optimized using constant folding and / or dead code elimination and / or operator layout adjustments.

6. An operator fusion system for a compiler for deep learning inference tasks, characterized in that, include: The identifier module is used to identify fusionable operators; it transforms the standard model of the deep learning inference task model into an initial graph-level IR. Traverse the initial graph-level IR and identify fusionable operators in the initial graph-level IR as similar or dissimilar operators; including a search submodule and an identification submodule; the search submodule is used to perform a breadth-first search on the abstract syntax tree of the initial graph-level IR and to identify operators at the same layer in the standard model in parallel to obtain multiple sets of fusionable operators; the identification submodule is used to identify the fusionable operators as similar or dissimilar operators; traverse each set of fusionable operators, identify fusionable operators with the same operator type and input / output within the same set as similar operators in the same set; identify fusionable operators with different operator types and / or input / output within the same set and whose maximum active thread block count is within a certain range during deployment as dissimilar operators in the same set; The same type operator fusion module is used to convert the initial graph-level IR into a tensor expression and fuse the same type operators in the tensor expression; The heterogeneous operator fusion module is used to compile the tensor expression into kernel source code and fuse the heterogeneous operators in the kernel source code. The runtime module is used to compile the kernel source code into executable code and deploy it for operation.

7. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the operator fusion method for a deep learning inference task compiler as described in any one of claims 1 to 5 is implemented.

8. A data processing apparatus comprising the computer-readable storage medium as claimed in claim 7, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it implements an operator fusion method for a deep learning inference task compiler.

Images