Cross-platform automatic code generation method, device, equipment, medium and product

Abstract

The embodiments of the present application provide a cross-platform automatic code generation method, apparatus, device, medium and product, which relate to the field of code processing technology. The method comprises: parsing a file of a set deep learning model, and performing operator fusion and memory reuse processing on the model to perform graph optimization; performing operator analysis on the model after graph optimization, and optimizing different operators using different conversion paths Pass; generating and tuning backend code based on the optimized intermediate representation IR; improving code generation efficiency, accelerating tuning, and improving the flexibility of code optimization and cross-platform deployment.

Description

Technical Field

[0001] The present application relates to the field of code processing technology, and in particular to a cross-platform automatic code generation method, device, equipment, medium and product. Background Art

[0002] In recent years, the field of deep learning has developed rapidly. Various network structures based on the Transformer neural network model have achieved remarkable results in fields such as natural language processing, and are increasingly being used in other fields. Therefore, it is becoming increasingly important to accelerate the Transformer model. Deep learning models represented by the Transformer model often have a large amount of computation. With the end of Moore's Law, traditional CPUs are difficult to meet the computing needs of deep learning models, so heterogeneous computing devices such as GPUs are developing rapidly. NVIDIA GPU graphics processors have attracted many developers with the CUDA programming model, which makes other computing devices relatively weak in software ecology compared to NVIDIA GPUs. However, developing the same software ecology on different platforms will lead to a huge amount of development.

[0003] At present, there are many popular tools for code generation and acceleration of common operators in neural networks, such as TVM and triton. The above tools are representative in the industry, but they also have many shortcomings. First, most of their application scenarios are targeted at NVIDIA GPUs, and the optimization effect on other platforms is poor; in addition, the existing code generation tools themselves have design shortcomings, which limit the granularity of their optimization and the representation of optimization steps. In terms of tuning, the above tools also have certain problems. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a cross-platform automatic code generation method, device, equipment, medium and product to solve the problems of inflexible cross-platform capabilities, time-consuming tuning and poor optimization algorithm effects of existing neural network model operator generation tools.

[0005] In a first aspect, an embodiment of the present application provides a cross-platform automatic code generation method, comprising:

[0006] Parse the file of the set deep learning model, and perform operator fusion and memory reuse on the model to optimize the graph;

[0007] Perform operator analysis on the model after graph optimization, and use different conversion paths Pass for optimization for different operators;

[0008] Generate and tune the backend code based on the optimized intermediate representation IR.

[0009] In the above implementation process, the file of the set deep learning model is parsed, and the model is subjected to operator fusion and memory reuse processing for graph optimization; the operator analysis is performed on the model after graph optimization, and different conversion paths Pass are used for optimization for different operators; the backend code is generated and tuned based on the optimized intermediate representation IR; the code generation efficiency is improved, the tuning is accelerated, and the flexibility of code optimization and cross-platform deployment is improved.

[0010] Furthermore, parsing the file of the set deep learning model includes:

[0011] Save deep learning models through the open neural network exchange model ONNX;

[0012] Load the ONNX model file and parse the structure, nodes, and tensors of the entire network model.

[0013] In the above implementation process, ONNX stands for Open Neural Network Exchange, which allows different artificial intelligence frameworks to adopt the same format to store and exchange model data, thereby achieving model interoperability and seamless conversion between frameworks.

[0014] Furthermore, the model is subjected to operator fusion and memory reuse processing to perform graph optimization, including:

[0015] The computational graph in a deep learning model consists of multiple nodes, each of which performs a set computational operation;

[0016] Identify fusible nodes in the deep learning model and merge the identified fusible nodes into a new first node;

[0017] Replace the original multiple nodes with the first fused node and update the structure of the computational graph;

[0018] Perform memory reuse on the updated computational graph.

[0019] In the above implementation process, operator fusion and memory reuse processing of the model are important means of graph optimization. By reducing the overhead of memory access and calculation, the reasoning speed and performance of the model can be improved.

[0020] Furthermore, the optimization using different conversion paths Pass for different operators includes:

[0021] Through the loop transformation Pass, the operators are loop-segmented, rearranged, and expanded;

[0022] Through the memory access pass, the operator is allocated shared memory, data is moved, and read and write vectorized at the GPU level;

[0023] Through the scheduling class Pass, double buffer, multi-stage asynchronous prefetching and conditional branch optimization operations are performed on the operator.

[0024] In the above implementation process, the operator is optimized.

[0025] Furthermore, the generation and tuning of the backend code based on the optimized intermediate representation IR includes:

[0026] Determine the framework of parallel IR structure;

[0027] Experiment with available computation schedules, record program performance under different block sizes, thread data layouts, and computation schedules, and select an implementation as the output of code generation.

[0028] In the above implementation process, the backend code is generated and tuned.

[0029] Furthermore, the framework for determining the parallel IR structure includes:

[0030] Search for data block size and determine the block strategy;

[0031] Perform a search to select a data layout.

[0032] In the above implementation process, a block partitioning strategy is determined, and a search is performed to select the optimal data layout.

[0033] Further, the searching and selecting data layout includes:

[0034] Determine the memory access mode of the thread warp and perform pruning if there is a memory bank conflict;

[0035] The setting parameter search is performed at the thread block and thread warp levels; wherein the setting parameters include Repeat parameters and Layout parameters.

[0036] In the above implementation process, the data layout search is performed to resolve bank conflicts. When searching for Repeat parameters and Layout parameters at the thread block and thread warp levels, it is necessary to judge the memory access mode of the thread warp in advance, and prune the bank conflicts to speed up the search for the optimal program.

[0037] Further, the attempting to schedule available computing includes:

[0038] Data is pre-fetched by double buffering.

[0039] In the above implementation process, the specified computing schedule is applied, and it is determined whether to apply the specified computing schedule for a specific data block size and data layout based on the actual measured results.

[0040] In a second aspect, an embodiment of the present application provides a cross-platform automatic code generation device, comprising:

[0041] The graph optimization module is used to parse the file of the set deep learning model and perform operator fusion and memory reuse processing on the model to perform graph optimization;

[0042] The operator analysis module is used to perform operator analysis on the model after graph optimization and optimize different operators using different conversion paths.

[0043] The generation and tuning module is used to generate and tune the backend code based on the optimized intermediate representation IR.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0045] A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the cross-platform automatic code generation method as described above.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a server, the cross-platform automatic code generation method as described above is implemented.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions, and when the instructions are executed by a computer, the computer implements the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a cross-platform automatic code generation method provided in an embodiment of the present application;

[0050] Figure 2A schematic diagram of a specific algorithm for operator fusion of a cross-platform automatic code generation method provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of Pass and its use introduced in the embodiment of the present application for a cross-platform automatic code generation method provided in the embodiment of the present application;

[0052] Figure 4 A schematic diagram of an example of the function of a memory access class Pass in a cross-platform automatic code generation method provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of an example of the role of a loop transformation pass in a cross-platform automatic code generation method provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram showing an example of the double_buffer Pass function of a cross-platform automatic code generation method provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of a code generation process of a cross-platform automatic code generation method provided in an embodiment of the present application;

[0056] Figure 8 It is a structural diagram of a cross-platform automatic code generation device provided in an embodiment of the present application;

[0057] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0059] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0060] In recent years, the field of deep learning has developed rapidly. Various network structures based on the Transformer neural network model have achieved remarkable results in fields such as natural language processing, and are increasingly being used in other fields. Therefore, it is becoming increasingly important to accelerate the Transformer model. Deep learning models represented by the Transformer model often have a large amount of computation. With the end of Moore's Law, traditional CPUs are difficult to meet the computing needs of deep learning models, so heterogeneous computing devices such as GPUs have developed rapidly. NVIDIA GPU has attracted many developers with its CUDA programming model, which makes other computing devices weaker than NVIDIA GPUs in software ecology. However, developing the same software ecology on different platforms will lead to a huge amount of development. Therefore, it is particularly important to study cross-platform code generation technology for the Transformer model.

[0061] At present, the existing cross-platform code generation technologies and tools mainly include:

[0062] (1) Template-based code generation: Representative works in this method include the search engine Freemarker and the template engine Thymeleaf. This method uses placeholders in the template and then fills the data into the placeholders to generate code.

[0063] (2) Macro-based code generation: This method uses macro definitions to generate code in the preprocessing stage. A macro is a special function or command that encapsulates a set of specific code operations. This method generates the corresponding code by defining macros in the source code and then expanding these macros in the preprocessing stage.

[0064] (3) TVM, a machine learning compiler framework: TVM is an end-to-end deep learning compiler stack designed to optimize the deployment and execution of deep learning models. TVM provides performance portability of deep learning workloads across different hardware backends by developing optimizations at the computational graph level and operator level.

[0065] (4) Parallel Programming Language Triton: Triton is a programming language and compiler designed by OpenAI specifically for deep learning and high-performance computing tasks. It aims to simplify the development of complex operations on GPUs and optimize their execution.

[0066] From the perspective of cross-platform capabilities, end-to-end capabilities, code readability, complexity, debuggability, and optimization methods, the above code generation methods all have certain advantages and disadvantages. With the development of domestic independent platforms, for domestic manufacturers of heterogeneous computing devices, in order to improve market competitiveness, in addition to improving the computing power of hardware devices, it is also important to improve the construction of the software ecosystem. In cross-platform scenarios, existing tools have limited support for platforms other than NVIDIA.

[0067] At present, there are many popular tools for code generation and acceleration of common operators in neural networks, such as the neural network compiler TVM and the cloud management platform triton. The above tools are representative in the industry, but they also have many shortcomings. First, most of their application scenarios are targeted at NVIDIA GPUs, and the optimization effect on other platforms is poor; in addition, the existing code generation tools themselves have design shortcomings, which limit the granularity of their optimization and the representation of optimization steps. In terms of tuning, the above tools also have certain problems.

[0068] TVM is an end-to-end deep learning compiler stack designed to optimize the deployment and execution of deep learning models. TVM provides performance portability of deep learning workloads across different hardware backends by exploiting both computation graph-level and operator-level optimizations. TVM addresses optimization challenges unique to deep learning, such as high-level operator fusion, primitives mapped to arbitrary hardware, and memory latency hiding. It also automatically optimizes low-level programs based on hardware characteristics by adopting a novel, learning-based cost modeling approach to rapidly explore code optimizations. Experimental results show that TVM delivers performance competitive with hand-tuned libraries for state-of-the-art low-power CPUs, mobile GPUs, and server-class GPUs on hardware backends. However, TVM is built on loop-oriented scheduling primitives that operate on loop structures in tensor programs in a declarative manner. Although loop-oriented scheduling primitives have achieved great success in simplifying the development of tensor programs, they are difficult to implement for some specific optimizations. In particular, loop-oriented scheduling primitives cannot express the fine-tuned tensor program transformations required for the following key optimizations: (1) double buffering; (2) thread block swizzle; and (3) multi-stage asynchronous prefetching. These optimizations are used extensively in the Kernel library, but are difficult to implement using declarative loop-oriented scheduling.

[0069] Triton is a programming language and compiler designed by OpenAI specifically for deep learning and high-performance computing tasks. It aims to simplify the development of complex operations on GPUs and optimize their execution. Triton's goal is to provide an open source environment for writing code quickly with higher productivity than CUDA. The core concept of Triton is based on a block-based programming paradigm, which is different from the traditional SIMT programming model written by CUDA. The traditional programming model programs at the fine-grained level of threads, while Triton programs at the fine-grained level of thread blocks. This simplifies the code writing process for developers to a certain extent, but it also makes it difficult for Triton to handle non-intensive computing tasks with finer granularity below the block. In addition, triton cannot support end-to-end model compilation, and users still need to write kernels by hand, and the quality of the code determines the quality of the generated kernel to a certain extent. In terms of tuning, the algorithms provided by triton are also relatively simple, relying on developers to build the tuning space by themselves and then search, which also limits the use of triton. In terms of performance, although triton also has cross-platform capabilities, after testing, it is mainly optimized for NVIDIA A100 GPU, and the effect on other GPUs is not ideal.

[0070] When conducting research on cross-platform automatic code generation technology for Transformer model reasoning, the embodiments of the present application found that the defects of the prior art are mainly reflected in the following aspects:

[0071] (1) Design defects limit optimization methods: For triton and similar block-based programming tools, their main focus is to simplify the programming paradigm to achieve the convenience of DSL, so they cannot achieve more refined optimization at a fine-grained level below the block layer; for TVM, since it is built on the basis of loop-oriented scheduling primitives, TVM cannot express the fine-tuning tensor program conversion steps required for some key optimizations. For example, common optimization methods such as Doublebuffering, thread block swizzle, and multi-stage asynchronous prefetching. In addition, for the time-consuming Mulit-head Attention calculation in the Transformer model, the Flash-Attention work proposed a fusion operator method to complete the three operations of matrix multiplication, Softmax, and matrix multiplication in one kernel function. This fusion operator cannot generate high-quality kernels under TVM.

[0072] (2) The search efficiency of the tuning framework is insufficient: TVM will experience long kernel tuning problems when searching for rare effective schedules in a huge tuning space. For example, it takes 15 hours for AutoTVM to tune a simple InceptionV3 convolutional neural network model on a modern GPU. For triton, the operator tuning framework it provides requires users to build the space and write config parameters by themselves, and its search algorithm is a simple traversal. Therefore, the tuning effect is subject to user experience, and it takes a long time when the tuning space is large.

[0073] (3) Limited cross-platform support: Triton and TVM can provide good support on a small number of GPUs, but the operator optimization effect on other platforms is poor.

[0074] The embodiments of this application investigate the compiler infrastructure LLVM and MLIR, and combine the design concepts of triton and TVM to find that the code generation technology based on MLIR can achieve fine-grained optimization and improve the quality of the flash-attention fusion kernel in the transformer model, while facilitating the reuse of existing optimization methods and expanding optimization methods. At the same time, development based on MLIR also ensures cross-platform support capabilities; by pre-configuring parameters, the user's development difficulty can be simplified and experience dependence can be reduced, and pruning can effectively improve the search efficiency when the tuning space is large; the difficulty of this invention is that operator optimization involves multiple optimization methods, such as Double buffering, thread block swizzle, and multi-stage asynchronous prefetching. These methods currently lack corresponding support in MLIR.

[0075] Through in-depth exploration, the embodiments of the present application cleverly abstracted the expression forms of these commonly used optimization methods under MLIR, and encapsulated them using Pass, making it more convenient to reuse these optimization methods. Next, the embodiments of the present application conducted an in-depth study of the data layout scheme in the tuning process by analyzing the configuration parameters of the kernel, and proposed a pruning method, which makes it possible to quickly search for the optimal configuration in a large tuning space. Finally, the embodiments of the present application attempt to combine the above parts, and supplement the front-end analysis of the onnx model and the optimization algorithm at the graph level, and finally propose an end-to-end code generation tool for Transformer models with cross-platform capabilities.

[0076] In order to address the problems of existing neural network model operator generation tools in terms of cross-platform capabilities, tuning time consumption, and optimization algorithms, the embodiments of the present application propose a cross-platform code generation tool for Transformer models, which can provide end-to-end support for domestic platforms and other platforms, improve code generation efficiency, and have scalability.

[0077] Please see Figure 1 , Figure 1 A schematic diagram of a cross-platform automatic code generation method provided in an embodiment of the present application. The cross-platform automatic code generation method includes:

[0078] 100. Parse the file of the set deep learning model, and perform operator fusion and memory reuse processing on the model to perform graph optimization.

[0079] Specifically, the deep learning model is saved through the open neural network exchange model ONNX; the ONNX model file is loaded to parse the structure, nodes and tensors of the entire network model.

[0080] Optionally, perform operator fusion and memory reuse processing on the deep learning model: the computational graph in the deep learning model consists of multiple nodes, and each node performs a set computing operation; identify the fusible nodes in the deep learning model and merge the identified fusible nodes into a new first node; replace the original multiple nodes with the fused first node, and update the structure of the computational graph; perform memory reuse processing on the updated computational graph.

[0081] ONNX model is an open file format designed for machine learning and used to store trained models. ONNX stands for Open Neural Network Exchange, which allows different artificial intelligence frameworks to use the same format to store and exchange model data, thereby achieving model interoperability and seamless conversion between frameworks. ONNX models are saved in files with the suffix .proto. Protoc Compiler can be used to generate corresponding C++ header files and source files based on proto files, which define the information of all data structures in the ONNX model, defined in the form of C++ classes, and also include internal member functions.

[0082] ONNX builds its own model format based on the serialization method of Google Protobuf, in which the key data structures include Node, which represents the node in the calculation graph, Initializer, which is mainly used to represent the weight, and Attributes, which represents the operator attributes. In order to complete the downward parsing of the ONNX model, the present invention defines a three-layer abstraction with Net, Node, and Tensor as the core at the upper layer, which is mainly used to store the calculation graph information extracted from ONNX and complete some basic calculation graph optimization.

[0083] Network tensor Tensor: Tensor corresponds to the TensorProto and ValueProto data structures in ONNX, where the former is used to save initialized tensors, such as model weights, operator tensor properties, and the latter is used to save uninitialized tensors, such as model inputs, feature graphs between nodes, etc. In the embodiment of the present application, we use the different life stages of Tensor to represent this difference. In particular, the feature graph between nodes in the ONNX model, that is, the Tensor that only exists at runtime, is represented as a connection property Connection Properties, and this data structure also represents the edge of the calculation graph. In order to better optimize the calculation graph and unified management at runtime, the embodiment of the present application also incorporates this representation into the Tensor data structure. In the generated code, Tensor exists as a parameter of the kernel Kernel, corresponding to an important data structure in MLIR. In the optimization process of the calculation graph, Tensor is the basis for optimizations such as constant folding and memory reuse.

[0084] Node: Node corresponds to the Node of the same name in ONNX. It is the basic unit for representing operators and the basic unit for code generation. The operator set of the embodiment of the present application inherits the Opset of ONNX and is compatible with ONNX versions 11 to 18. At the same time, in order to complete optimization operations such as operator fusion, the embodiment of the present application supports custom operator types, such as the fusion operator Flash_Attention. Node contains different attribute variables, which are variable when the model is built. When the model is exported as an ONNX model, these variables are fixed as constants. For example, for a deep learning framework based on dynamic graphs such as Pytorch, for control statements such as if, the results of the branches will be fixed when exporting ONNX. These attributes are also used as constants to guide how to generate Kenrel code during code generation, such as the axis attribute of the LayerNormalization operator, which determines which dimensions of the input Tensor will be Normalized by the final generated code. During the running stage, Node marks the corresponding computing function with a function pointer. The OPset in ONNX is very large, and it takes a lot of work to support all of it. In order to support operators that cannot complete efficient code generation for the time being and complete end-to-end deployment, the upper layer of the embodiment of the present application also supports operators provided by third-party libraries, such as Cudnn (a GPU acceleration library designed specifically for deep learning tasks) on the CUDA (parallel computing platform and programming model) platform, Cublas (a library that provides GPU accelerated implementation of basic linear algebra subroutines BLAS), etc.

[0085] Network Net: Net corresponds to the entire ONNX model. Unlike the ONNX model, which is only a storage format for the calculation graph, the Net data structure maintains and manages the entire life cycle of the calculation graph from creation, operation, export, and destruction. Specifically, Net saves all nodes Node, and uses them as operation objects to complete simple graph optimization based on these nodes, such as some operator fusion, constant folding, etc. Similar to ONNX, Net also saves the calculation graph based on the non-continuous storage method of nodes and edges, which is similar to the cross-linked list in general graph storage. This method is generally used for sparse directed graphs with fewer edges, which is very consistent with the calculation graph of the neural network model. And it is easier to optimize the calculation graph based on this data structure. Net also saves all Tensors, including the input and output of the model, weights, and feature graphs between nodes. In order to complete the correct creation of the network, optimization of the calculation graph, and the correct execution of the final code, it is necessary to uniformly manage these Tensors.

[0086] For example, please refer to Figure 2 , demonstrating the specific algorithm of operator fusion.

[0087] As described above, the operator fusion and memory reuse processing of the model in the embodiment of the present application is an important means of graph optimization. By reducing the overhead of memory access and calculation, the reasoning speed and performance of the model can be improved.

[0088] 200. Perform operator analysis on the model after graph optimization, and use different conversion paths Pass for optimization for different operators.

[0089] Specifically, through the loop transformation Pass, the operator is loop-segmented, rearranged, and expanded; through the memory access Pass, the operator is GPU-level shared memory allocated, data moved, and read and write vectorized; through the scheduling class Pass, the operator is double-buffered, multi-stage asynchronous prefetched, and conditional branch optimized.

[0090] Pass is a term in MLIR, which is used to transform the dialect in MLIR, transform and convert operations, and optimize the code. Pass is the key implementation part of gradually reducing MLIR expressions to machine code, and is also the core of MLIR compiler to achieve code optimization and acceleration.

[0091] For example, Figure 3 The Pass introduced in the present invention and its uses are shown. These Passes are used to optimize operators, which are mainly divided into three categories: loop transformation, memory access, and scheduling. Through the loop transformation Pass, loop segmentation, rearrangement, and expansion can be achieved; through the memory access Pass, shared memory allocation, data movement, read and write vectorization, and other optimizations at the GPU level can be achieved; through the scheduling Pass, currently common optimization methods such as double buffer, multi-stage asynchronous prefetching, conditional branch optimization, and other operations can be achieved. Figure 4 , 5 As shown in Figure 6, an example of the effect of the above Pass is given, showing the original IR and the converted IR respectively.

[0092] 300. Generate and tune the backend code based on the optimized intermediate representation IR.

[0093] Specifically, determine the framework of the parallel IR structure; try out available computational schedules, record program performance under different block sizes, thread data layouts, and computational schedules, and select implementations as outputs for code generation.

[0094] Optionally, the framework for determining the parallel IR structure includes: searching for the data block size and determining the block strategy; and searching and selecting the data layout.

[0095] Optionally, the searching and selecting data layout includes: determining the memory access mode of the thread bundle and performing pruning processing in the case of memory bank conflicts; performing setting parameter searches at the levels of thread blocks and thread bundles; wherein the setting parameters include Repeat parameters and Layout parameters.

[0096] Optionally, the attempting to schedule available computations includes pre-fetching data by means of double buffering.

[0097] For example, Figure 7 The overall framework of the code generation strategy is described. The first step is to search for the data block size. For computationally intensive tasks such as matrix multiplication and fused attention, data block and outer product calculation are common optimization methods, but the block size is platform-dependent when performing cross-platform code generation. After determining a block strategy, the layout of the block data on all threads of the thread block is also a variable, which affects the shared memory access mode. It is necessary to search to select the optimal data layout, which can be achieved through storage allocation and data transfer. After completing the first two steps, the framework of the parallel IR structure is determined. Finally, the available computation scheduling is tried, the adjustment strategy is selected, and the program performance under different block sizes, thread data layouts, and computation scheduling is recorded, and the optimal implementation is selected as the output of code generation. Among them, different performance test examples are obtained: time consumption, block size, data layout, and scheduling strategy.

[0098] When performing data block division for matrix multiplication, the block parameters include not only the block size of the C matrix at the thread block and thread level , , , , which also contains the block parameters for matrices A and B The block parameters required for the fusion Attention calculation are more but also limited. Because the size of the input matrix is ​​also limited, the optimal block size can be selected by exhaustive method. In order to improve the efficiency of code generation, the embodiment of the present application performs pruning operation when performing parameter search.

[0099] After determining the data block size, the size of the data block processed by a thread block and the size of the data block processed by a thread are already determined. The reason for searching the data layout is to resolve bank conflicts. When searching for Repeat and Layout parameters at the thread block and warp level, it is necessary to judge the memory access mode of the warp in advance, prune the bank conflicts, and speed up the search for the optimal program.

[0100] Tail processing is mainly responsible for applying the specified computing schedule, and decides whether to apply the specified computing schedule for a specific data block size and data layout based on the measured results. The typical computing schedule is pipeline processing, that is, pre-fetching data through double buffering. Figure 8 Soft pipelining in matrix multiplication is demonstrated.

[0101] The time-consuming operators in the Transformer model of the present application embodiment mainly include matrix multiplication and attention. Targeted optimization of these two operators will improve the execution efficiency of the model. The present application embodiment performs graph detection and operator fusion on the neural network model, and then performs different pass optimization and parameter tuning according to the operator type, and finally generates the kernel of the operator to achieve the effect of running acceleration.

[0102] As described above, the embodiment of the present application parses the file of the set deep learning model, and performs operator fusion and memory reuse processing on the model to perform graph optimization; performs operator analysis on the model after graph optimization, and optimizes different operators using different conversion paths Pass; generates and tunes the backend code based on the optimized intermediate representation IR; improves code generation efficiency, accelerates tuning, and improves the flexibility of code optimization and cross-platform deployment.

[0103] The embodiment of the present application is based on the operator optimization pass design of MLIR: for the double-buffer and multi-stage asynchronous prefetching in the operator optimization process, the corresponding pass is designed to realize the encapsulation and reuse of the optimization means. The optimization algorithm of the tuning search process: the parameter configuration in the tuning space is pruned through the algorithm to achieve the purpose of narrowing the search range to accelerate the tuning.

[0104] The above steps are not to be performed in a strict order as described in the numbers, but should be understood as an overall solution.

[0105] In the second aspect, based on the above embodiments, Figure 8 This is a schematic diagram of the structure of a cross-platform automatic code generation device provided by an embodiment of the present application. Figure 8 The cross-platform automatic code generation device provided in this embodiment specifically includes: a graph optimization module 801, an operator analysis module 802 and a generation and tuning module 803.

[0106] Among them, the graph optimization module 801 is used to parse the file of the set deep learning model, and perform operator fusion and memory reuse processing on the model to perform graph optimization; the operator analysis module 802 is used to perform operator analysis on the graph-optimized model, and use different conversion paths Pass for optimization for different operators; the generation and tuning module 803 is used to generate and tune the backend code based on the optimized intermediate representation IR.

[0107] As described above, the embodiment of the present application parses the file of the set deep learning model, and performs operator fusion and memory reuse processing on the model to perform graph optimization; performs operator analysis on the model after graph optimization, and optimizes different operators using different conversion paths Pass; generates and tunes the backend code based on the optimized intermediate representation IR; improves code generation efficiency, accelerates tuning, and improves the flexibility of code optimization and cross-platform deployment.

[0108] The cross-platform automatic code generation device provided in the embodiment of the present application can be used to execute the cross-platform automatic code generation method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0109] In a third aspect, an embodiment of the present application further provides an electronic device, which can integrate the cross-platform automatic code generation device provided by an embodiment of the present application. Fig. 9 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig. 9 The electronic device includes: an input device 93, an output device 94, a memory 92 and one or more processors 91; the memory 92 is used to store one or more programs; when the one or more programs are executed by the one or more processors 91, the one or more processors 91 implement the cross-platform automatic code generation method provided in the above embodiment. The input device 93, the output device 94, the memory 92 and the processor 91 can be connected by a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0110] The processor 91 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 92, that is, implements the above-mentioned cross-platform automatic code generation method.

[0111] The electronic device provided above can be used to execute the cross-platform automatic code generation method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0112] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the cross-platform automatic code generation method as described above, and can achieve the same beneficial effects.

[0113] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present application is not limited to the cross-platform automatic code generation method described above, and can also execute related operations in the cross-platform automatic code generation method provided in any embodiment of the present application.

[0114] In a fifth aspect, the embodiments of the present application also provide a computer program product, and the methods described in the various embodiments of the present application can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the processes or functions described in the various embodiments of the present application are executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM (Open Application Model) or other programmable device.

[0115] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0116] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0118] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0119] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0120] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0121] It should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

Claims

1. A cross-platform automatic code generation method, characterized in that: include: Parse the file of the set deep learning model, and perform operator fusion and memory reuse on the model to optimize the graph; Perform operator analysis on the model after graph optimization, and use different conversion paths Pass for optimization for different operators; Generate and tune the backend code based on the optimized intermediate representation IR; The performing of operator fusion and memory reuse processing on the model to perform graph optimization includes: The computational graph in a deep learning model consists of multiple nodes, each of which performs a set computational operation; Identify fusible nodes in the deep learning model and merge the identified fusible nodes into a new first node; Replace the original multiple nodes with the first fused node and update the structure of the computational graph; Perform memory reuse on the updated computational graph; The generation and tuning of the backend code based on the optimized intermediate representation IR includes: Determine the framework of parallel IR structure; Experiment with available computation schedules, record program performance under different block sizes, thread data layouts, and computation schedules, and select implementations as output for code generation; Wherein, the framework for determining the parallel IR structure includes: Searching for the data block size and determining the block strategy; wherein the layout of the block data on all threads of the thread block is a variable; Conduct a search to select the data layout; The searching and selecting data layout comprises: Determine the memory access mode of the thread warp and perform pruning if there is a memory bank conflict; The setting parameter search is performed at the thread block and thread warp levels; wherein the setting parameters include Repeat parameters and Layout parameters.

2. The cross-platform automatic code generation method according to claim 1, characterized in that: The step of parsing the file of the set deep learning model includes: Save deep learning models through the open neural network exchange model ONNX; Load the ONNX model file and parse the structure, nodes, and tensors of the entire network model.

3. The cross-platform automatic code generation method according to claim 1, characterized in that: The optimization using different conversion paths Pass for different operators includes: Through the loop transformation Pass, the operators are loop-segmented, rearranged, and expanded; Through the memory access pass, the operator is allocated shared memory, data is moved, and read and write vectorized at the GPU level; Through the scheduling class Pass, double buffer, multi-stage asynchronous prefetching and conditional branch optimization operations are performed on the operator.

4. The cross-platform automatic code generation method according to claim 1, characterized in that: The attempting of available computing scheduling includes: Data is pre-fetched by double buffering.

5. A cross-platform automatic code generation device, characterized in that: include: The graph optimization module is used to parse the file of the set deep learning model and perform operator fusion and memory reuse processing on the model to perform graph optimization; The operator analysis module is used to perform operator analysis on the model after graph optimization and optimize different operators using different conversion paths. Generate and tune the module, which is used to generate and tune the backend code based on the optimized intermediate representation IR; The generation and tuning of the backend code based on the optimized intermediate representation IR includes: Determine the framework of parallel IR structure; Experiment with available computation schedules, record program performance under different block sizes, thread data layouts, and computation schedules, and select implementations as output for code generation; Wherein, the framework for determining the parallel IR structure includes: Searching for the data block size and determining the block strategy; wherein the layout of the block data on all threads of the thread block is a variable; Conduct a search to select the data layout; The searching and selecting data layout comprises: Determine the memory access mode of the thread warp and perform pruning if there is a memory bank conflict; The setting parameter search is performed at the thread block and thread warp levels; wherein the setting parameters include Repeat parameters and Layout parameters.

6. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the processor is connected to the memory via the bus, and the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, they are used to implement the cross-platform automatic code generation method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a server, implements the cross-platform automatic code generation method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product comprises instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1-4.

Images