Operator-based compiling method and device, computer device and storage medium
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN SMARTMORE TECH CO LTD
- Filing Date
- 2022-11-01
- Publication Date
- 2026-08-07
AI Technical Summary
但是，仅利用编译优化框架进行编译优化的效果不佳
[0025]上述基于算子的编译方法、装置、计算机设备、存储介质和计算机程序产品，通过获取神经网络对应的初始计算图；初始计算图包括多个第一算子；第一算子为基于编译优化框架支持的第一编程语言所定义得到的算子；根据预设的算子转换映射信息，确定与各第一算子相匹配的第二算子；第二算子为根据第二编程语言进行自定义得到的算子；将初始计算图中各第一算子转换为相匹配的第二算子以进行计算图更新处理，得到与编译优化框架适配的目标计算图；基于编译优化框架对目标计算图进行编译，以生成神经网络对应的目标程序代码。也就是说，本申请通过将第二编程语言进行自定义得到的第二算子作为独立的算子，代替基于编译优化框架支持的第一编程语言所定义得到的第一算子接入至编译优化框架的生态之中，能够在编译优化框架的基础上，利用第二编程语言的性能优势使第二算子充分享受到算子融合等编译优化操作，进而提高编译优化效果。
Smart Images
Figure CN115686527B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of compiler technology, and in particular to an operator-based compiler method, apparatus, computer device, and storage medium. Background Technology
[0002] With the widespread adoption of deep learning, enabling the rapid and high-performance deployment of artificial neural networks on a wide variety of hardware devices has become a crucial challenge for engineers. Consequently, deep learning-based compiler optimization frameworks have emerged. These frameworks can compile and optimize the computation graph corresponding to the neural network, thereby automatically generating target program code that can run on the target platform. However, relying solely on compiler optimization frameworks for compilation optimization is not always effective. Summary of the Invention
[0003] Therefore, it is necessary to provide an operator-based compilation method, apparatus, computer device, and storage medium that can improve the compilation optimization effect in response to the above-mentioned technical problems.
[0004] Firstly, this application provides an operator-based compilation method. The method includes:
[0005] Obtain the initial computation graph corresponding to the neural network; the initial computation graph includes multiple first operators; the first operators are operators defined based on the first programming language supported by the compiler optimization framework;
[0006] Based on the preset operator transformation mapping information, a second operator matching each first operator is determined; the second operator is a custom operator obtained according to a second programming language.
[0007] Each first operator in the initial computation graph is converted into a matching second operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework;
[0008] The target computation graph is compiled using a compiler optimization framework to generate the target program code corresponding to the neural network.
[0009] Secondly, this application also provides an operator-based compilation apparatus. The apparatus includes:
[0010] The acquisition module is used to acquire the initial computation graph corresponding to the neural network; the initial computation graph includes multiple first operators; the first operators are operators defined based on the first programming language supported by the compiler optimization framework;
[0011] The matching module is used to determine the second operator that matches each first operator based on the preset operator transformation mapping information; the second operator is a custom operator obtained according to the second programming language.
[0012] The conversion module is used to convert each first operator in the initial computation graph into a matching second operator for computation graph update processing, so as to obtain a target computation graph that is compatible with the compilation optimization framework.
[0013] The compilation module is used to compile the target computation graph based on the compilation optimization framework to generate the target program code corresponding to the neural network.
[0014] In some embodiments, the matching module is further configured to obtain operator conversion mapping information between each first operator and each second operator from a preset operator registry; and to traverse each first operator based on the operator conversion mapping information to determine the second operator that has a mapping relationship with the first operator.
[0015] In some embodiments, the conversion module includes an operator conversion unit, an operator matching unit, and an update unit. The operator conversion unit is used to convert each first operator in the initial computation graph into a matching second operator; the operator matching unit is used to determine the second operator to be fused from the converted second operators according to the execution relationship between them; the update unit is used to fuse the second operators to be fused to generate a fusion operator, so as to perform computation graph update processing and obtain a target computation graph adapted to the compiler optimization framework.
[0016] In some embodiments, the operator matching unit is further configured to traverse the inputs and outputs of each transformed second operator based on the execution relationship, so as to identify a group of second operators with input-output relationships from the transformed second operators; the outputs of some second operators in the group of second operators are the inputs of other second operators.
[0017] In some embodiments, the update unit is further configured to fuse the second operators in the second operator group to generate a fusion operator for computation graph update processing, thereby obtaining a target computation graph adapted to the compilation optimization framework.
[0018] In some embodiments, the second operator is obtained after generating at least one of the following custom information based on the second programming language: custom input and output information of the second operator; custom algorithm information of the second operator; and custom scheduling information of the second operator.
[0019] In some embodiments, the custom input and output information of the second operator includes at least one of scalar input information, image input information, and operator result input information; the scalar input information is defined based on the fixed-point data type and floating-point data type of the second programming language; the image input information is defined based on the image data storage object of the second programming language; and the operator result input information is used to define the binding relationship between the inputs and outputs of multiple second operators.
[0020] In some embodiments, the algorithm customization information of the second operator includes information on the declaration and definition of various variables of function types in the second programming language.
[0021] In some embodiments, the custom scheduling information for the second operator includes at least one of a first scheduling mode, a second scheduling mode, and a third scheduling mode. The first scheduling mode is used to indicate that when a non-current second operator uses the output of the current second operator as input, the computation of the current second operator is inlined; the second scheduling mode is used to indicate that when a non-current second operator uses the output of the current second operator as input and the non-current second operator is scheduled based on each computation partition, the non-current second operator is computed only after the computation of the current second operator corresponding to the computation partition of the non-current second operator is completed; the third scheduling mode is used to indicate that the non-current second operator is computed only after the computation of the current second operator is completed.
[0022] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program implementing the steps in the above-described operator-based compilation method.
[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps in the above-described operator-based compilation method.
[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps in the operator-based compilation method described above.
[0025] The aforementioned operator-based compilation method, apparatus, computer device, storage medium, and computer program product obtain an initial computation graph corresponding to a neural network. This initial computation graph includes multiple first operators, each defined using a first programming language supported by a compilation optimization framework. Based on preset operator transformation mapping information, second operators matching each first operator are determined. These second operators are custom-defined using a second programming language. Each first operator in the initial computation graph is converted into a matching second operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework. The target computation graph is then compiled using the compilation optimization framework to generate the target program code corresponding to the neural network. In other words, this application, by using custom-defined second operators obtained using a second programming language as independent operators to replace the first operators defined using the first programming language supported by the compilation optimization framework, integrates them into the ecosystem of the compilation optimization framework. This allows the second operators to fully benefit from compilation optimization operations such as operator fusion, leveraging the performance advantages of the second programming language within the framework, thereby improving the compilation optimization effect. Attached Figure Description
[0026] Figure 1 This is a flowchart illustrating the operator-based compilation method in some embodiments of this application;
[0027] Figure 2 This is an initial computational graph in some embodiments of this application;
[0028] Figure 3 This is a schematic diagram showing the update of the initial computation graph in some embodiments of this application;
[0029] Figure 4 This is a target computation graph in some embodiments of this application;
[0030] Figure 5 This is a structural block diagram of an operator-based compilation device in some embodiments of this application;
[0031] Figure 6 This is a diagram showing the internal structure of a computer device in some embodiments of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0033] In some embodiments, such as Figure 1As shown, an operator-based compilation method is provided. This embodiment illustrates the application of this method to a computer device, which can be a server or a terminal. It is understood that the method can also be applied to a system including a server and a terminal, and implemented through the interaction between the server and the terminal, wherein the terminal includes at least one of a mobile phone, tablet computer, laptop computer, or desktop computer. In this embodiment, the method includes the following steps:
[0034] Step 102: Obtain the initial computation graph corresponding to the neural network.
[0035] Neural networks are complex network systems formed by the extensive interconnection of a large number of simple processing units (called neurons). Neural networks are a subset of machine learning and are the core of deep learning algorithms.
[0036] In some embodiments, the neural network in this application refers to a neural network used for image processing (e.g., image recognition, image segmentation, etc.), such as a convolutional neural network and a generative adversarial neural network.
[0037] A computational graph is a graphical representation of the computational process of a neural network; in this case, the graph refers to a data structure graph. This graph includes multiple nodes and edges, where nodes can be represented by operators, and the lines connecting the nodes are called edges. An initial computational graph refers to a graph that uses multiple first operators to represent the computational process of the neural network.
[0038] The first operator is the operator defined by the first programming language supported by the compiler optimization framework.
[0039] A compiler optimization framework refers to a compiler that can generate effective code implementations of deep learning models on various hardware platforms, making the conversion from model definition to specific code implementation highly optimized for model specifications and hardware architecture.
[0040] A programming language is a language that both computers and humans can understand, enabling communication between them. First-level programming languages are those supported by compiler optimization frameworks and are distinct from second-level programming languages.
[0041] Specifically, the computer device acquires an initial computation graph corresponding to the neural network, which includes multiple first operators.
[0042] In some embodiments, one possible initial computational graph of the neural network includes one or more inputs, multiple first operators, and an output. For example... Figure 2As shown, input A is used as the input of the first operator 1. The output obtained from the calculation of the first operator 1 and input B are used together as the input of the first operator 2. The output obtained from the calculation of the first operator 2 is used as the input of the first operator 3. Then, the calculation of the first operator 3 is continued to obtain the final output of the initial calculation graph.
[0043] In some embodiments, the compilation optimization framework includes at least one of a Multi-Level Intermediate Representation (MLIR) framework or an end-to-end deep learning compiler (Tensor Virtual Machine, TVM). MLIR is a novel framework for building reusable and scalable compilers, aiming to address software fragmentation, improve compilation on heterogeneous hardware, reduce the cost of building domain-specific compilers, and facilitate the connection of existing compilers. TVM, on the other hand, is an open-source, end-to-end deep learning model compilation framework designed to optimize the inference speed of deep learning models in arbitrary target environments.
[0044] Step 104: Determine the second operator that matches each of the first operators based on the preset operator transformation mapping information.
[0045] The operator transformation mapping information stores the mapping relationship between the first operator and each second operator defined by the first programming language supported by the compilation optimization framework.
[0046] The second operator is a custom operator derived from a second programming language other than the first programming language.
[0047] In some embodiments, the second programming language may be, but is not limited to, at least one of a halide language, namely the Halide language or the C++ programming language.
[0048] Halide is a programming language that is characterized by its convenience, speed, and high performance in image processing and matrix calculations. Halide is not an independent programming language, but a domain-specific language (DSL) based on C++.
[0049] C++ is a high-level programming language that was developed and extended from the C language. C++ allows for procedural programming like C, object-based programming with abstract data types, and object-oriented programming with inheritance and polymorphism.
[0050] Specifically, the computer device determines the second operator that matches each of the first operators based on the mapping relationship between the first operator and each of the second operators.
[0051] Step 106: Convert each first operator in the initial computation graph into a matching second operator for computation graph update processing to obtain a target computation graph adapted to the compilation optimization framework.
[0052] Specifically, the computer device converts each first operator in the initial computation graph into a matching second operator and updates the initial computation graph to obtain a target computation graph that is compatible with the compiler optimization framework, that is, a target computation graph that meets the compilation requirements of the compiler optimization framework.
[0053] It should be noted that during the computation graph update process, there may be cases where some first operators cannot be identified as having a matching second operator in the operator transformation mapping information; that is, these first operators do not have a mapping relationship with the second operator. Therefore, there are situations where the first operators cannot be transformed into second operators. Based on this, during the computation graph update process, we consider replacing the first operators that can be identified as having a matching second operator in the operator transformation mapping information with the second operator, while leaving the first operators that cannot be identified as having a matching second operator unchanged.
[0054] In some embodiments, for Figure 2 The various first operators shown are transformed to update the initial computation graph, resulting in the schematic diagram as follows. Figure 3 As shown. The specific conversion process is as follows: First operator 1 is converted to obtain second operator 1, first operator 2 is converted to obtain second operator 2, and first operator 3 is converted to obtain second operator 3. After obtaining second operator 1, second operator 2, and second operator 3, the directly converted second operator 1, second operator 2, and second operator 3 may not meet the compilation requirements of the compiler optimization framework itself. Therefore, it is necessary to perform format conversion on second operator 1, second operator 2, and second operator 3 respectively so that the converted conversion adaptation operators 1, 2, and 3 all meet the compilation requirements of the compiler optimization framework. Specifically, conversion adaptation operators 1, 2, and 3 can all be stored using attributes in the compiler optimization framework, such as MLIR.
[0055] Step 108: Compile the target computation graph based on the compilation optimization framework to generate the target program code corresponding to the neural network.
[0056] Specifically, the computer device compiles the target computation graph based on the compilation function built into the compilation optimization framework to generate the target program code corresponding to the neural network.
[0057] In some embodiments, the neural network of this application can be a network for image processing. By transforming the initial computation graph of the neural network and compiling the transformed target computation graph, target program code corresponding to the neural network and used for image processing can be generated.
[0058] In some embodiments, since each of the second operators in this application is custom-developed based on a second programming language, the code generation function can also be implemented by the second programming language. Specifically, if the second programming language is Halide, the Just-In-Time (JIT) compiler built into Halide can be used to compile, optimize, and execute the target computation graph represented by the neural network at runtime. Alternatively, the Ahead-of-Time (AOT) compiler in Halide can be used to compile the target computation graph to generate the corresponding binary library file, providing callable C or C++ interfaces for user programs.
[0059] JIT (Just-In-Time) compilation utilizes just-in-time compiler technology to compile called functions or program segments into machine code and load them into memory at runtime, thereby speeding up program execution. AOT (Ahead-of-Time) compilation, on the other hand, refers to the process of converting bytecode into machine code before the program runs.
[0060] In the aforementioned operator-based compilation method, an initial computation graph corresponding to the neural network is obtained. This initial computation graph includes multiple first operators, which are operators defined using a first programming language supported by the compilation optimization framework. Second operators matching each first operator are determined based on preset operator transformation mapping information. These second operators are custom-defined using a second programming language. Each first operator in the initial computation graph is converted into a matching second operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework. The target computation graph is then compiled using the compilation optimization framework to generate the target program code corresponding to the neural network. In other words, this application integrates the second operators, custom-defined using a second programming language, into the compilation optimization framework ecosystem as independent operators, replacing the first operators defined using the first programming language supported by the framework. This allows the second operators to fully benefit from operator fusion and other compilation optimization operations, thereby improving the compilation optimization effect.
[0061] In some embodiments, step 104 specifically includes, but is not limited to: obtaining operator conversion mapping information between each first operator and each second operator from a preset operator registry; and traversing each first operator based on the operator conversion mapping information to determine the second operator that has a mapping relationship with the first operator.
[0062] The operator registry is a static variable in C++ that is defined during software initialization. The operator conversion mapping information in the operator registry has already established mapping relationships between multiple first operators and multiple second operators; for example, first operator 1 can be mapped to second operator 1, first operator 2 can be mapped to second operator 2, first operator 3 can be mapped to second operator 3, and so on.
[0063] Specifically, the computer device retrieves the conversion mapping relationship between each first operator and each second operator from the operator registry, and iterates through all the first operators to determine the second operators that have a mapping relationship with the first operators. This application, by pre-defining the operator registry, can automatically determine the second operators that match the first operators based on the operator conversion mapping information recorded in the operator registry, thereby improving the efficiency of compilation optimization.
[0064] In some embodiments, step 106 specifically includes, but is not limited to: converting each first operator in the initial computation graph into a matching second operator; determining the second operator to be fused from the converted second operators according to the execution relationship between them; and fusion the second operators to be fused to generate a fusion operator for computation graph update processing to obtain a target computation graph adapted to the compilation optimization framework.
[0065] The execution relation is used to indicate the execution order of operator calculations for each transformed second operator.
[0066] The second operator to be fused refers to the operators that can be fused among the transformed second operators.
[0067] Specifically, after determining the second operators matching each first operator based on the operator transformation mapping information, the computer device converts each first operator in the initial computation graph into a matching second operator. Next, based on the execution relationship of the converted second operators, the computer device determines the second operator to be fused from the converted second operators. The computer device then fuses the second operators to be fused to generate a fusion operator, replacing the operators to be fused with the fusion operator, thereby updating the computation graph and obtaining a computation graph adapted to the compiler optimization framework. This application, by combining operators to be fused that meet the operator fusion conditions into a fusion operator, can reduce some unnecessary steps in the computation process, thereby improving the efficiency of operator computation.
[0068] In some embodiments, for Figure 3 The target computation graph obtained by fusing the various second operators shown can be referred to Figure 4 The specific fusion process is as follows: Second operator 1, second operator 2, and second operator 3 are fused to obtain a fusion operator, which includes two inputs, input A and input B. Since the fusion operator obtained by direct fusion may not meet the compilation requirements of the compiler optimization framework itself, it is also necessary to perform format conversion on the fusion operator so that the fusion adaptation operator obtained after format conversion meets the compilation requirements of the compiler optimization framework. The fusion adaptation operator can be stored using attributes in compiler optimization frameworks, such as MLIR.
[0069] In some embodiments, the step "determine the second operator to be fused from the converted second operators according to the execution relationship between the converted second operators" specifically includes, but is not limited to, traversing the inputs and outputs of the converted second operators based on the execution relationship to identify a group of second operators with input-output relationships from the converted second operators.
[0070] In this second operator group, the output of some second operators is the input of others. For example, if the second operators include second operator A and second operator B, and the output of second operator A is equal to the input of second operator B, then second operator A and second operator B have an input-output relationship. Furthermore, second operator A and second operator B can be combined into a second operator group, and further attempts can be made to fuse second operator A and second operator B to obtain a fused operator.
[0071] Specifically, the computer device, based on the execution relationship between the various second operators, sequentially traverses the inputs and outputs of each transformed second operator, identifying operator groups with input-output relationships from the transformed second operators. This application, by traversing the inputs and outputs of each second operator, can quickly and effectively identify second operator groups capable of operator fusion.
[0072] In some embodiments, the step "fusion of the second operators to be fused to generate a fusion operator, so as to perform computation graph update processing and obtain a target computation graph adapted to the compiler optimization framework" specifically includes, but is not limited to: fusion of the second operators in the second operator group to generate a fusion operator, so as to perform computation graph update processing and obtain a target computation graph adapted to the compiler optimization framework.
[0073] Specifically, the computer device merges the individual second operators in the second operator group to generate a fused operator, and replaces each second operator in the second operator group with the corresponding fused operator to update the initial computation graph, thereby obtaining a target computation graph adapted to the compiler optimization framework. This application effectively identifies the second operator group capable of operator fusion and merges the individual second operators, reducing unnecessary steps in the computation process and improving the efficiency of operator computation.
[0074] In some embodiments, the operator fusion process can be as follows: Iterate through each of the transformed second operators. If there exists a second operator C, and one of its inputs is the result of another second operator D, then attempt to fuse the second operator C and the second operator D. Specifically, if, in the compiler optimization framework, the input name of the second operator D to the second operator C is a string named StringDC, then search for an input with the same name, StringDC, in the input definition of the second operator C. If found, this input is defined as the output of the second operator D. Simultaneously, all the original inputs of the second operator D and all the inputs of the second operator C except for the second operator D are called the inputs of the fused operator. The fused operator will schedule the second operator D according to the scheduling information in the second operator D.
[0075] In some embodiments, the second operator is obtained after generating at least one of the following custom information based on the second programming language: custom input and output information of the second operator; custom algorithm information of the second operator; and custom scheduling information of the second operator.
[0076] Among them, the input and output customization information of the second operator is used to customize the input and output of the second operator, the algorithm customization information of the second operator is used to customize the calculation content of the second operator, and the customization scheduling information of the second operator is used to customize the scheduling mode of the second operator during operation.
[0077] In some embodiments, the custom input / output information of the second operator includes at least one of scalar input information, image input information, and operator result input information.
[0078] Among them, scalar input information is defined based on fixed-point and floating-point data types of the second programming language; image input information is defined based on image data storage objects of the second programming language; and operator result input information is used to define the binding relationship between the inputs and outputs of multiple second operators.
[0079] In some embodiments, if the second programming language refers to C++, the scalar input information is obtained based on the definitions of basic fixed-point and floating-point data types in C++.
[0080] In some embodiments, if the second programming language refers to Halide, the image input information is customized based on the image data storage object in Halide. The image data storage object can be a buffer object. A buffer object is a container for a fixed amount of data, serving as a storage or segmented transport area. The information stored in the image data storage object includes at least one of the following: image type, image dimensions, or image size. It should be noted that when customizing the input to the second operator, if the input to the second operator is an image, the image type and image dimensions must be explicitly defined, while the image size can be selectively defined.
[0081] In some embodiments, operator result input information is used to define the binding relationship between the inputs and outputs of multiple second operators. Specifically, when customizing the input of a current second operator, the input of the current second operator can correspond to the output of a non-current second operator. This is one way to bind the output of a non-current second operator as the input of the current second operator. Here, the current second operator refers to the second operator that is being customized or being calculated, and the non-current second operators refer to all other second operators besides the current second operator.
[0082] It should be noted that while binding the current second operator and non-current second operators, it is also necessary to check the constraints such as type and dimension of the above operators to ensure that the output of the bound non-current second operator and the input of the current second operator can be adapted. In this way, multiple second operators can be chained together to form various computational graphs.
[0083] In some embodiments, the algorithm customization information of the second operator includes information on the declaration and definition of various variables of function types in the second programming language.
[0084] Specifically, the algorithm-defined information of the second operator describes the computational content of the second operator. Essentially, this information involves declaring and defining variables in one or more second programming languages, such as the C++ function types in Halide. Defining a variable means allocating storage space for it and specifying initial values. Declaring a variable means indicating its type and name to the program.
[0085] For example, in the Halide language, the variables in the C++ function types include variables A and B. If variable A has not been defined, it can be defined using "int A". If variable B has been defined, it can be declared using "extern int B".
[0086] In some embodiments, the custom scheduling information of the second operator includes at least one of a first scheduling mode, a second scheduling mode, and a third scheduling mode.
[0087] In some embodiments, the custom scheduling information of the second operator describes how the second operator is computed at runtime, and each second operator is scheduled entirely based on a second programming language, such as Halide. Building upon this, this application further abstracts the cross-operator scheduling method, allowing users to schedule using the second operator as the basic unit.
[0088] In some embodiments, a first scheduling mode is used to inline the computation of the current second operator when a non-current second operator uses the output of the current second operator as input. A second scheduling mode is used to instruct that, when a non-current second operator uses the output of the current second operator as input and the non-current second operator is scheduled based on computation partitions, the non-current second operator is computed only after the computation of the current second operator corresponding to the computation partition of the non-current second operator is completed. A third scheduling mode is used to instruct that the non-current second operator is computed only after the computation of the current second operator is completed.
[0089] Inlined computation of the current second operator means that the current second operator will not occupy a separate computation step, and all places that use the result of the current second operator need to substitute and replace its definition. For example, if there are operators f(x,y) = x + y and operators g(x,y) = f(x,y) + f(x+1,y+1), if operator f is inlined into g, then f will not be computed, and the definition of g will be rewritten as g(x,y) = x + y + (x+1) + (y+1).
[0090] Scheduling computational partitions means processing each partition sequentially, only moving to the next partition after the current one has been processed. For example, if the input to a second operator is a two-dimensional image, the normal scheduling order is to process each pixel in the first row from left to right. After the first row is completed, the second row is processed from left to right, and so on. In this application, a computational partition refers to a partition formed by cutting the image into blocks along all or some dimensions. Scheduling computational partitions means that the next partition's computation must begin only after all pixels within the current partition have been processed, while the computation order of pixels within a single partition remains normal.
[0091] In some embodiments, if the input of a second operator is a two-dimensional image, the third scheduling mode means that, for the entire image, all calculations of the current second operator must be completed before the calculation of subsequent second operators can begin.
[0092] In some embodiments, the compilation optimization framework is MLIR, and the second programming language is Halide. The operator-based compilation method of this application further includes, but is not limited to, the following steps:
[0093] First, obtain the initial computation graph corresponding to the neural network.
[0094] In some embodiments, the initial computation graph includes a plurality of first operators, which are operators defined based on a first programming language supported by MLIR.
[0095] Secondly, the operator transformation mapping information between each first operator and each second operator is obtained from the preset operator registry; based on the operator transformation mapping information, each first operator is traversed to determine the second operator that has a mapping relationship with the first operator.
[0096] In some embodiments, the second operator is a custom operator derived from the Halide language.
[0097] Next, each first operator in the initial computation graph is converted into a matching second operator; the inputs and outputs of each converted second operator are traversed according to the execution relationship between the converted second operators to identify the second operator group with input-output relationship from the converted second operators; the second operators in the second operator group are fused to generate a fusion operator for computation graph update processing to obtain the target computation graph adapted to MLIR.
[0098] Finally, the target computation graph is compiled based on MLIR to generate the target program code corresponding to the neural network.
[0099] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0100] Based on the same inventive concept, this application also provides an operator-based compilation apparatus for implementing the operator-based compilation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more operator-based compilation apparatus embodiments provided below can be found in the limitations of the operator-based compilation method described above, and will not be repeated here.
[0101] In some embodiments, such as Figure 5 As shown, an operator-based compilation device is provided, including: an acquisition module 502, a matching module 504, a conversion module 506, and a compilation module 508, wherein:
[0102] The acquisition module 502 is used to acquire the initial computation graph corresponding to the neural network; the initial computation graph includes multiple first operators; the first operators are operators defined based on the first programming language supported by the compiler optimization framework;
[0103] The matching module 504 is used to determine the second operator that matches each first operator based on the preset operator transformation mapping information; the second operator is a custom operator obtained according to the second programming language.
[0104] The conversion module 506 is used to convert each first operator in the initial computation graph into a matching second operator for computation graph update processing, so as to obtain a target computation graph that is compatible with the compilation optimization framework.
[0105] Compiler module 508 is used to compile the target computation graph based on the compilation optimization framework to generate the target program code corresponding to the neural network.
[0106] In the aforementioned operator-based compilation device, an initial computation graph corresponding to the neural network is obtained. This initial computation graph includes multiple first operators, which are operators defined using a first programming language supported by the compilation optimization framework. Second operators matching each first operator are determined based on preset operator transformation mapping information. These second operators are custom-defined using a second programming language. Each first operator in the initial computation graph is converted into a matching second operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework. The target computation graph is then compiled using the compilation optimization framework to generate the target program code corresponding to the neural network. In other words, this application integrates the second operators, custom-defined using a second programming language, into the compilation optimization framework ecosystem as independent operators, replacing the first operators defined using the first programming language supported by the framework. This allows the second operators to fully benefit from operator fusion and other compilation optimization operations, thereby improving the compilation optimization effect.
[0107] In some embodiments, the matching module 504 is further configured to obtain operator conversion mapping information between each first operator and each second operator from a preset operator registry; and to traverse each first operator based on the operator conversion mapping information to determine the second operator that has a mapping relationship with the first operator.
[0108] In some embodiments, the conversion module 506 includes an operator conversion unit, an operator matching unit, and an update unit. The operator conversion unit is used to convert each first operator in the initial computation graph into a matching second operator; the operator matching unit is used to determine the second operator to be fused from the converted second operators according to the execution relationship between them; the update unit is used to fuse the second operators to be fused to generate a fusion operator, so as to perform computation graph update processing and obtain a target computation graph adapted to the compilation optimization framework.
[0109] In some embodiments, the operator matching unit is further configured to traverse the inputs and outputs of each transformed second operator based on the execution relationship, so as to identify a group of second operators with input-output relationships from the transformed second operators; the outputs of some second operators in the group of second operators are the inputs of other second operators.
[0110] In some embodiments, the update unit is further configured to fuse the second operators in the second operator group to generate a fusion operator for computation graph update processing, thereby obtaining a target computation graph adapted to the compilation optimization framework.
[0111] In some embodiments, the second operator is obtained after generating at least one of the following custom information based on the second programming language: custom input and output information of the second operator; custom algorithm information of the second operator; and custom scheduling information of the second operator.
[0112] In some embodiments, the custom input and output information of the second operator includes at least one of scalar input information, image input information, and operator result input information; the scalar input information is defined based on the fixed-point data type and floating-point data type of the second programming language; the image input information is defined based on the image data storage object of the second programming language; and the operator result input information is used to define the binding relationship between the inputs and outputs of multiple second operators.
[0113] In some embodiments, the algorithm customization information of the second operator includes information on the declaration and definition of various variables of function types in the second programming language.
[0114] In some embodiments, the custom scheduling information for the second operator includes at least one of a first scheduling mode, a second scheduling mode, and a third scheduling mode. The first scheduling mode is used to indicate that when a non-current second operator uses the output of the current second operator as input, the computation of the current second operator is inlined; the second scheduling mode is used to indicate that when a non-current second operator uses the output of the current second operator as input and the non-current second operator is scheduled based on each computation partition, the non-current second operator is computed only after the computation of the current second operator corresponding to the computation partition of the non-current second operator is completed; the third scheduling mode is used to indicate that the non-current second operator is computed only after the computation of the current second operator is completed.
[0115] Each module in the aforementioned operator-based compiler can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processor in a computer device, or stored in software within the memory of the computer device, so that the processor can invoke and execute the operations corresponding to each module.
[0116] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the compiler optimization framework, the first operator, and the second operator. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements an operator-based compilation method.
[0117] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0118] In some embodiments, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0119] In some embodiments, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0120] In some embodiments, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0121] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. An operator-based compilation method, characterized in that, include: Obtain the initial computation graph corresponding to the neural network; The initial computation graph includes multiple first operators; The first operator is an operator defined based on a first programming language supported by a compiler optimization framework; Obtain operator conversion mapping information between each of the first operators and each of the second operators from the preset operator registry; Based on the operator transformation mapping information, each of the first operators is traversed to determine the second operator that has a mapping relationship with the first operator; the second operator is a custom operator obtained according to a second programming language other than the first programming language; Each of the first operators in the initial computation graph is converted into a matching second operator; based on the execution relationship between the converted second operators, the second operator to be fused is determined from the converted second operators. The second operator to be fused is fused to generate a fusion operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework; the second programming language has a built-in just-in-time compiler or a static early-stage compiler. The just-in-time compiler compiles, optimizes, and executes the target computation graph represented by the neural network at runtime; The static early compiler compiles the target computation graph to generate the corresponding binary library file, providing callable C or C++ interfaces for user programs; The target computation graph is compiled based on the aforementioned compilation optimization framework to generate the target program code corresponding to the neural network.
2. The method according to claim 1, characterized in that, The second programming language is Halide.
3. The method according to claim 1, characterized in that, The compilation optimization framework includes at least one of a multi-level intermediate representation framework or an end-to-end deep learning compiler.
4. The method according to claim 1, characterized in that, The step of determining the second operator to be fused from the converted second operators based on the execution relationship between the converted second operators includes: Based on the execution relationship, the inputs and outputs of each of the transformed second operators are traversed to identify a group of second operators with input-output relationships from the transformed second operators; the outputs of some second operators in the second operator group are the inputs of other second operators; The step of fusing the second operator to be fused to generate a fusion operator for computation graph update processing, to obtain a target computation graph adapted to the compilation optimization framework, includes: The second operator in the second operator group is fused to generate a fusion operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework.
5. The method according to any one of claims 1 to 4, characterized in that, The second operator is obtained by generating at least one of the following custom information based on the second programming language: Custom input and output information for the second operator; The algorithm-defined information for the second operator; Custom scheduling information for the second operator.
6. The method according to claim 5, characterized in that, The custom input and output information of the second operator includes at least one of scalar input information, image input information, and operator result input information; the scalar input information is defined based on the fixed-point data type and floating-point data type of the second programming language; the image input information is defined based on the image data storage object of the second programming language; and the operator result input information is used to define the binding relationship between the inputs and outputs of multiple second operators.
7. The method according to claim 5, characterized in that, The algorithm-defined information for the second operator includes information on the declaration and definition of various variables of the function type in the second programming language.
8. The method according to claim 5, characterized in that, The custom scheduling information of the second operator includes at least one of the first scheduling mode, the second scheduling mode, and the third scheduling mode; The first scheduling mode is used to indicate that the computation of the current second operator is inlined when the output of the current second operator is used as input, and the second second operator is not currently used as input. The second scheduling mode is used to indicate that when the non-current second operator uses the output of the current second operator as input and the non-current second operator is scheduled based on each computing partition, the non-current second operator is calculated only after the current second operator corresponding to the computing partition of the non-current second operator has been calculated. The third scheduling mode is used to indicate that calculations of non-current second operators should be performed only after the current second operator has been calculated.
9. An operator-based compiler device, characterized in that, include: The acquisition module is used to acquire the initial computation graph corresponding to the neural network. The initial computation graph includes multiple first operators; The first operator is an operator defined based on a first programming language supported by a compiler optimization framework; The matching module is used to obtain operator conversion mapping information between each first operator and each second operator from a preset operator registry; Based on the operator transformation mapping information, each of the first operators is traversed to determine the second operator that has a mapping relationship with the first operator; the second operator is a custom operator obtained according to a second programming language other than the first programming language; A conversion module is used to convert each of the first operators in the initial computation graph into a matching second operator; and to determine the second operator to be fused from the converted second operators based on the execution relationship between them. The second operator to be fused is fused to generate a fusion operator for computation graph update processing, resulting in a target computation graph adapted to the compilation optimization framework; the second programming language has a built-in just-in-time compiler or a static early-stage compiler. The just-in-time compiler compiles, optimizes, and executes the target computation graph represented by the neural network at runtime; The static early compiler compiles the target computation graph to generate the corresponding binary library file, providing callable C or C++ interfaces for user programs; The compilation module is used to compile the target computation graph based on the compilation optimization framework to generate the target program code corresponding to the neural network.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.