Compilation optimization method and apparatus based on neural network model, and electronic device

Abstract

The present application relates to the technical field of artificial intelligence, and in particular to a compilation optimization method and apparatus based on a neural network model, and an electronic device. The compilation optimization method comprises: acquiring, from a model file to be optimized, a computational graph of a neural network model; performing derivation processing on operators in the computational graph, so as to obtain an initial derivative population which comprises several derivative computational graphs, and storing the initial derivative population into a derivative population set; performing screening on the derivative population set, so as to obtain an optimal derivative computational graph; and generating an instruction code according to the topological structure of the optimal derivative computational graph, and writing the instruction code into said model file. By means of the compilation optimization method based on a neural network model provided in the present application, after a derivative population which comprises several derivative computational graphs is obtained by means of derivation, a derivative computational graph having optimal performance is selected, and thus the reasoning performance of a neural network model can be improved.

Description

Compilation optimization method, device and electronic equipment based on neural network model

[0001] This application claims priority to the Chinese patent application filed with the Patent Office of China on June 5, 2023, with application number 202310662959.0, and entitled “A compilation optimization method, device and electronic device based on a neural network model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a compilation optimization method, device, and electronic device based on a neural network model. Background Art

[0003] In recent years, artificial intelligence technologies, centered around machine learning and deep learning, have experienced rapid development, with deep neural networks widely used across industries. These applications have spawned many new models, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTMs), generative adversarial networks (GANs), and graph neural networks (GNNs), and have also spurred the emergence of deep learning frameworks such as TensorFlow, PyTorch, MXNet, and Caffe.

[0004] However, for deep learning algorithms to be truly practical, they must be deployed on end-user hardware. Compilers are crucial for deploying models trained with different deep learning frameworks to different end-user hardware.

[0005] Taking the neural network compiler TVM as an example, TVM implements a unified software stack for different deep learning frameworks and hardware platforms, deploying models from different deep learning frameworks onto hardware platforms in the most efficient manner possible. TVM's AutoScheduler technology implements multi-level abstraction of the computational graph of neural network models, leveraging evolutionary search and machine learning-based cost models for automated performance tuning, primarily for single operator performance.

[0006] Summary of the Invention

[0007] In view of this, the embodiments of the present application provide a compilation optimization method, device and electronic device based on a neural network model to improve the performance tuning of the overall operator of the neural network model.

[0008] In the first aspect, an embodiment of the present application provides a compilation optimization method based on a neural network model, including: obtaining a computational graph of the neural network model from a model file to be optimized; performing derivative processing on the operators in the computational graph to obtain an initial derivative population including several derivative computational graphs, and storing the initial derivative population in a derivative population cluster; screening the optimal derivative computational graph from the derivative population cluster; generating instruction code according to the topological structure of the optimal derivative computational graph, and writing it into the model file.

[0009] This embodiment obtains a derivative population including several derivative computation graphs through derivation, and then screens out the derivative computation graph with the best performance, which can improve the reasoning performance of the neural network model.

[0010] Furthermore, the compilation optimization method performs derivative processing on the operators in the computation graph to obtain an initial derivative population consisting of several derivative computation graphs. Before storing the initial derivative population in the derivative population cluster, it also includes: fusing the operators in the computation graph to obtain a fused computation graph. This can improve the optimization efficiency of the neural network model.

[0011] Furthermore, the compilation optimization method performs derivative processing on the operators in the computation graph to obtain an initial derivative population comprising several derivative computation graphs, and stores the initial derivative population in a derivative population cluster. This includes topologically sorting the computation graph, performing derivative processing on the operators in the computation graph according to a derivative rule, and obtaining an initial derivative population comprising several derivative computation graphs. Derivative computation graphs in the initial derivative population are mutated to obtain offspring derivative populations, and the initial and offspring derivative populations are stored in the derivative population cluster. This method can expand the derivative computation graphs of a wider variety of neural network models and select computation graphs with higher performance as much as possible.

[0012] In the second aspect, an embodiment of the present application provides a compilation optimization device based on a neural network model, including: an acquisition module for acquiring a computational graph of a neural network model from a model file to be optimized; a derivation module for performing derivation processing on operators in the computational graph to obtain an initial derivative population including several derivative computational graphs, and storing the initial derivative population into a derivative population cluster; a screening module for screening the optimal derivative computational graph from the derivative population cluster; and a writing module for generating instruction code according to the topological structure of the optimal derivative computational graph and writing it into the model file.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the compilation optimization method based on the neural network model as described in the embodiment of the first aspect is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the compilation optimization method based on the neural network model as described in the embodiment of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the compilation optimization method based on the neural network model as described in the embodiment of the first aspect.

[0016] It should be understood that the beneficial effects of the embodiments of the second to fifth aspects can be found in the relevant description of the embodiment of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] FIG1 is a schematic diagram of an implementation flow of a compilation optimization method based on a neural network model provided in one embodiment of the present application;

[0019] FIG2 is a schematic diagram of an implementation flow of a compilation optimization method based on a neural network model provided in another embodiment of the present application;

[0020] FIG3A is a schematic diagram of an operator fusion provided in an embodiment of the present application;

[0021] FIG3B is a schematic diagram of an operator fusion provided by another embodiment of the present application;

[0022] FIG3C is a schematic diagram of an operator fusion provided by another embodiment of the present application;

[0023] FIG3D is a schematic diagram of an operator fusion provided by another embodiment of the present application;

[0024] FIG4 is a schematic diagram of an implementation flow of step S120 in a compilation optimization method based on a neural network model according to an embodiment of the present application;

[0025] FIG5 is a schematic diagram of the structure of a calculation graph provided in an embodiment of the present application;

[0026] FIG6 is a schematic diagram of an implementation flow of a derivative process of a computation graph provided in an embodiment of the present application;

[0027] FIG7 is a schematic diagram of the structure of a derivative calculation graph provided in one embodiment of the present application;

[0028] FIG8 is a schematic diagram of the structure of a compilation optimization device based on a neural network model provided in one embodiment of the present application;

[0029] FIG9 is a schematic structural diagram of a compilation optimization device based on a neural network model provided in another embodiment of the present application;

[0030] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0032] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0034] In addition, in the description of this application, "a plurality of" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0035] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0036] Currently, the AutoScheduler technology in the neural network compiler TVM mainly optimizes the performance of single operators. It does not utilize on-chip memory such as static random-access memory (SRAM) to automatically optimize the overall performance of the computation graph. Moreover, it is difficult to apply to embedded neural network processors (NPUs) with coarse-grained instructions.

[0037] In order to solve the technical problems of related technologies, the embodiments of the present application provide a compilation optimization method, device and electronic device based on a neural network model to improve the reasoning performance of the neural network model, especially to improve the reasoning performance of the neural network model for image processing on an embedded neural network processor.

[0038] FIG1 is a schematic diagram illustrating an implementation flow of a neural network model-based compilation optimization method provided in one embodiment of the present application. The neural network model-based compilation optimization method in this embodiment can be executed by an electronic device. Electronic devices include, but are not limited to, computers, tablet computers, servers, etc. Servers include, but are not limited to, standalone servers or cloud servers. As shown in FIG1 , the neural network model-based compilation optimization method can include steps S110 to S140.

[0039] S110, obtaining a computational graph of the neural network model from the model file to be optimized.

[0040] The model file is the data display / storage format of the neural network model in the electronic device. The compiled model file can be deployed on the hardware platform. Neural network models can include convolutional neural networks, recurrent neural networks, long short-term memory networks, generative adversarial networks, and graph neural networks. In the computational graph of a neural network model, each node is an operator, and the mathematical operation function corresponding to the operator is the functional relationship between the node's input data and output data.

[0041] S120, performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster.

[0042] In some embodiments, S120 specifically includes: topologically sorting the computation graph, performing derivative processing on the operators in the computation graph according to the derivative rule, and obtaining an initial derivative population including several derivative computation graphs.

[0043] Specifically, the derived rule is the maximum amount of data that the hardware platform can run. Considering the limited space of hardware platforms running neural network models, it is necessary to determine whether the amount of data operated by a single operator exceeds the maximum amount of data that the hardware platform can run.

[0044] Among them, derivative processing includes: selecting skip processing, tiling processing, and read / write (DMA Read+Write) processing for the current operator based on whether the amount of data operated by the current operator exceeds the maximum amount of data that the hardware platform can run, and whether the current operator can be multi-layered. It can be understood that skip processing: no processing is required for the current operator; tiling processing: the current operator generates a corresponding number of sub-operators according to the randomly selected number of tilings; read / write processing: because the amount of data operated by the current operator exceeds the maximum amount of data that the hardware platform can run, the operation process of the current operator needs to be transferred to other memory, so a direct memory access (DMA) read operator (DMA Read) is added before the current operator (i.e., on the input side), and a DMA write operator (DMA Write) is added after the current operator (i.e., on the output side).

[0045] In this embodiment, the segmentation process involves randomly segmenting an operator into several sub-operators. These sub-operators scale the input image so that the output consists of several sub-images with smaller resolutions. This is equivalent to an operator that originally computes one image at a time. After random segmentation, the resulting sub-operators evenly divide the image into several sub-image blocks for batched computation.

[0046] In this embodiment, the read and write processing is to add a read operator before the current operator operation and add a write operator after the current operator operation. The read operator can be a Direct Memory Access (DMA) read operator, and the write operator can be a DMA write operator. Adding a read operator before the current operator operation can move the input of the current operator to the on-chip memory, and the read processing of the on-chip memory is completed. After the on-chip memory operation is completed, the write processing of the on-chip memory outputs the data and moves it from the on-chip memory to the on-chip memory, thereby speeding up the overall operation speed of the operator.

[0047] On-chip memory can be static random-access memory (SRAM), while on-chip memory can be synchronous dynamic random-access memory (SDRAM), double-data-rate random-access memory (DDR), or bus-based dynamic random-access memory (RDRAM). Typical DRAM has only one asynchronous interface and uses asynchronous transmission. Asynchronous transmission technology is simple and relatively stable, but the data transmission rate is relatively slow. SDRAM has both asynchronous and synchronous interfaces. The synchronous interface requires a unified clock signal for both the sender and receiver, which is relatively complex and results in higher transmission rates and concurrency. SDRAM's synchronous interface is controlled by a unified clock signal, transmitting data once per clock cycle. DDR can transmit data twice per clock cycle, doubling the data transmission rate of SDRAM. In other words, DDR is SDRAM with a double data rate. SRAM, on the other hand, is a static access memory that does not require a refresh circuit to retain internally stored data, thereby increasing the overall computing speed of the operator with an increased number of read and write operators.

[0048] S130, screening out the optimal derivative computation graph from the derivative seed cluster.

[0049] In this embodiment, the derivative processing can be expanded to obtain several derivative calculation graphs. Different derivative calculation graphs have different reasoning performance. After screening, the optimal derivative calculation graph can be selected, so that the compiled neural network model has better reasoning performance.

[0050] S140: Generate instruction code according to the topological structure of the optimal derivative calculation graph and write it into the model file.

[0051] The topology of the derived computation graph determines how the instruction stream is generated. Specifically, each operator in the derived computation graph corresponds to an instruction, and the entire derived computation graph corresponds to the instruction stream, which includes DMA read and write instructions. Finally, the instruction stream is written to the model file, completing the compilation and optimization process.

[0052] It is understandable that the neural network model or its computational graph cannot directly implement the functions of the model, so the model needs to be compiled to generate a model file to store the neural network model in the electronic device in the form of data. The stored model file can be called by the processor, and when the processor calls the model file, it can implement the operation of the neural network model according to the instruction code therein.

[0053] The embodiment of the present application performs derivative processing on the operators in the complete computation graph, and uses evolutionary search, pre-trained performance models and other means to find the optimal operator segmentation method and the execution order of the operators. The input and output data of each operator in the computation graph can be stored on SRAM, and DMA operations can be executed in parallel with NPU instructions. Since the read and write speed and bandwidth of SRAM are better than DDR, the efficiency of data reading and writing is greatly improved, thereby improving the performance of the neural network model corresponding to the computation graph during inference.

[0054] Referring to FIG. 2 , in some other embodiments, step S111 is further included before step S120: fusing the operators in the computation graph to obtain a fused computation graph. Subsequent steps are executed based on the fused computation graph, which reduces the number of operators and thus improves the efficiency of encoding optimization. In this embodiment, any overlap with the embodiment described in FIG. 1 is not further described here.

[0055] It should be noted that due to the deep number of network layers, complex structure, and large number of generated operators in the neural network model, huge computing resources are required for operation. Fusion processing refers to analyzing the related basic operators in the model's calculation graph, and fusing multiple basic operators into combined operators under the condition that performance benefits can be obtained, so as to reduce the overhead between operator executions and improve the utilization of device computing resources, thereby optimizing execution time. Operator fusion mainly fuses operators with data dependencies on the calculation graph, that is, imitating the producer-consumer relationship: if the result of the producer operator can be directly transmitted to the consumer operator without the need for intermediate result representation, the two operators can be fused. In practical applications, the appropriate operator fusion strategy can be selected according to the specific framework, and there is no restriction here.

[0056] As a possible implementation, as shown in FIG3A , the convolution layer (Convolution), the batch normalization layer (BatchNorm, BN) and the quantization layer (Scale) can be fused, that is, the BN layer and the quantization layer are embedded in the calculation of the convolution layer. As another possible implementation, the convolution layer (Convolution), the BN layer and the activation function can be fused, that is, the BN layer and the activation function are embedded in the calculation of the convolution layer. As another possible implementation, as shown in FIG3B , the convolution layer (Convolution) and the activation function (ReLU) can be fused, that is, the activation function is embedded in the calculation of the convolution layer. As another possible implementation, as shown in FIG3C , the element-wise operation operator (Eltwise) and the activation function (ReLU) can be fused, that is, the activation function is embedded in the calculation of the element-wise operation operator. Eltwise operations include point multiplication (product), addition and subtraction (sum) and taking the maximum value (max). As another possible implementation, as shown in FIG3D , the concatenation operator (Concat) and the activation function (ReLU) after the convolution layer (Convolution) can be fused, that is, the activation function can be embedded in the calculation of the concatenation operator.

[0057] As a possible implementation, as shown in FIG4 , step S120 may include steps S121 to S122 .

[0058] S121, topologically sorting the computation graph, performing derivative processing on the operators in the computation graph according to the derivative rule, and obtaining an initial derivative population including several derivative computation graphs.

[0059] In which, the computation graph is topologically sorted so that there is a certain execution order during the operation of the computation graph. Based on the computation graph of the execution order, the corresponding derivative population can be obtained according to the derivation rule. The derivative processing includes selecting skip processing, split processing and / or read-write processing for the current operator. In this embodiment, the content repeated in the embodiment described in Figure 1 is not repeated here. In some embodiments, the derivative population may only include the initial derivative population. The difference between different derivative computation graphs in the initial derivative population is mainly that some operators therein have undergone different splits.

[0060] S122, performing mutation processing on the derivative computation graph in the initial derivative population to obtain a subsequent derivative population, and storing the initial derivative population and the subsequent derivative population into a derivative population cluster.

[0061] The mutation process involves adjusting the depth of the image input to the operator. Depth refers to the number of bits used to store each pixel in the image. Derivative processing is performed on the operators in the computation graph, where random block segmentation causes the corresponding operators to segment the input image. In step S122, the mutation process changes the depth of the image input to the operator. It can be understood that a computation graph is composed of several operators. After a derivation or mutation process, a new computation graph is obtained. By performing several different mutation processes on a derivative computation graph in the initial derivative population, several subsequent derivative computation graphs are obtained, forming a subsequent population. The initial derivative population includes several derivative computation graphs. After the corresponding mutation processes, each derivative computation graph corresponds to a subsequent derivative population. In this embodiment, the derivative population includes an initial derivative population and several subsequent derivative populations.

[0062] As another possible implementation, step S120 may also include: obtaining a computation graph of the neural network model from the model file to be optimized; randomly generating several execution orders, and performing derivative processing on the operators in the computation graph based on the different execution orders to obtain several derivative seed clusters. It will be understood that each execution order corresponds to a derivative seed cluster, and each derivative seed cluster includes several computation graphs. The optimal derivative computation graph is then screened from the derivative seed clusters.

[0063] In some embodiments, the structural diagram of the computation graph is shown in Figure 5. Operators B and C can have different orders of execution during operation, operators D and E can have different orders of execution during operation, operators F and G can have different orders of execution during operation, and operators H and I can also have different orders of execution during operation. Depending on the execution order, different computation graphs will ultimately be used during operation, and computation graphs with different execution orders will result in different derivative seed clusters. It can be understood that before the operators in the computation graph are derived, several execution orders of operators are randomly generated. This execution order is based on the computation graph execution of the neural network model. For example, in a neural network model including two parallel convolutions (a first convolution and a second convolution), executing the first convolution first and then the second convolution is a feasible first execution order, and executing the second convolution first and then the first convolution is another feasible second execution order. A first derivative seed cluster is generated based on the first execution order, and a second derivative seed cluster is generated based on the second execution order. The first and second derivative seed clusters are provided here as examples. In actual operation, the execution order may vary, resulting in different derivative seed clusters. By generating different execution orders, we can cover as many search possibilities as possible within a limited time and space, thereby selecting computational graphs with higher performance.

[0064] In some implementations, a simulated annealing algorithm or a genetic mutation algorithm is used to identify the optimal derivative computational graph for different computational graphs obtained using different execution orders and different derivative processing methods. The computational graph of the neural network model is obtained from the model file to be optimized; a random execution order is generated using the simulated annealing algorithm or the genetic mutation algorithm, and operators in the computational graph are subjected to derivative processing based on the current execution order to obtain a derivative population, from which an optimal solution is selected; another execution order is repeatedly obtained to obtain an optimal solution for the computational graph under that execution order; multiple optimal solutions are obtained and the optimal derivative computational graph is selected from them.

[0065] As one possible implementation, for a computation graph in a certain execution order, the computation graph is topologically sorted, and the operators in the computation graph are derived according to the derivation rules to obtain an initial derivative population comprising several derivative computation graphs. The derivative computation graphs in the initial derivative population are mutated to obtain a subsequent derivative population, and the initial and subsequent derivative populations are stored in a derivative population cluster. Step S130 includes: using the trained performance model to evaluate the performance of each derivative population in the derivative population cluster, obtaining a derivative population with the best performance, and then selecting the optimal derivative computation graph from the derivative population.

[0066] In other embodiments, simulated annealing or genetic mutation algorithms are used to obtain computational graphs with different execution orders, and the computational graphs with different execution orders are subjected to derivation and mutation processing to obtain multiple derivative clusters. Step S130 includes: using the trained performance model to evaluate the performance of the derivative clusters in any derivative cluster, obtaining a derivative cluster with the best performance in the corresponding derivative cluster, and then screening the optimal solution from the derivative cluster; repeating the above steps to obtain the optimal solutions corresponding to all derivative clusters, and then screening the optimal derivative computation graph from the multiple optimal solutions.

[0067] The performance model is a neural network model composed of several convolutional layers, pooling layers, or encoders and decoders. In some embodiments, the performance model is trained using operators labeled with performance metrics (such as runtimes under various instructions and configurations) as a training set to obtain a trained performance model. This allows the performance model to quickly output performance scores for each operator in the derived computation graph.

[0068] As a possible implementation, the derivation processing steps of the computation graph are shown in FIG6 , which may specifically include steps S1201 to S1208 .

[0069] S1201, topologically sorting the computation graph of the neural network model.

[0070] S1202: traverse all operators in sequence. Specifically, traverse each operator in the computation graph of the neural network model in sequence according to topological sorting.

[0071] S1203, determine whether the operation data volume of the current operator exceeds the maximum data volume that the hardware platform can run. If so, execute steps S1204 to S1206; if not, execute step S1207.

[0072] S1204: Add read and write operators before and after the current operator.

[0073] Specifically, step S1204 includes adding a DMA read operator (DMA Read) before the current operator (i.e., on the input side) and adding a DMA write operator (DMA Write) after the current operator (i.e., on the output side). The detailed description of the read and write operators is repeated in the above description of the embodiment of FIG1 and will not be repeated here.

[0074] S1205: Randomly split the current operator to generate a corresponding number of sub-operators.

[0075] Specifically, the division of the operator generates a corresponding number of sub-operators, and the number of sub-operators represents the number of slices of each dimension in each input tensor of the operator. As a non-limiting example, after the operator is randomly divided into sub-operators, the feature map of the input operator will be divided into several sub-image blocks in the height direction (h) and the width direction (w), and each sub-operator processes one sub-image block respectively. The number of divisions in the height direction and the width direction are random. For example, if the operator is divided into 4 sub-operators, the feature map is divided into 2 parts in the height direction and 2 parts in the width direction. In this way, the feature map is divided into 4 blocks. For another example, if the current operator is divided into 2 sub-operators, the feature map of the input operator is not divided in the height direction, but is divided into 2 parts in the width direction. In this way, the input feature map is divided into 2 blocks. One division corresponds to the generation of a derivative computation graph.

[0076] S1206, determine whether all operators have been traversed. If so, generate a derivative calculation graph; if not, return to step S1202 and traverse the next operator.

[0077] S1207, determine whether the current operator can be multi-layer fused, if so, execute step S1208; if not, return to execute step S1206 to traverse the next operator or complete the traversal.

[0078] Specifically, when an operator does not meet the fusion processing conditions, it cannot be fused in multiple layers, that is, the operator will not be split; when an operator meets the fusion processing conditions, it can be fused in multiple layers, that is, the operator will be split. If an operator does not meet the fusion processing conditions, it means that there is no dependency between the operator and other operators, and since the amount of computing data of the operator does not exceed the maximum amount of data that the hardware platform can run, the operator is an optimal solution. If the operator is still split, more calculation graphs will be obtained, which will reduce the efficiency of screening the optimal derivative calculation graph from the derived population. In this embodiment, skipping the operator whose computing data amount does not exceed the maximum amount of data that the hardware platform can run and does not meet the fusion processing conditions can improve compilation efficiency.

[0079] S1208: Randomly split the current operator to generate a corresponding number of sub-operators. After step S1208, step S1206 is executed again.

[0080] In some embodiments, in order to make the topological sorting more efficient and have a certain degree of stability, step S1208 splits the current operator according to the splitting method of the previous operator, that is, generates an equal number of sub-operators according to the number of splits of the previous operator.

[0081] In a specific embodiment, step S130, screening out the optimal derivative computation graph from the derivative seed cluster, includes: performing performance evaluation on the operators in the derivative computation graph one by one to obtain the performance score of the operator, and then obtaining the comprehensive score of the derivative computation graph; selecting the one with the highest comprehensive score from several derivative computation graphs as the optimal derivative computation graph.

[0082] The performance of each operator in the derivative computation graph is evaluated one by one to obtain a performance score for the operator, which can be calculated using a trained performance model. The performance model can be a cost model. The performance model is also a neural network model composed of several convolutional layers, pooling layers, or encoders and decoders. Operators labeled with performance indicators (such as the running time under various instructions and configurations) are used as a training set to train the performance model. The trained performance model is obtained so that the performance model can quickly output the performance score of each operator in the derivative computation graph.

[0083] In other embodiments, an evolutionary search algorithm can be used to determine the optimal derivative computation graph based on multiple derivative computation graphs. This method simulates the natural evolutionary process to search for the optimal solution. Its main features are direct operations on structural objects, without the constraints of derivatives or function continuity; inherent implicit parallelism and improved global optimization capabilities; and a probabilistic optimization method that automatically acquires and guides the optimized search space without the need for specific rules, adaptively adjusting the search direction.

[0084] As shown in Figure 7, this is a schematic diagram of a derivative process generated according to step S120 provided by an embodiment of the present application, wherein the upper figure of Figure 7 is a calculation diagram before the derivative process, and the lower figure of Figure 7 is a schematic diagram after the derivative process. The amount of operating data of the operator in Figure 7 exceeds the maximum amount of data that the hardware platform can run, so read and write operators are added before and after the operator, specifically DMA Read and DMA Write, and the operator is randomly divided to obtain n sub-operators. Before the derivative process, the operator processing feature graph runs on the DDR. After the derivative process, DMA Read and DMA Write will allow the SRAM to share the amount of calculation on the DDR. In addition, by reusing the calculation address, as many operators as possible can be executed in the limited address of the on-chip memory, thereby improving processing efficiency.

[0085] It can be understood that step S121 and step S128 can be summarized as: topologically sorting the computation graph; traversing all operators, adding read-write operators to operators that need to be block-split for random block splitting, and performing random block splitting for operators with multi-layer fusion to obtain several sub-operators; after traversing all operators, generating a derivative computation graph based on the several sub-operators.

[0086] In a compilation optimization method based on a neural network model provided in this embodiment, an operator is divided into several sub-operators, that is, the feature map of the input operator is divided into several sub-graphs, and the optimal division method is found. The input and output data of each sub-operator can be completely stored in an on-chip memory such as SRAM, and the DMA operation can be executed in parallel with the NPU instructions. Since the read and write speed and bandwidth of SRAM are better than those of off-chip memories such as DDR, the efficiency of data reading and writing is greatly improved, thereby improving the inference performance of the calculation graph on the NPU.

[0087] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0088] One embodiment of the present application further provides a compilation optimization device based on a neural network model. For details not described in detail in the compilation optimization device, please refer to the relevant description in the aforementioned compilation optimization method embodiment.

[0089] 8 is a schematic diagram of a neural network model-based compilation optimization device according to an embodiment of the present invention. The compilation optimization device includes an acquisition module 71 , a derivation module 72 , a screening module 73 , and a writing module 74 .

[0090] Among them, the acquisition module 71 is used to obtain the calculation graph of the neural network model; the derivation module 72 is used to perform derivation processing on the operators in the calculation graph to obtain an initial derivative population including several derivative calculation graphs, and store the initial derivative population in a derivative population cluster; the screening module 73 is used to screen the optimal derivative calculation graph from the derivative population cluster; the writing module 74 is used to generate instruction code according to the topological structure of the optimal derivative calculation graph and write it into the model file.

[0091] In some embodiments, based on the embodiment shown in FIG8 , an improvement is made. As shown in FIG9 , the compilation optimization device based on the neural network model further includes a fusion module 75. The fusion module 75 is used to perform operator fusion on the computation graph of the neural network model according to the fusion rule to obtain a fused computation graph.

[0092] An embodiment of the present application also provides an electronic device, as shown in FIG10 , which may include one or more processors 100 (only one is shown in FIG10 ), a memory 101, and a computer program 102 stored in the memory 101 and executable on the one or more processors 100, for example, a compilation optimization program. When one or more processors 100 execute the computer program 102, the various steps in the embodiment of the compilation optimization method based on the neural network model can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the embodiment of the compilation optimization device based on the neural network model can be implemented, which is not limited here.

[0093] Those skilled in the art will appreciate that FIG10 is merely an example of an electronic device and does not limit the scope of the electronic device. An electronic device may include more or fewer components than shown, or may combine certain components or different components. For example, an electronic device may also include input and output devices, network access devices, buses, etc.

[0094] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0095] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the electronic device. The memory 101 may also be used to temporarily store data that has been output or is about to be output.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in the embodiment of the compilation optimization method based on the neural network model.

[0098] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the embodiment of the compilation optimization method based on a neural network model.

[0099] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0100] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0103] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0104] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned method embodiment, and the computer program that can be completed by instructing the relevant hardware through a computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0105] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A compilation optimization method based on a neural network model, characterized in that: include: Get the computational graph of the neural network model from the model file to be optimized; Performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster; Filtering and obtaining an optimal derivative computation graph from the derivative seed cluster; Instruction codes are generated according to the topological structure of the optimal derivative computation graph and written into the model file.

2. The compilation optimization method according to claim 1, characterized in that: The step of performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster, further includes: The operators in the computation graph are fused to obtain a fused computation graph.

3. The compilation optimization method according to claim 1, characterized in that: The step of performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster includes: The computation graph is topologically sorted, and operators in the computation graph are derived according to a derivation rule to obtain an initial derived population including a plurality of derived computation graphs, and the initial derived population is stored in a derived population cluster.

4. The compilation optimization method according to claim 3, characterized in that: Also includes: The derivative computation graph in the initial derivative population is mutated to obtain a subsequent derivative population, and the subsequent derivative population is stored in a derivative population cluster.

5. The compilation optimization method according to claim 3, characterized in that: The derivative process includes random block segmentation and read and write operator addition.

6. The compilation optimization method according to claim 1, characterized in that: The step of performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster includes: Several execution orders are randomly generated, and operators in the computation graph are derived based on the different execution orders to obtain several initial derived populations.

7. The compilation optimization method according to claim 1, characterized in that: The step of screening the derived species cluster to obtain the optimal derived computation graph comprises: Performing performance evaluation on the operators in the derived computation graph one by one to obtain performance scores of the operators, and then obtaining a comprehensive score of the derived computation graph; The derivative calculation graph with the highest comprehensive score is selected from the plurality of derivative calculation graphs as the optimal derivative calculation graph.

8. A compilation optimization device based on a neural network model, characterized in that: include: An acquisition module is used to obtain the computational graph of the neural network model; A derivative module, used for performing derivative processing on the operators in the computation graph to obtain an initial derivative population including a plurality of derivative computation graphs, and storing the initial derivative population in a derivative population cluster; A screening module, used for screening the best derivative computation graph from the derivative seed cluster; A writing module is used to generate instruction codes according to the topological structure of the optimal derivative calculation graph and write the instruction codes into the model file.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the compilation optimization method based on the neural network model as described in any one of claims 1 to 7 is implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the compilation optimization method based on the neural network model as described in any one of claims 1 to 7 is implemented.

Images