Operator fusion method and system for image recognition deep neural network

Abstract

The application discloses an operator fusion method and system for image recognition deep neural networks, and comprises the following steps: dividing the calculation graph of an input deep neural network model into calculation subgraphs; randomly constructing multiple candidate fusion schemes for each calculation subgraph, testing and selecting the best fusion scheme on hardware; the fusion of two continuous complex operators in the candidate fusion scheme is realized by recycling and recombining the latter complex operator, unifying the external loops of the two complex operators, and then directly combining the external loops based on the dimensions and shapes of the intermediate result tensors; and combining the best fusion schemes of the calculation subgraphs to obtain the best fusion execution scheme of the deep neural network model. The application can solve the problem that complex operators in the image recognition deep neural network cannot be fused, can improve the execution efficiency of the image recognition task through the fusion of complex operators and the fusion of the whole neural network, and can relieve the problem of tight cache resources when the image recognition task is deployed on hardware.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of program code optimization, in particular to an operator fusion method and system for image recognition deep neural networks. BACKGROUND

[0002] With the continuous development of machine learning, network models represented by deep neural networks (DNN) have made significant progress in image recognition applications. In order to meet the demand for higher prediction performance, the depth of neural networks for image recognition is increasing, and the number of network layers of some models even reaches thousands of layers. The continuous increase in scale brings higher computing power and greater storage pressure challenges to existing hardware facilities. Deep learning compiler, as a kind of compilation optimization means in recent years, provides a new optimization direction for the deployment and execution of deep neural network models for image recognition. Compared with traditional manual compilation, the input of deep learning compiler is a unified computation graph representation; deep learning compiler contains different kinds of computation graph optimization and has automatic tuning means; deep learning compiler can be connected to different backends for hardware deployment. For deep learning compiler, through operator fusion optimization, the efficiency of neural networks for image recognition is improved on related hardware, and the problem of hardware cache tension is reduced. As a main optimization means in computation graph optimization, the main purpose of operator fusion is to combine multiple small operators into one large operator. In the case of unchanged existing hardware resources, operator fusion optimization relieves the storage pressure of deep neural network models during execution and improves efficiency.

[0003] At present, the research based on operator fusion mainly includes two aspects: the mechanism of operator fusion and the strategy of model fusion. The mechanism of operator fusion is to determine whether two adjacent operators in the neural network can be fused and which fusion method to adopt; the fusion strategy of the model is to determine whether the operators in the entire model are fused and how to fuse. For most neural networks for image recognition, the operators contained in the network can be divided into two categories: simple operators and complex operators. Simple operators (element-wise type operators) are operators whose input and output tensors have a one-to-one mapping relationship. Complex operators (non-element-wise type operators) are operators whose input and output tensors have inter-layer dependencies, and each element is not a one-to-one mapping relationship (i.e., the mapping relationship is complex), but a many-to-one, many-to-many or one-to-many function mapping, as shown in Table 1.

[0004] Table 1: Examples of simple operators and complex operators.

[0005]

[0006] The basic idea of the current fusion method is the same as the traditional loop fusion by optimizing the compiler. The traditional loop fusion method is to combine multiple loop calculations into one loop calculation. In existing neural network compilation and execution frameworks, only two fusion modes are supported: one is the fusion mode of multiple consecutive simple operators; the other is the fusion mode of simple operators and complex operators. The fusion between two complex operators is considered to be algorithmically complex and has no benefit. However, in existing image recognition neural networks, there are a large number of adjacent complex operators. This greatly reduces the opportunity for fusion in the entire neural network. Therefore, most mainstream image recognition-oriented DNN inference frameworks only support the fusion of simple operators with other operators, while the fusion between complex operators is still not supported. In terms of operator fusion mechanism, in existing neural network compilation and execution frameworks, the operator fusion mechanism is subject to many restrictions, only supporting specific operator fusion modes and specific operator type fusion modes. Therefore, the existing fusion mechanism is limited, resulting in a large number of adjacent complex operators in the neural network that cannot be fused, and the reduction of fusion opportunities limits the efficiency of image recognition task execution optimization. SUMMARY

[0007] The technical problem to be solved by the present application is to provide an operator fusion method and system for image recognition deep neural networks, which can solve the problem of complex operator fusion in image recognition deep neural networks, and improve the execution efficiency of image recognition tasks through complex operator fusion and entire neural network fusion, and alleviate the problem of cache resource shortage when deploying image recognition tasks on hardware.

[0008] To solve the above technical problems, the technical scheme adopted by the present application is:

[0009] An operator fusion method for image recognition deep neural networks, comprising:

[0010] S101, dividing the computation graph of the input deep neural network model into computation subgraphs;

[0011] S102, respectively constructing a plurality of candidate fusion schemes for each computation subgraph, testing the candidate fusion schemes on hardware to obtain execution time, and selecting the candidate fusion scheme with the shortest execution time as the best fusion scheme; the operator fusion type in the candidate fusion scheme includes fusion of two consecutive simple operators, fusion of two consecutive complex operators, and fusion of consecutive simple and complex operators, and the fusion of two consecutive complex operators is to reorganize the latter complex operator in cycle to unify the external cycles of the two complex operators into a dimension and shape based on the intermediate result tensor, and then directly combine the external cycles;

[0012] S103, combine the optimal fusion schemes of each calculation sub-graph to obtain an optimal fusion execution scheme of the deep neural network model.

[0013] Optionally, the loop restructuring of the latter complex operator and the direct merging of the outer loop of the two complex operators after the loop restructuring of the latter complex operator include:

[0014] S201, for the two continuous complex operators to be fused, keeping the calculation strategy of the former complex operator unchanged as the output tensor-based calculation strategy, modifying the calculation strategy of the latter complex operator as the input tensor-based calculation strategy through loop restructuring, the mapping relationship in the input tensor-based calculation strategy being a single input scalar to the entire output tensor, and the outer loop being based on the input tensor of the operator, so that the outer loops of the calculation strategies of the two complex operators are unified as the dimensions and shapes of the intermediate result tensors of the two complex operators.

[0015] S202, directly merging the outer loops of the two complex operators after the loop restructuring of the calculation strategy and the unification of the outer loops as the dimensions and shapes of the intermediate result tensors of the two complex operators, to obtain the fused code.

[0016] Optionally, after step S202, the method further includes correctness verification for the fused code: taking the calculation results of the two complex operators before the fusion as the reference values, comparing the calculation results of the fused code with the reference values, if the two are the same, determining that the correctness verification passes, otherwise, determining that the correctness verification fails, then readjusting the operator fusion manner in the candidate fusion scheme and performing the correctness verification again until the fusion of the two continuous complex operators in the candidate fusion scheme all pass the correctness verification.

[0017] Optionally, when testing the candidate fusion scheme on the hardware, before testing the candidate fusion scheme, the method further includes fusion code optimization on the base code of the candidate fusion scheme, the fusion code optimization including part or all of parallel optimization, loop unrolling, and vectorization optimization.

[0018] Optionally, when testing the candidate fusion scheme on the hardware, before testing the candidate fusion scheme, the method further includes sparse optimization on the complex operators in the candidate fusion scheme: modifying the activation function between the two complex operators in the candidate fusion scheme into a control calculation execution condition, through the control calculation execution condition, making the latter complex operator part perform function mapping on an intermediate data scalar obtained by only the former complex operator part when the intermediate data scalar meets the activation condition of the activation function, and otherwise, the latter complex operator part cancels the function mapping on the intermediate data scalar.

[0019] Optionally, when testing the candidate fusion scheme on hardware, before testing the candidate fusion scheme, further comprising sub-operator fusion optimization of complex operators in the candidate fusion scheme: detecting complex operators with sub-operators in the candidate fusion scheme, for each sub-operator of the complex operator, if the sub-operator can be fused with the sub-operator of the adjacent other complex operator or the simple operator, then fusing the sub-operator with the sub-operator of the adjacent other complex operator or the simple operator.

[0020] Optionally, in step S102, a plurality of candidate fusion schemes are randomly generated for each calculation sub-graph, the candidate fusion schemes are tested on hardware to obtain execution time, and the candidate fusion scheme with the shortest execution time is selected as the optimal fusion scheme, and the processing of each calculation sub-graph includes:

[0021] S301, based on a predefined basic population quantity M, M initial candidate fusion schemes are randomly generated for the calculation sub-graph, an initial candidate fusion scheme test set is obtained, and each candidate fusion scheme in the initial candidate fusion scheme test set is tested on hardware to obtain execution time;

[0022] S302, updating the candidate fusion schemes in the candidate fusion scheme test set;

[0023] S303, testing the candidate fusion schemes in the updated candidate fusion scheme test set on hardware to obtain execution time;

[0024] S304, judging whether the multiple candidate fusion schemes with shorter execution time in the updated candidate fusion scheme test set remain unchanged, if the candidate fusion schemes remain stable and unchanged, it is determined that the candidate fusion scheme test set has been stable, a candidate fusion scheme with the shortest execution time is selected from the stable candidate fusion scheme test set as the optimal fusion scheme of the calculation sub-graph, and the processing of the next calculation sub-graph or the exit is entered; otherwise, step S302 is jumped to.

[0025] Optionally, in step S302, updating the candidate fusion schemes in the candidate fusion scheme test set includes: sorting the candidate fusion schemes in the candidate fusion scheme test set according to execution time, deleting part of the candidate fusion schemes with longer execution time; for the remaining part of the candidate fusion schemes in the candidate fusion scheme test set, a new candidate fusion scheme is generated by using a genetic algorithm according to a predefined mutation probability P, and the new candidate fusion scheme and the remaining part of the candidate fusion schemes constitute the updated candidate fusion scheme test set.

[0026] In addition, the application also provides an operator fusion system for an image recognition deep neural network, comprising a microprocessor and a memory connected to each other, the microprocessor is programmed or configured to execute the operator fusion method for the image recognition deep neural network.

[0027] Further, the application also provides a computer readable storage medium, wherein a computer program is stored in the computer readable storage medium, and the computer program is used for programming or configuring a microprocessor to execute the operator fusion method of the image recognition oriented deep neural network.

[0028] Compared with the prior art, the application mainly has the following advantages:

[0029] 1. The application includes fusion of two consecutive complex operators, and the outer loops of the two complex operators are unified based on the dimensions and shapes of the intermediate result tensors by cyclically reorganizing the latter complex operator, and then the outer loops are directly combined, thereby solving the problem that complex operators cannot be fused in the image recognition deep neural network, improving the execution efficiency of the image recognition task through fusion of complex operators, and relieving the problem of cache resource shortage when the image recognition task is deployed on hardware.

[0030] 2. The application includes randomly constructing multiple candidate fusion schemes for each subgraph, testing the candidate fusion schemes on hardware to obtain execution time, and selecting the candidate fusion scheme with the shortest execution time as the optimal fusion scheme, wherein the operator fusion types in the candidate fusion scheme include fusion of two consecutive simple operators, fusion of two consecutive complex operators, and fusion of a simple operator and a complex operator, thereby enabling all-round operator fusion of the deep neural network model to automatically generate the optimal neural network operator fusion scheme, improving the execution efficiency of the image recognition neural network and reducing the burden of the developer, and further improving the execution efficiency of the image recognition task and relieving the problem of cache resource shortage when the image recognition task is deployed on hardware. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 FIG. 1 is a basic flowchart of the method of the first embodiment of the application.

[0032] Figure 2 FIG. 2 is a flowchart of the complex operator fusion and subsequent optimization and testing of the candidate fusion scheme in the first embodiment of the application.

[0033] Figure 3 FIG. 3 is a detailed flowchart of the method of the first embodiment of the application.

[0034] Figure 4 FIG. 4 is a flowchart of the search mechanism for generating the optimal fusion scheme from multiple candidate fusion schemes in the second embodiment of the application. DETAILED DESCRIPTION

[0035] Embodiment I:

[0036] As Figure 1 andFigure 3 As shown, the operator fusion method of the embodiment facing the image recognition deep neural network comprises:

[0037] S101, block the calculation graph of the input deep neural network model into calculation subgraphs;

[0038] S102, respectively, randomly construct multiple candidate fusion schemes for each calculation subgraph, test the candidate fusion schemes on the hardware to obtain the execution time, and select the candidate fusion scheme with the shortest execution time as the optimal fusion scheme; the operator fusion types in the candidate fusion scheme include fusion of two consecutive simple operators, fusion of two consecutive complex operators, and fusion of consecutive simple operators and complex operators, and the fusion of two consecutive complex operators is to recombine the latter complex operator to unify the external loops of the two complex operators into the dimensions and shapes based on the intermediate result tensor, and then directly combine the external loops to implement;

[0039] S103, combine the optimal fusion schemes of each calculation subgraph to obtain the optimal fusion execution scheme of the deep neural network model.

[0040] As can be seen from steps S101-S103, the operator fusion framework adopted by the operator fusion method of the embodiment facing the image recognition deep neural network comprises two aspects: one is a complex operator fusion mechanism based on loop restructuring, and the other is a search mechanism for generating the optimal fusion scheme from multiple candidate fusion schemes.

[0041] A: complex operator fusion mechanism based on loop restructuring.

[0042] During the execution of a neural network, due to the multi-dimensionality of the data and the corresponding iterative algorithms in neural network computation, the execution of each operator is completed through multiple layers of iterative computation. Typically, the computational strategy of an operator is determined by the dimension of its output data; that is, the outer loop of the multiple loops in the operator's execution strategy determines the dimension of the operator's output data. Because there are no dependencies between data in simple operators—that is, the mapping from input data to output data is one-to-one—the dimensions of the input and output data of a simple operator are the same. Due to the lack of inter-layer data dependencies in simple operators, simple operators can be directly merged with adjacent simple operators, that is, merged with the same outer loop to form a new operator. This does not affect the correctness of the data and can reduce data movement while improving computational efficiency; this method is called operator fusion. However, due to the special computational properties of complex operators, the dimensions of the input and output data are different, and there are inter-layer dependencies between the data. Therefore, complex operators can only be merged with adjacent simple operators. Adjacent complex operators cannot be directly merged due to inconsistencies in their computational strategies, i.e., inconsistent outer loops. The fundamental obstacle to complex operator fusion is the inconsistent multidimensional loops resulting from output-centric computational strategies. Current neural network compilation and execution frameworks for image recognition treat each complex operator as a separate producer-consumer cycle with its own independent computational strategy. Based on this, this embodiment addresses the inability to fuse two complex operators by considering the computational strategy holistically, thereby improving the execution efficiency of the fused operation. Due to the internal mapping and storage call relationships within complex operators, some of the same input data is invoked when computing consecutive output scalars; therefore, one input scalar will be mapped to multiple output scalars. During computation, finding a mapping function that maps a single input scalar to all its associated output data can avoid repeated calls to the input data. The computational strategy for such complex operators involves mapping the scalars in a single input data point to the entire output data tensor, while the outer loop in the computational strategy is based on the data dimension of the input tensor. This computational strategy can be described as an input-centric computational strategy; in this strategy, the internal computational function is the mapping function from each individual input scalar to the entire output tensor, while the outer loop is based on the shape and dimension of the input tensor. For two adjacent complex operators, the computation strategy of the preceding complex operator (an output-centric computation strategy) is maintained, while the computation strategy of the following complex operator is modified to an input-centric computation strategy. This ensures that the computation strategies of both adjacent complex operators are based on the dimension of the intermediate data between them, allowing for direct loop merging. Specifically, in this embodiment, the loop reorganization of the following complex operator, unifying the outer loops of both complex operators to be based on the dimension and shape of the intermediate result tensor, before directly merging the outer loops includes:

[0043] S201, for the two continuous complex operators to be fused, keeping the calculation strategy of the previous complex operator as the output tensor-based calculation strategy unchanged, modifying the calculation strategy of the next complex operator to the input tensor-based calculation strategy through loop restructuring, the mapping relationship in the input tensor-based calculation strategy is that a single input scalar is mapped to the entire output tensor, and the external loop is based on the input tensor of the operator, so that the external loops of the calculation strategies of the two complex operators are unified based on the dimensions and shapes of the intermediate result tensors of the two complex operators;

[0044] S202, directly merging the two complex operators after the external loop of the calculation strategy is unified based on the dimensions and shapes of the intermediate result tensors of the two complex operators, to obtain the fused code.

[0045] The example code 1 gives the execution code of the GEMM-ReLU-GEMM operator group after fusion.

[0046]

[0047] The numbers on the left in the example code 1 are line numbers, and the inputs of the example code 1 are tensors A (tensor dimensions M x K), B (tensor dimensions K x N), and C (tensor dimensions N x L). Through fusion, it can be seen that through the output-centered calculation strategy, a total of intermediate scalar M[m][n] is calculated, then through the activation ReLU (the calculation method is to compare with 0 through the max function), and then through the input-centered calculation strategy, M[m][n] is mapped to the related output scalar D[m][l], and through the loop, the final output tensor D (tensor dimensions M x L) is output.

[0048] As shown in step S201, in the complex operator fusion mechanism based on loop restructuring of the present embodiment, each intermediate result scalar is obtained by the first complex operator (output-centered calculation strategy), and through the second complex operator (input-centered calculation strategy), the intermediate result scalar can be mapped to the final related output tensor without repeatedly calling the intermediate result, so as to remove redundant calculation. On the other hand, when modifying the calculation strategy of the second complex operator, the external loop thereof is based on the dimensions of its input tensor (intermediate data of the two operators). In this way, the two continuous complex operators unify the external loop, and they can be directly loop-merged. The operator fusion method based on loop restructuring modifies the calculation strategy of the complex operator, unifies the external loop axes of the two complex operators, so as to achieve the fusion of the two complex operators, converts the output tensor-centered calculation strategy to the input tensor-centered calculation strategy, facilitates the user to fuse the two continuous complex operators, and has the advantage of universality.

[0049] B: search mechanism of generating optimal fusion scheme from multiple candidate fusion schemes.

[0050] Another important part of the operator fusion framework adopted by the operator fusion method for image recognition deep neural network is the operator fusion strategy for the entire network, that is, to decide which fusion and how to fuse the fusion operators in the entire image recognition network. Since the fusion support of existing fusion methods is limited, the search space is limited. In contrast, the search mechanism of generating optimal fusion scheme from multiple candidate fusion schemes in the embodiment includes three parts: first, the input deep neural network model is divided into calculation subgraphs; then, a sufficient number of multiple candidate fusion schemes are randomly constructed for each calculation subgraph; and then, the execution time of the candidate fusion scheme is obtained by hardware testing, and the candidate fusion scheme with the shortest execution time is selected as the optimal fusion scheme. It should be noted that when selecting the candidate fusion scheme with the shortest execution time as the optimal fusion scheme, the search space of the candidate fusion scheme can also be expanded based on the initial candidate fusion scheme through genetic algorithm or other optimization search methods to improve the search optimization effect of the optimal fusion scheme.

[0051] Step S101 divides the calculation graph into calculation subgraphs. The main purpose of this is to reduce the time of searching for the optimal scheme. In the embodiment, step S101 divides the calculation graph of the input deep neural network model into calculation subgraphs. The calculation graph of the deep neural network model can be directly provided or converted from TVM, ONNX or MNN software from pytorch, TensorFlow and other image recognition task execution frameworks. The deep neural network model in the embodiment is a deep learning neural network, which is used to classify the input image to obtain the classification result of the image, for example, VGG, AlexNet, ResNet and MobileNet, etc. Needless to say, the method of the embodiment does not depend on the specific structure of the deep neural network model.

[0052] As an optional implementation, in order to ensure the correctness of the operator fusion optimization, the calculation results of the non-fused continuous complex operators are taken as the benchmark for comparison to verify the correctness of the algorithm optimization. Specifically, after step S202, the correctness verification is further included for the fused code: for the input tensor of the same previous complex operator, the calculation results of the two complex operators before fusion are taken as the benchmark value, and the calculation results of the fused code are compared with the benchmark value. If they are the same, it is determined that the correctness verification passes, otherwise, it is determined that the correctness verification fails, and the operator fusion mode in the candidate fusion scheme is adjusted again to perform the correctness verification, until the fusion of the two continuous complex operators in the candidate fusion scheme all passes the correctness verification.

[0053] Referring to Figure 2 In the embodiment, when testing the candidate fusion scheme on the hardware, the base code of the candidate fusion scheme is further optimized before testing the candidate fusion scheme, and the fusion code optimization includes part or all of parallel optimization, loop unrolling, and vectorization optimization, so that the code execution is more efficient.

[0054] Referring to Figure 2 In the embodiment, when testing the candidate fusion scheme on the hardware, the complex operators in the candidate fusion scheme are further optimized before testing the candidate fusion scheme: the activation function between the two complex operators in the candidate fusion scheme is modified into a judgment condition for controlling the execution of the calculation, so that the function mapping of the intermediate data scalar obtained by only the previous complex operator is performed by the next complex operator part only when the intermediate data scalar meets the activation condition of the activation function, otherwise the function mapping of the intermediate data scalar is cancelled by the next complex operator part. Sparse optimization refers to reducing the redundant calculation caused by the sparsity of the intermediate data through optimization means. In the neural network, the sparsity of the intermediate data is caused by the activation function such as ReLU, which sets the data not meeting the activation condition to zero. However, the existing neural network compilation and execution framework only fuses the activation function such as ReLU as a normal simple operator, but does not take advantage of the characteristics of the activation function. We consider the characteristics of the activation function and modify the activation function between the two complex operators into a judgment condition for controlling the execution of the calculation. The function mapping of the intermediate data scalar obtained by only the previous complex operator is performed by the next complex operator when the intermediate data scalar meets the activation condition of the activation function ReLU, otherwise the function mapping of the intermediate data scalar is cancelled by the next complex operator. In this way, the redundant calculation caused by the sparsity of the intermediate data can be reduced, and the efficiency of the function execution after fusion is improved.

[0055] The example code 2 gives the pseudo code of the GEMM-ReLU-GEMM operator group fusion after sparse optimization.

[0056]

[0057] In the example code 2, the input is a tensor A (tensor dimension is M x K), a tensor B (tensor dimension is K x N), and a tensor C (tensor dimension is N x L). Through fusion, it can be seen that through the output-centered calculation strategy, a total of intermediate scalars M[m][n] are calculated, and then the ReLU is activated (the calculation method is to compare with 0 through the max function). If the activated value M[m][n] is greater than 0, the input-centered calculation strategy is used to map M[m][n] to the related output scalar D[m][l], and the final output tensor D (tensor dimension is M x L) is output through the loop; if the activated value M[m][n] is less than 0, the function mapping is not performed, and the next scalar D[m][l] is calculated and the mapping of the corresponding D[m][l] is judged.

[0058] Referring to Figure 2 In the embodiment, when testing the candidate fusion scheme on the hardware, the sub-operator fusion optimization of the complex operator in the candidate fusion scheme is further included before testing the candidate fusion scheme: the complex operator with a sub-operator in the candidate fusion scheme is detected, and for each sub-operator of the complex operator, if the sub-operator can be fused with the sub-operator or the simple operator of the adjacent complex operator, the sub-operator is fused with the sub-operator or the simple operator of the adjacent complex operator. The sub-operator fusion optimization is for some complex operators with multiple calculation steps, such as the Softmax operator and the Batchnorm operator. These operators can be divided into many small operators, and we can use this feature to divide these complex operators with multiple calculation steps into many sub-operators, and then fuse these sub-operators with the adjacent operators to improve the execution efficiency of the fused operators. The Softmax operator can be divided into three sub-operators: an exponential sub-operator, a summation sub-operator, and a division sub-operator. In the DNN calculation graph, the Softmax operator is connected to a GEMM operator, and the exponential sub-operator and the summation sub-operator of the Softmax operator can be fused with the GEMM operator. Such fusion can improve the overall execution efficiency of the GEMM-Softmax operator group fusion.

[0059] The various fusion code optimization, sparse optimization and sub-operator fusion optimization described above in the embodiment are divided into two aspects: one is code self-generating optimization, including the above-mentioned calculation parallelism, loop unrolling, calculation vectorization and a series of optimization means; the other is the optimization of the characteristics of the operator and the specific optimization between the operators. The method of the embodiment provides some operator-specific optimization and characteristic optimization between operators, which includes sparse optimization, sub-operator optimization and operator sequence optimization. Programmers can select the corresponding optimization means to improve the efficiency of execution after fusion through the characteristics of the two consecutive operators. In addition, the embodiment also includes operator sequence optimization, that is, by adjusting the execution order of the operators to reduce redundant calculation to improve execution efficiency. This fusion mode can be used for GEMM-Dropout-GEMM fusion mode. Before the operator executes, we can get the intermediate data that is zeroed in advance and eliminate the calculation process of calculating this data from the first GEMM operator.

[0060] Referring to Figure 2 After the above-mentioned various fusion code optimization, sparse optimization and sub-operator fusion optimization are completed, the code of the candidate fusion scheme can be deployed to the required hardware (CPU, GPU, DSP, etc.) for testing by using the required software backend (LLVM, CUDA, OPENCL, etc.), so as to obtain the execution time of the candidate fusion scheme.

[0061] The search mechanism for generating the best fusion scheme of the various candidate fusion schemes in the embodiment can use the required search method as needed. For example, as an optional implementation, the genetic algorithm is used in the embodiment to optimize the search mechanism for generating the best fusion scheme of the various candidate fusion schemes to promote better fusion performance of the entire network.

[0062] Specifically, when the step S102 randomly generates a plurality of candidate fusion schemes for each calculation subgraph, tests the candidate fusion schemes on the hardware to obtain the execution time, and selects the candidate fusion scheme with the shortest execution time as the best fusion scheme, the processing for each calculation subgraph includes:

[0063] S301, based on the pre-defined basic population number M, randomly generating M initial candidate fusion schemes for the calculation subgraph to obtain an initial candidate fusion scheme test set, and testing each candidate fusion scheme in the initial candidate fusion scheme test set on the hardware to obtain the execution time;

[0064] S302, updating the candidate fusion schemes in the candidate fusion scheme test set;

[0065] S303, testing the candidate fusion schemes in the updated candidate fusion scheme test set on the hardware to obtain the execution time;

[0066] S304, it is judged whether the multiple candidate fusion schemes with shorter execution time in the updated candidate fusion scheme test set remain unchanged, if the candidate fusion schemes remain unchanged, it is determined that the candidate fusion scheme test set has been stabilized, a candidate fusion scheme with the shortest execution time is selected from the stabilized candidate fusion scheme test set as the optimal fusion scheme of the calculation subgraph, and the processing of the next calculation subgraph or exit is entered; otherwise, step S302 is jumped to.

[0067] In the embodiment, updating the candidate fusion schemes in the candidate fusion scheme test set in step S302 includes: sorting the candidate fusion schemes in the candidate fusion scheme test set according to the execution time, and deleting part of the candidate fusion schemes with longer execution time; generating new candidate fusion schemes by using a genetic algorithm according to a pre-defined mutation probability P, and constructing the updated candidate fusion scheme test set by using the new candidate fusion schemes and the remaining part of the candidate fusion schemes. It should be noted that the genetic algorithm for generating new candidate fusion schemes according to the pre-defined mutation probability P is a known method of the genetic algorithm, and thus the implementation method will not be described again. In the embodiment, the test results (the running time of the execution subgraph) of all the fusion schemes in the neural network subgraph are sorted according to the time from short to long by iteration, part of the fusion scheme candidates with longer execution time are removed, new fusion scheme candidates are generated based on the mutation probability P and added to the fusion scheme population with shorter execution time in the front half, a new fusion scheme population is formed, the updated fusion scheme population with M number of fusion schemes is sent to the test module, and the execution effect of all the fusion scheme candidates on the actual hardware is recorded. Through the judgment condition that the candidate fusion scheme test set has been stabilized, the optimal fusion scheme of the neural network subgraph can be found, and then the next neural network subgraph to be divided into blocks is obtained from step one. In this way, until all the subgraphs of the neural network are iterated, the optimal fusion scheme of the whole deep neural network model can be obtained.

[0068] In summary, the operator fusion method for image recognition deep neural network can achieve the following technical effects: first, the neural network operator fusion framework adopted by the method outputs the fused computation graph and deploys it on different hardware backends by inputting the computation graph of the neural network (converted by ONNX or TVM). Programmers input the neural network computation graph to be optimized by operator fusion, define the related parameters of optimization means and fusion scheme search, obtain the computation graph after operator fusion optimization, and can select different backends for deployment on related hardware devices. Second, the complex operator fusion method based on cycle reorganization and the related optimization means proposed by the method can solve the problem that two consecutive complex operators in the neural network compilation and execution of image recognition cannot be fused. The fusion of adjacent complex operators improves the efficiency of operator execution and provides more opportunities for operator fusion of the entire neural network. The increase of the neural network fusion scheme search space is beneficial to the search algorithm to find the optimal neural network fusion scheme. The complex operator fusion method and the entire network fusion search method improve the execution efficiency of the image recognition task. Third, the neural network fusion scheme search strategy based on genetic algorithm proposed by the method can effectively find the best neural network fusion scheme in the network fusion scheme search space. Programmers can set different parameters to adjust the search algorithm to meet different search needs.

[0069] In addition, the embodiment also provides an operator fusion system for image recognition deep neural network, which comprises a microprocessor and a memory connected with each other, and the microprocessor is programmed or configured to execute the operator fusion method for image recognition deep neural network. In addition, the embodiment also provides a computer readable storage medium, wherein a computer program for programming or configuring the microprocessor to execute the operator fusion method for image recognition deep neural network is stored in the computer readable storage medium.

[0070] Embodiment two:

[0071] The embodiment is basically the same as embodiment one, and the main difference is that the iteration end condition of the genetic algorithm in the search mechanism of generating the best fusion scheme from a plurality of candidate fusion schemes is different. For example, Figure 2As shown, step S304 in the embodiment can also use the number of iterations to control iteration, for example: add 1 to the number of iterations, if the number of iterations is equal to the preset number N+1, select a candidate fusion scheme with the shortest execution time in the candidate fusion scheme test set as the optimal fusion scheme of the computational subgraph, and enter the processing of the next computational subgraph or exit; otherwise, jump to step S302. Through the judgment condition of N times of iteration, the optimal fusion scheme of the neural network subgraph can be found, and then the next blocked neural network subgraph is obtained from step one. In this way, until all subgraphs of the neural network are iterated, the optimal fusion scheme of the entire deep neural network model can be obtained.

[0072] In addition, the embodiment also provides an operator fusion system for an image recognition deep neural network, which comprises a microprocessor and a memory connected to each other, the microprocessor is programmed or configured to execute the operator fusion method for the image recognition deep neural network. In addition, the embodiment also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is used for programming or configuring the microprocessor to execute the operator fusion method for the image recognition deep neural network.

[0073] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes. The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1the functions specified in the flow or flows and / or blocks of the flow or flows. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processing, and the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flow or flows and / or blocks of the flow or flows. Figure 1 the functions specified in the flow or flows and / or blocks of the flow or flows. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processing, and the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flow or flows and / or blocks of the flow or flows. Figure 1 the functions specified in the flow or flows and / or blocks of the flow or flows. Such computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are generated to realize the computer-implemented processing, and the instructions executed on the computer or other programmable devices provide a process for implementing the functions specified in the flow or flows and / or blocks of the flow or flows.

[0074] The above description is only the preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solutions falling within the concept of the present application shall be deemed to fall within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application shall also be deemed to fall within the protection scope of the present application.

Claims

1. An operator fusion method for an image recognition oriented deep neural network, characterized in that, The method comprises the following steps: S101, dividing a calculation graph of an input deep neural network model into calculation subgraphs; S102, randomly constructing multiple candidate fusion schemes for each calculation subgraph, testing the candidate fusion schemes on hardware to obtain execution time, and selecting a candidate fusion scheme with the shortest execution time as an optimal fusion scheme; the operator fusion types in the candidate fusion schemes comprise fusion of two continuous simple operators, fusion of two continuous complex operators, and fusion of a simple operator and a complex operator in sequence, and the fusion of two continuous complex operators is to recombine a loop of a latter complex operator to unify external loops of the two complex operators based on dimensions and shapes of intermediate result tensors and then directly combine the external loops; S103, combining the optimal fusion schemes of the calculation subgraphs to obtain an optimal fusion execution scheme of the deep neural network model.

2. The operator fusion method for image recognition oriented deep neural networks according to claim 1, characterized in that, The recombination of the loop of the latter complex operator to unify the external loops of the two complex operators based on the dimensions and shapes of the intermediate result tensors and then the direct combination of the external loops comprises: S201, for the two continuous complex operators to be fused, keeping a calculation strategy of a former complex operator unchanged as an output tensor-based calculation strategy, modifying a calculation strategy of a latter complex operator as an input tensor-based calculation strategy through loop recombination, the mapping relationship in the input tensor-based calculation strategy being mapping of a single input scalar to an entire output tensor, and the external loop being based on an input tensor of the operator, so that the external loops of the calculation strategies of the two complex operators are unified as dimensions and shapes of intermediate result tensors of the two complex operators; S202, directly combining the external loops of the two complex operators after the external loops of the calculation strategies are unified as the dimensions and shapes of the intermediate result tensors of the two complex operators, to obtain fused code.

3. The operator fusion method for image recognition oriented deep neural networks according to claim 2, characterized in that, After step S202, the method further comprises correctness verification on the fused code: taking calculation results of the two complex operators before fusion as reference values for an input tensor of the same former complex operator, comparing a calculation result of the fused code with the reference values, and if the two are the same, determining that the correctness verification passes, otherwise, determining that the correctness verification fails, and then readjusting an operator fusion manner in the candidate fusion scheme and performing the correctness verification again until the fusion of the two continuous complex operators in the candidate fusion scheme all pass the correctness verification.

4. The operator fusion method for image-agnostic deep neural networks according to claim 2, wherein, When the candidate fusion scheme is tested on the hardware, the method further comprises fusion code optimization on base code of the candidate fusion scheme before the candidate fusion scheme is tested, and the fusion code optimization comprises part or all of parallel optimization, loop unrolling, and vectorization optimization.

5. The operator fusion method for image-agnostic deep neural networks according to claim 2, wherein, The sparse optimization of the complex operators in the candidate fusion scheme before testing the candidate fusion scheme comprises: modifying an activation function located between two complex operators in the candidate fusion scheme into a control calculation execution condition, so that the function mapping of an intermediate data scalar obtained by a previous complex operator part is performed only when the intermediate data scalar meets the activation condition of the activation function, and otherwise the function mapping of the intermediate data scalar by the subsequent complex operator part is cancelled.

6. The operator fusion method for image-agnostic deep neural networks according to claim 2, wherein, The sub-operator fusion optimization of the complex operators in the candidate fusion scheme before testing the candidate fusion scheme comprises: detecting the complex operators with sub-operators in the candidate fusion scheme, and fusing each sub-operator of the complex operators with the sub-operators of adjacent complex operators or simple operators.

7. The operator fusion method for image-agnostic deep neural networks according to claim 2, wherein, When the candidate fusion schemes are tested on the hardware to obtain the execution time, and the candidate fusion scheme with the shortest execution time is selected as the optimal fusion scheme, the processing of each calculation subgraph comprises: S301, generating M initial candidate fusion schemes for the calculation subgraph based on a predefined basic population quantity M to obtain an initial candidate fusion scheme test set, and testing each candidate fusion scheme in the initial candidate fusion scheme test set on the hardware to obtain the execution time; S302, updating the candidate fusion schemes in the candidate fusion scheme test set; S303, testing the candidate fusion schemes in the updated candidate fusion scheme test set on the hardware to obtain the execution time; S304, determining whether the candidate fusion schemes with shorter execution time in the updated candidate fusion scheme test set remain unchanged, if the candidate fusion schemes remain stable, it is determined that the candidate fusion scheme test set has been stable, a candidate fusion scheme with the shortest execution time is selected from the stable candidate fusion scheme test set as the optimal fusion scheme of the calculation subgraph, and the processing of the next calculation subgraph or the exit is entered; otherwise, step S302 is jumped to.

8. The operator fusion method for image recognition oriented deep neural networks according to claim 7, characterized in that, The updating of the candidate fusion schemes in the candidate fusion scheme test set in step S302 comprises: sorting the candidate fusion schemes in the candidate fusion scheme test set according to the execution time, and deleting part of the candidate fusion schemes with longer execution time; generating new candidate fusion schemes from the remaining part of the candidate fusion schemes in the candidate fusion scheme test set according to the predefined mutation probability P by using a genetic algorithm, and constructing the updated candidate fusion scheme test set by combining the new candidate fusion schemes with the remaining part of the candidate fusion schemes.

9. An operator fusion system for image recognition oriented deep neural networks, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to perform the operator fusion method for the image recognition deep neural network according to any one of claims 1-8.

10. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program is used for programming or configuring a microprocessor to execute the operator fusion method for the image recognition oriented deep neural network according to any one of claims 1-8.

Images