Model reasoning operator fusion optimization method and system based on operator calling

Abstract

The invention relates to the technical field of computer artificial intelligence, and discloses a model reasoning operator fusion optimization method and system based on operator calling, and the method comprises the steps: analyzing operators in a model file, carrying out the sorting according to the influence degree of the operators on the model reasoning performance, and carrying out the fusion optimization processing of the operators in the model file in sequence, thereby achieving the fusion optimization of the model reasoning performance. And the optimization effect is verified, so that the optimization of the whole model reasoning process is realized, the data transmission delay of the platform is reduced, the data access speed is improved, and the scene application performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence technology, and discloses a model reasoning operator fusion optimization method and system based on operator calling. Background Art

[0002] The reasoning operation of deep neural network models on intelligent platforms depends on the completeness and efficiency of the operator library implemented on the intelligent platform. Operator fusion technology is the current mainstream optimization method for operator library implementation, including parallel operator fusion, loop operator fusion, matrix operator fusion and other methods. However, general optimization methods cannot achieve the best results in the reasoning process of a specific network model. Summary of the invention

[0003] The purpose of the present invention is to provide a model reasoning operator fusion optimization method and system based on operator call, which can reduce the data transmission delay of the platform and improve the data access speed.

[0004] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0005] A model reasoning operator fusion optimization method based on operator call, comprising:

[0006] Step 1: Analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform;

[0007] Step 2: According to the calling frequency of each operator on the platform and the operator computing performance, the influence of each operator on the model reasoning performance is analyzed and obtained, and all operators of the model file are sorted according to the influence on the model file;

[0008] Step 3: According to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, the operator fusion method is evaluated, and whether the current operator can be fused to form a new operator is determined in the sorting order; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has a performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform the operation in the model inference stage;

[0009] If the operators cannot be merged into a new operator or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed according to step 3 until the last operator is judged or merged.

[0010] Furthermore, the degree of influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

[0011] Furthermore, the method for determining whether the current operator can be fused to form a new operator includes:

[0012] The on-chip storage resources satisfy the data interaction between the current operator and at least one operator in the platform operator library;

[0013] Or, the computation process between the current operator and at least one operator in the platform operator library can be processed in parallel;

[0014] Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0015] Furthermore, the basis for judging whether the new operator has performance improvement over the current operator includes:

[0016] During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

[0017] In order to achieve the above technical effects, the present invention also provides a model reasoning operator fusion optimization system based on operator call, which is used to implement the model reasoning operator fusion optimization method based on operator call, including:

[0018] The model file preprocessing module is used to analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators; preprocess the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform;

[0019] The first analysis module is used to analyze and obtain the influence of each operator on the model reasoning performance according to the calling frequency of each operator on the platform and the operator operation performance, and sort all operators of the model file according to the influence on the model file;

[0020] An evaluation and fusion module is used to evaluate the operator fusion method according to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and determine in sequence according to the sorting order whether the current operator can be fused to form a new operator; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has improved performance compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference stage; if it cannot be fused into a new operator, or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

[0021] Furthermore, in the first analysis module, the influence degree of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

[0022] Furthermore, in the evaluation fusion module, the method for judging whether the current operator can be fused to form a new operator includes:

[0023] The on-chip storage resources satisfy the data interaction between the current operator and at least one operator in the platform operator library;

[0024] Or, the computation process between the current operator and at least one operator in the platform operator library can be processed in parallel;

[0025] Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0026] Furthermore, in the evaluation fusion module, the basis for judging whether the new operator has performance improvement over the current operator includes:

[0027] During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

[0028] Compared with the prior art, the present invention has the following beneficial effects: the present invention analyzes the operators in the model file, sorts the operators according to their influence on the model reasoning performance, performs fusion optimization processing on the operators in the model file in turn, and verifies the optimization effect, thereby optimizing the entire model reasoning process, reducing the data transmission delay of the platform, increasing the data access speed, and improving the scenario application performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the model reasoning operator fusion optimization method based on operator call in the embodiment;

[0030] Figure 2 It is a structural block diagram of the model reasoning operator fusion optimization system based on operator call in the embodiment;

[0031] Among them, 1. Model file preprocessing module; 2. First analysis module; 3. Evaluation fusion module. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the embodiments and drawings. However, it should not be understood that the scope of the above subject matter of the present invention is limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0033] Example 1

[0034] See also Figure 1-Figure 2, a model reasoning operator fusion optimization method based on operator call, comprising:

[0035] Step 1: Analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform;

[0036] Step 2: According to the calling frequency of each operator on the platform and the operator computing performance, the influence of each operator on the model reasoning performance is analyzed and obtained, and all operators of the model file are sorted according to the influence on the model file;

[0037] Step 3: According to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, the operator fusion method is evaluated, and whether the current operator can be fused to form a new operator is determined in the sorting order; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has a performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform the operation in the model inference stage;

[0038] If the operators cannot be merged into a new operator or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed according to step 3 until the last operator is judged or merged.

[0039] In this embodiment, by analyzing the operators mapped in the model file, the computing performance and calling frequency of each operator on the platform are sorted out, and the operators are prioritized according to the operator calling frequency and computing performance; according to the operator library implemented on the current platform, the platform on-chip storage resources, input data dimensions, computing resource parallelism and other factors are comprehensively considered to perform operator fusion evaluation on the sorted operators; if the current operator can be fused to form a new operator, the performance of the fused operator is tested, and if the performance of the current new operator is improved, the new operator is marked and added to the platform operator library; if the new operator is not formed or the current operator is not the last one, it is processed in sequence according to the sorting of the model file operator list. After processing all the operators in the model file operator list, a new network model file with improved reasoning performance can be obtained. The new network model file calls the new operator to replace the previous operator to perform calculations in the model reasoning stage, thereby optimizing the reasoning performance of the network model file. This embodiment optimizes the entire model reasoning process by analyzing the operators in the model file, sorting the operators according to their influence on the model reasoning performance, fusing and optimizing the operators in the model file in turn, and verifying the optimization effect, thereby reducing the data transmission delay of the platform, increasing the data access speed, and improving the scenario application performance.

[0040] Based on the same inventive concept, this embodiment also provides a model reasoning operator fusion optimization system based on operator call, including:

[0041] Model file preprocessing module 1 is used to analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependency relationship between operators; preprocess the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform;

[0042] The first analysis module 2 is used to analyze and obtain the influence of each operator on the model reasoning performance according to the calling frequency of each operator on the platform and the operator operation performance, and sort all operators of the model file according to the influence on the model file;

[0043] Evaluation and fusion module 3 is used to evaluate the operator fusion method according to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and judge whether the current operator can be fused to form a new operator in the sorting order; if the current operator can form a new operator, the performance of the new operator is tested. If the new operator has improved performance compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference stage; if it cannot be fused into a new operator, or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

[0044] Example 2

[0045] like Figure 1 As shown, a model reasoning operator fusion optimization method based on operator call includes the following steps:

[0046] Step (1): Analyze the model file of the inference model of the input platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators, pre-process the model file on the platform, sort out the computing performance and calling frequency of the operators on the specific platform, and execute step (2).

[0047] Step (2): Comprehensively consider the operator call frequency and operator computing performance on a specific platform, and use the product of the operator call frequency on the platform and the operator computing performance to evaluate the impact of each operator on the model reasoning performance, and sort the operator mapping table according to the impact on the model file, and execute step (3). In this embodiment, the product of the frequency and the operator computing performance can be used as the weight value of the evaluation operator for the model, and this sorting can accurately locate the relevant operators with the maximum performance improvement, thereby maximizing the performance benefits after optimization.

[0048] Step (3): Based on the operator library implemented on the current platform, comprehensively consider factors such as on-chip storage resources, input data dimensions, and computing resource parallelism on the platform to evaluate the operator fusion method. If the current operator can be fused to form a new operator, execute step (4); if the new operator cannot be formed and the current operator is not the last operator, execute step (6); In this embodiment, the method for determining whether the current operator can be fused to form a new operator includes: the on-chip storage resources meet the data interaction between the current operator and at least one operator in the platform operator library; or, the computing process between the current operator and at least one operator in the platform operator library can be processed in parallel; or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0049] Step (4): Test the performance of the new operator. If the new operator has improved performance compared with the previous operator, execute step (5). If the performance of the current new operator has not improved and the current operator is not the last operator, execute step (6). In this embodiment, the basis for judging whether the new operator has improved performance compared with the current operator is that the time for calling the fused new operator during the model inference process is shorter than the time for calling the current operator.

[0050] Step (5): Mark the new operator and add it to the operator library of the platform. In the model inference phase, call the new operator to replace the previous operator to perform the operation. If the current operator is the last operator, execute step (7), otherwise execute step (6).

[0051] Step (6): If the current operator is not the last one, optimize the operators in the model file in sequence and execute step (3).

[0052] Step (7): After optimizing the last operator, use the optimized platform operator library to test the inference performance of the model file.

[0053] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A model reasoning operator fusion optimization method based on operator call, characterized in that: include: Step 1: Analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform; Step 2: According to the calling frequency of each operator on the platform and the operator computing performance, the influence of each operator on the model reasoning performance is analyzed and obtained, and all operators of the model file are sorted according to the influence on the model file; Step 3: According to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, the operator fusion method is evaluated, and whether the current operator can be fused to form a new operator is determined in the sorting order; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has a performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform the operation in the model inference stage; If the operators cannot be merged into a new operator or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed according to step 3 until the last operator is judged or merged.

2. The model reasoning operator fusion optimization method based on operator call according to claim 1 is characterized in that: The influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

3. The model reasoning operator fusion optimization method based on operator call according to claim 2 is characterized in that: Methods for determining whether the current operators can be fused to form new operators include: The on-chip storage resources satisfy the data interaction between the current operator and at least one operator in the platform operator library; Or, the computation process between the current operator and at least one operator in the platform operator library can be processed in parallel; Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

4. The model reasoning operator fusion optimization method based on operator call according to claim 1, characterized in that: The basis for judging whether the new operator has performance improvement over the current operator includes: During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

5. A model reasoning operator fusion optimization system based on operator call, used to implement the model reasoning operator fusion optimization method based on operator call according to any one of claims 1 to 4, characterized in that: include: The model file preprocessing module is used to analyze the model file of the inference model input into the platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators; preprocess the model file on the platform, and sort out the computing performance and calling frequency of each operator on the platform; The first analysis module is used to analyze and obtain the influence of each operator on the model reasoning performance according to the calling frequency of each operator on the platform and the operator operation performance, and sort all operators of the model file according to the influence on the model file; An evaluation and fusion module is used to evaluate the operator fusion method according to the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and determine in sequence according to the sorting order whether the current operator can be fused to form a new operator; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has improved performance compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference stage; if it cannot be fused into a new operator, or the performance of the new operator is not improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

6. The model reasoning operator fusion optimization system based on operator call according to claim 5, characterized in that: In the first analysis module, the influence degree of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

7. The model reasoning operator fusion optimization system based on operator call according to claim 5, characterized in that: In the evaluation fusion module, the method for judging whether the current operator can be fused to form a new operator includes: The on-chip storage resources satisfy the data interaction between the current operator and at least one operator in the platform operator library; Or, the computation process between the current operator and at least one operator in the platform operator library can be processed in parallel; Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

8. The model reasoning operator fusion optimization system based on operator call according to claim 5, characterized in that: In the evaluation fusion module, the basis for judging whether the new operator has performance improvement over the current operator includes: During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

Images