A model inference operator fusion optimization method and system based on operator call

Abstract

The application relates to the technical field of computer artificial intelligence, and discloses a model inference operator fusion optimization method and system based on operator calling, which analyzes operators in a model file, sorts the operators according to the influence degree of the operators on model inference performance, sequentially performs fusion optimization processing on the operators in the model file, verifies the optimization effect, and then realizes optimization of the whole model inference process, reduces the data transmission delay of a platform, improves the data access speed, and improves the scene application performance.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence technology, and discloses a model reasoning operator fusion optimization method and system based on operator call. Background Art

[0002] The inference and execution of deep neural network models on intelligent platforms depends on the completeness and efficiency of the operator library implemented on the intelligent platform. Operator fusion technology is the current mainstream optimization method for operator library implementation, including parallel operator fusion, loop operator fusion, matrix operator fusion and other methods. However, general optimization methods cannot achieve the best results in the inference process of specific network models. Summary of the Invention

[0003] The purpose of the present invention is to provide a model reasoning operator fusion optimization method and system based on operator call, which can reduce the data transmission delay of the platform and improve the data access speed.

[0004] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0005] A model inference operator fusion optimization method based on operator call, comprising:

[0006] Step 1: Analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform;

[0007] Step 2: Analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file;

[0008] Step 3: Evaluate the operator fusion method based on the operator library implemented on the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform. Determine whether the current operators can be fused to form a new operator in the sorting order. If the current operators can form a new operator, test the performance of the new operator. If the new operator has improved performance compared to the current operator, mark the new operator and add it to the platform's operator library. In the model inference phase, call the new operator to replace the previous operator to perform the operation.

[0009] If the new operator cannot be fused or the performance of the new operator is not improved, and the current operator is not the last operator, execute the next operator of the current operator according to step 3 until the last operator is judged or fused.

[0010] Furthermore, the degree of influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

[0011] Furthermore, methods for determining whether the current operators can be fused to form a new operator include:

[0012] On-chip storage resources satisfy data interaction between the current operator and at least one operator in the platform operator library;

[0013] Alternatively, the computational process between the current operator and at least one operator in the platform operator library can be processed in parallel;

[0014] Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0015] Furthermore, the basis for determining whether the new operator has performance improvement over the current operator includes:

[0016] During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

[0017] To achieve the above technical effects, the present invention also provides a model reasoning operator fusion optimization system based on operator call, which is used to implement the model reasoning operator fusion optimization method based on operator call, including:

[0018] The model file preprocessing module is used to analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; preprocess the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform;

[0019] The first analysis module is used to analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file;

[0020] The evaluation and fusion module is used to evaluate the operator fusion method based on the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and to determine in sequence according to the sorting order whether the current operator can be fused to form a new operator; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference phase; if it cannot be fused into a new operator, or the performance of the new operator has not been improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

[0021] Furthermore, in the first analysis module, the degree of influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

[0022] Furthermore, in the evaluation and fusion module, the method for determining whether the current operator can be fused to form a new operator includes:

[0023] On-chip storage resources satisfy data interaction between the current operator and at least one operator in the platform operator library;

[0024] Alternatively, the computational process between the current operator and at least one operator in the platform operator library can be processed in parallel;

[0025] Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0026] Furthermore, in the evaluation and fusion module, the basis for determining whether the new operator has performance improvement over the current operator includes:

[0027] During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

[0028] Compared with the existing technology, the beneficial effects of the present invention are: the present invention analyzes the operators in the model file, sorts the operators according to their influence on the model reasoning performance, performs fusion optimization processing on the operators in the model file in turn, and verifies the optimization effect, thereby optimizing the entire model reasoning process, reducing the platform's data transmission delay, increasing data access speed, and improving scenario application performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart of the model reasoning operator fusion optimization method based on operator call in the embodiment;

[0030] Figure 2 This is a structural block diagram of the model reasoning operator fusion optimization system based on operator call in the embodiment;

[0031] Among them, 1. Model file preprocessing module; 2. First analysis module; 3. Evaluation fusion module. DETAILED DESCRIPTION

[0032] The present invention will be described in further detail below with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the present invention to the following embodiments, as all technologies implemented based on the present invention fall within the scope of the present invention.

[0033] Example 1

[0034] See also Figure 1-Figure 2, a model inference operator fusion optimization method based on operator call, including:

[0035] Step 1: Analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform;

[0036] Step 2: Analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file;

[0037] Step 3: Evaluate the operator fusion method based on the operator library implemented on the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform. Determine whether the current operators can be fused to form a new operator in the sorting order. If the current operators can form a new operator, test the performance of the new operator. If the new operator has improved performance compared to the current operator, mark the new operator and add it to the platform's operator library. In the model inference phase, call the new operator to replace the previous operator to perform the operation.

[0038] If the new operator cannot be fused or the performance of the new operator is not improved, and the current operator is not the last operator, execute the next operator of the current operator according to step 3 until the last operator is judged or fused.

[0039] In this embodiment, by analyzing the operators mapped in the model file, the computing performance and calling frequency of each operator on the platform are sorted out, and the operators are prioritized according to the operator calling frequency and computing performance; according to the operator library implemented by the current platform, the platform on-chip storage resources, input data dimensions, computing resource parallelism and other factors are comprehensively considered to evaluate the operator fusion of the sorted operators; if the current operator can be fused to form a new operator, the performance of the fused operator is tested, and if the performance of the current new operator is improved, the new operator is marked and added to the platform operator library; if the new operator is not formed or the current operator is not the last one, it is processed in sequence according to the sorting of the model file operator list. After processing all the operators in the model file operator list, a new network model file with improved reasoning performance can be obtained. The new network model file calls the new operator to replace the previous operator to perform operations in the model reasoning stage, thereby optimizing the reasoning performance of the network model file. This embodiment analyzes the operators in the model file, sorts them according to their impact on the model reasoning performance, performs fusion optimization processing on the operators in the model file in turn, and verifies their optimization effects, thereby optimizing the entire model reasoning process, reducing the platform's data transmission delay, increasing data access speed, and enhancing scenario application performance.

[0040] Based on the same inventive concept, this embodiment also provides a model reasoning operator fusion optimization system based on operator call, including:

[0041] Model file preprocessing module 1 is used to analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; preprocess the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform;

[0042] The first analysis module 2 is used to analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file;

[0043] Evaluation and fusion module 3 is used to evaluate the operator fusion method based on the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and judge whether the current operator can be fused to form a new operator in the sorting order; if the current operator can form a new operator, the performance of the new operator is tested. If the new operator has performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference stage; if it cannot be fused into a new operator, or the performance of the new operator has not been improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

[0044] Example 2

[0045] like Figure 1 As shown, a model reasoning operator fusion optimization method based on operator call includes the following steps:

[0046] Step (1): Analyze the model file of the inference model of the input platform, sort out the operators involved in the model file, the data transmission between operators and the dependencies between operators, preprocess the model file on the platform, sort out the computing performance and calling frequency of the operators on the specific platform, and execute step (2).

[0047] Step (2): Comprehensively consider the operator call frequency and operator computing performance on a specific platform, and use the product of the operator call frequency on the platform and the operator computing performance to evaluate the impact of each operator on the model reasoning performance, and sort the operators in the operator mapping table according to the degree of impact on the model file, and execute step (3). In this embodiment, the product of the frequency and the operator computing performance can be used as the weight value of the evaluation operator on the model, and this ranking can accurately locate the relevant operators with the greatest performance improvement, thereby maximizing the performance benefits after optimization.

[0048] Step (3): Based on the operator library implemented on the current platform, comprehensively consider factors such as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and evaluate the operator fusion method. If the current operator can be fused to form a new operator, execute step (4); if the new operator cannot be formed and the current operator is not the last one, execute step (6); In this embodiment, the method for determining whether the current operator can be fused to form a new operator includes: on-chip storage resources meet the data interaction between the current operator and at least one operator in the platform operator library; or, the computing process between the current operator and at least one operator in the platform operator library can be processed in parallel; or, the input of the current operator and at least one operator in the platform operator library is the same data source.

[0049] Step (4): Test the performance of the new operator. If the new operator has improved performance compared to the previous operator, execute step (5). If the performance of the current new operator has not improved and the current operator is not the last one, execute step (6). In this embodiment, the basis for judging whether the new operator has improved performance compared to the current operator is that the time to call the fused new operator during the model inference process is shorter than the time to call the current operator.

[0050] Step (5): Mark the new operator and add it to the operator library of the platform. In the model inference phase, call the new operator to replace the previous operator to perform the operation. If the current operator is the last operator, execute step (7), otherwise execute step (6).

[0051] Step (6): If the current operator is not the last one, optimize the operators in the model file in sequence and execute step (3).

[0052] Step (7): After optimizing the last operator, use the optimized platform operator library to test the inference performance of the model file.

[0053] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A model reasoning operator fusion optimization method based on operator call, characterized in that: include: Step 1: Analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; pre-process the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform; Step 2: Analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file; Step 3: Evaluate the operator fusion method based on the operator library implemented on the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform. Determine whether the current operators can be fused to form a new operator in the sorting order. If the current operators can form a new operator, test the performance of the new operator. If the new operator has improved performance compared to the current operator, mark the new operator and add it to the platform's operator library. In the model inference phase, call the new operator to replace the previous operator to perform the operation. If the new operator cannot be fused or the performance of the new operator is not improved, and the current operator is not the last operator, execute the next operator of the current operator according to step 3 until the last operator is judged or fused.

2. The operator-call-based model inference operator fusion optimization method according to claim 1, characterized in that: The degree of influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator's computing performance.

3. The operator-call-based model inference operator fusion optimization method according to claim 2, characterized in that: Methods for determining whether the current operators can be fused to form new operators include: On-chip storage resources satisfy data interaction between the current operator and at least one operator in the platform operator library; Alternatively, the computational process between the current operator and at least one operator in the platform operator library can be processed in parallel; Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

4. The model reasoning operator fusion optimization method based on operator call according to claim 1 is characterized in that: The basis for determining whether the new operator has performance improvement over the current operator includes: During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

5. A model reasoning operator fusion optimization system based on operator call, used to implement the model reasoning operator fusion optimization method based on operator call according to any one of claims 1 to 4, characterized in that: include: The model file preprocessing module is used to analyze the model file of the inference model input to the platform, sort out the operators involved in the model file, the data transmission between operators, and the dependencies between operators; preprocess the model file on the platform, and sort out the computing performance and call frequency of each operator on the platform; The first analysis module is used to analyze the influence of each operator on the model reasoning performance based on the call frequency and operation performance of each operator on the platform, and sort all operators of the model file according to the influence on the model file; The evaluation and fusion module is used to evaluate the operator fusion method based on the operator library implemented by the platform, as well as the on-chip storage resources, input data dimensions, and computing resource parallelism on the platform, and to determine in sequence according to the sorting order whether the current operator can be fused to form a new operator; if the current operator can form a new operator, the performance of the new operator is tested; if the new operator has performance improvement compared with the current operator, the new operator is marked and added to the operator library of the platform, and the new operator is called to replace the previous operator to perform operations in the model inference phase; if it cannot be fused into a new operator, or the performance of the new operator has not been improved, and the current operator is not the last operator, the next operator of the current operator is executed until the last operator is judged or fused.

6. The model reasoning operator fusion optimization system based on operator call according to claim 5 is characterized in that: In the first analysis module, the degree of influence of each operator on the model reasoning performance is the product of the calling frequency of the corresponding operator on the platform and the operator computing performance.

7. The model reasoning operator fusion optimization system based on operator call according to claim 5 is characterized in that: In the evaluation and fusion module, the method for determining whether the current operator can be fused to form a new operator includes: On-chip storage resources satisfy data interaction between the current operator and at least one operator in the platform operator library; Alternatively, the computational process between the current operator and at least one operator in the platform operator library can be processed in parallel; Or, the input of the current operator and at least one operator in the platform operator library is the same data source.

8. The model reasoning operator fusion optimization system based on operator call according to claim 5 is characterized in that: In the evaluation and fusion module, the basis for determining whether the new operator has performance improvement over the current operator includes: During model inference, the time to call the fused new operator is shorter than the time to call the current operator.

Images