Model reasoning performance optimization method, device and related products
By performing operator fusion processing in the inference acceleration engine, the calculation diagram of the deep learning model is optimized, which solves the problem of long-term model inference time caused by insufficient hardware optimization in the prior art, and significantly improves the model inference performance.
Patent Information
- Application Number
- CN202210700299.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-20
AI Technical Summary
The existing inference acceleration engine has not fully optimized the hardware-related optimization link, resulting in the on-chip memory space reading and writing data for a long time during the model inference process, affecting the model inference performance.
By obtaining the initial model file of the target model from the preset artificial intelligence framework, using the inference acceleration engine for optimization processing, analyzing the calculation graph to determine the sub-computation graph that meets the operator fusion conditions, and performing operator fusion processing on the target operator. Generate fusion operators to optimize model inference performance.
The number of operator calls and the number of read and writes of on-chip memory space is reduced, and the time of the model reasoning process is significantly reduced.
Smart Images

Figure CN115034402B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device and related products for optimizing model reasoning performance. Background Art
[0002] With the development of artificial intelligence technology, deep learning models are widely used. Deep learning models can be used for data processing such as data classification and target detection. When using deep learning models for data processing, an accelerator card is generally used in combination with a supporting software stack. Among them, there is an inference acceleration engine in the software stack. The inference acceleration engine is used to receive model files exported by various artificial intelligence frameworks. After optimizing them, it can generate corresponding host-side and device-side binary codes for various accelerator cards.
[0003] Currently, when the inference acceleration engine optimizes the model file, it first converts the model file exported by the artificial intelligence framework into an intermediate representation, and then performs hardware-independent and hardware-related optimizations on the computational graph of the intermediate representation to obtain the optimized computational graph.
[0004] However, when the inference acceleration engine optimizes the model file, it does not fully optimize the hardware-related optimization links, resulting in a longer time for reading and writing data in the on-chip memory space, which in turn leads to a longer model inference process. Summary of the invention
[0005] The embodiments of the present application provide a method, device and related products for optimizing model reasoning performance, so as to solve the technical problem that the hardware-related optimization links in the prior art are not fully optimized, resulting in a long time for reading and writing data in the on-chip memory space, which in turn leads to a long time for the model reasoning process.
[0006] In a first aspect, the present application provides a method for optimizing model reasoning performance, comprising:
[0007] Obtain an initial model file of the target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model;
[0008] The initial model file is optimized using a preset inference acceleration engine to obtain an automatically optimized calculation graph;
[0009] Analyzing the automatically optimized computation graph to determine at least one sub-computation graph that satisfies the operator fusion condition; the operator in the sub-computation graph is the target operator;
[0010] For at least one sub-computation graph, perform operator fusion processing on a target operator in the same sub-computation graph to obtain at least one fused operator;
[0011] The inference performance of the target model is optimized based on at least one fusion operator.
[0012] In a second aspect, the present application provides a device for optimizing model reasoning performance, comprising:
[0013] An acquisition module, used to acquire an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model;
[0014] An automatic optimization module, used to optimize the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph;
[0015] A determination module, used to analyze the automatically optimized computation graph and determine at least one sub-computation graph that meets the operator fusion condition; the operator in the sub-computation graph is the target operator;
[0016] A fusion processing module, used for performing operator fusion processing on a target operator in the same sub-computation graph for at least one sub-computation graph, so as to obtain at least one fused operator;
[0017] The fusion optimization processing module optimizes the reasoning performance of the target model based on at least one fusion operator.
[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;
[0019] The memory stores computer-executable instructions;
[0020] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.
[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the method described in the first aspect is implemented.
[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.
[0023] The optimization method, device, and related products for model reasoning performance provided by the embodiments of the present application obtain an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model; the initial model file is optimized using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph; the automatically optimized calculation graph is parsed to determine at least one sub-calculation graph that meets the operator fusion condition; the operator in the sub-calculation graph is the target operator; for at least one sub-calculation graph, the target operator in the same sub-calculation graph is subjected to operator fusion processing to obtain at least one fused operator; the reasoning performance of the target model is optimized based on the at least one fused operator; due to the target operator in the at least one sub-calculation graph being subjected to operator fusion processing, the target operator in the sub-calculation graph is subjected to operator fusion processing to obtain at least one fused operator; Perform operator fusion processing to obtain at least one fused operator. The target operator in the same sub-computation graph meets the operator fusion condition. Therefore, the operator fusion processing at least fuses two target operators into one fused operator. Then, the reasoning performance of the target model is optimized based on at least one fused operator. Before optimization, it is necessary to call each target operator in the same sub-computation graph, but after optimization, it is only necessary to call the fused operator corresponding to the sub-computation graph, which reduces the number of operator calls, and thus reduces the reading of data from the on-chip memory space each time the operator is called. After the operator calculation is completed, the processed data is written to the on-chip memory space, which effectively reduces the number of reads and writes of data in the on-chip memory space. Therefore, it can effectively reduce the time for reading and writing data in the on-chip memory space, thereby reducing the time of the model reasoning process. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0025] Figure 1 A network architecture diagram of a method for optimizing model reasoning performance provided in an embodiment of the present application;
[0026] Figure 2 A flowchart of a method for optimizing model reasoning performance provided in Example 1 of the present application;
[0027] Figure 3 A flowchart of a method for optimizing model reasoning performance provided in Example 2 of the present application;
[0028] Figure 4 A flowchart of a method for optimizing model reasoning performance provided in Example 3 of the present application;
[0029] Figure 5 A flow chart of a method for optimizing model reasoning performance provided in Example 4 of the present application;
[0030] Figure 6A flowchart of a method for optimizing model reasoning performance provided in Example 5 of the present application;
[0031] Figure 7 A flowchart of a method for optimizing model reasoning performance provided in Example 6 of the present application;
[0032] Figure 8 A flowchart of a method for optimizing model reasoning performance provided in Example 7 of the present application;
[0033] Figure 9a A schematic diagram of the structure of a portion of a target model in the method for optimizing model reasoning performance provided in Example 8 of the present application;
[0034] Figure 9b A schematic diagram of the structure of the first branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application;
[0035] Fig.9c A schematic diagram of the structure of the second branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application;
[0036] Figure 9d A schematic diagram of the structure of the third branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application;
[0037] Fig.9e A final optimized computational graph corresponding to a portion of a target model in the method for optimizing model reasoning performance provided in Example 8 of the present application;
[0038] Fig.10 A schematic diagram of the structure of a device for optimizing model reasoning performance provided in Example 10 of the present application;
[0039] Fig.11 This is a schematic diagram of the structure of an electronic device provided in Example 11 of the present application.
[0040] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0042] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.
[0043] When using deep learning models for data processing, the software stack that comes with the accelerator card has an inference acceleration engine. The inference acceleration engine is used to receive model files exported by the artificial intelligence framework, optimize them, and generate corresponding host-side and device-side binary codes for a variety of accelerator cards.
[0044] At present, the operation process of the inference acceleration engine is mainly divided into three parts: the front-end is responsible for supporting different languages and parsing program files written in different languages into a unified intermediate representation; the middle part is responsible for hardware-independent optimization of the intermediate representation; the back-end is responsible for adding hardware information, performing hardware-related optimization on the optimized intermediate representation, and then generating binary code.
[0045] At present, the inference acceleration engine does not fully optimize the model file when optimizing it. In the calculation graph automatically optimized by the inference acceleration engine, all operator call processes are single operators. Each operator call only performs one calculation, and the input data needs to be read from the DDR to the on-chip storage space, and the on-chip computing resources are used for calculation, and then the output data is stored in the corresponding position on the DDR. This means that when the model is inferring, the on-chip memory space needs to communicate with the DDR multiple times to read and write data, which results in a longer data reading and writing time, which in turn leads to a longer model inference process.
[0046] Therefore, when facing the technical problems of the prior art, the inventors found through creative research that in order to reduce the time for reading and writing data in the on-chip memory space, and thus reduce the time of the model reasoning process, it is necessary to reduce the number of operator calls to avoid the operator call process in the model reasoning process being all single operators. Therefore, the inventors proposed the solution of this application: obtaining the initial model file of the target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model; optimizing the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph; parsing the automatically optimized calculation graph to determine at least one sub-calculation graph that meets the operator fusion conditions; the operator in the sub-calculation graph is the target operator; for at least one sub-calculation graph, the target operator in the same sub-calculation graph is subjected to operator fusion processing to obtain at least one fused operator; optimizing the reasoning performance of the target model based on at least one fused operator. Since the target operator in at least one sub-computation graph is subjected to operator fusion processing to obtain at least one fused operator, and the target operator in the same sub-computation graph meets the operator fusion condition, the operator fusion processing at least fuses two target operators into one fused operator, and then optimizes the reasoning performance of the target model based on at least one fused operator. Before the optimization, it is necessary to call each target operator in the same sub-computation graph, but after the optimization, it is only necessary to call the fused operator corresponding to the sub-computation graph, thereby reducing the number of operator calls, and thus reducing the reading of data from the on-chip memory space each time the operator is called, and after the operator calculation is completed, the processed data is written to the on-chip memory space, that is, the number of reading and writing of data in the on-chip memory space is effectively reduced, so the time for reading and writing data in the on-chip memory space can be effectively reduced, thereby reducing the time of the model reasoning process.
[0047] Therefore, the inventors have proposed a technical solution of an embodiment of the present invention based on the above creative discovery. The following specific embodiments are used to describe in detail how the technical solution of the present invention and the technical solution of the present application solve the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.
[0048] Figure 1 A network architecture diagram of a method for optimizing model reasoning performance provided in an embodiment of the present application, such as Figure 1 As shown, a network architecture diagram corresponding to an application scenario provided by an embodiment of the present application includes: an accelerator card 11 and a software stack 12. The software stack 12 is a software stack supporting the accelerator card 11, and the software stack 12 is configured with an inference acceleration engine 51 and an artificial intelligence framework 52, and the artificial intelligence framework 52 stores the initial model file of the target model.
[0049] The accelerator card 11 may be a DLP accelerator card, and illustratively, may adopt a DLParch02 architecture or a DLParch03 architecture. Figure 1 The DLP accelerator card in the figure is based on the DLParch02 architecture. Figure 1 As shown, it includes a double data rate synchronous dynamic random access memory DDR 21 and at least one cluster 22. Multiple clusters in the accelerator card 11 share the DDR, and each cluster can communicate with the DDR. DDR is a double data rate synchronous dynamic random access memory, the English name is Double Data Rate synchronous dynamic random-access memory, referred to as DDR SDRAM or DDR.
[0050] The cluster 22 includes a synchronous dynamic random access memory SRAM 31 and at least one processor core 32. Multiple processor cores in each cluster share one SRAM. The SRAMs of each cluster can communicate with each other. The processor cores in the cluster cannot communicate directly. Among them, SRAM is a static random access memory, the English name is Static Random-Access Memory, referred to as SRAM.
[0051] The processor core 32 includes a computing unit 41, a register 42, a non-volatile random access memory NRAM 43, and a window random access memory WRAM 44. The register 42, the non-volatile random access memory NRAM 43 and the window random access memory WRAM 44 are used to store scalars, neurons and weights respectively. Among them, WRAM is a window random access memory, the English name is Window Random Access Memory, abbreviated as WRAM. The computing unit 41 can be a Neuron Function Unit, abbreviated as NFU.
[0052] The software stack 12 includes an inference acceleration engine 51, which can receive an initial model file of a target model, which can be obtained from a preset artificial intelligence framework 52, and the initial model file has an initial calculation graph corresponding to the target model. The inference acceleration engine 51 can optimize the initial model file to obtain an automatically optimized calculation graph; parse the automatically optimized calculation graph to determine at least one sub-calculation graph that meets the operator fusion condition; the operator in the sub-calculation graph is the target operator; for at least one sub-calculation graph, the target operator in the same sub-calculation graph is subjected to operator fusion processing to obtain at least one fusion operator; and optimize the inference performance of the target model based on at least one fusion operator. After the inference performance of the target model is optimized, binary code can be generated and deployed on the acceleration card 11. So that the acceleration card 11 can call its processor core according to the binary code to implement model inference.
[0053] Embodiment 1
[0054] Figure 2 This is a flow chart of a method for optimizing model reasoning performance provided in Example 1 of the present application, such as Figure 2 As shown, the execution subject of the present application is a device for optimizing model reasoning performance, and the device is located in an electronic device. The method for optimizing model reasoning performance provided in this embodiment includes steps 201 to 205.
[0055] Step 201, obtaining an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file contains an initial calculation graph corresponding to the target model.
[0056] In this embodiment, the program file for executing the target model can be input into a preset artificial intelligence framework. The preset artificial intelligence framework can generate a corresponding initial calculation graph of the target model based on the target model, and then obtain the initial model file of the target model.
[0057] Among them, the artificial intelligence framework can be TensorFlow, PyTorch, Caffe, ONNX, etc. The artificial intelligence framework stores the initial model file of the target model.
[0058] The target model can be a network model of the YOLO series, such as a YOLOv3 network, a YOLOv4 network, a YOLOv5 network, or a network that improves the feature extraction network and / or feature enhancement network of the YOLOv5 network. The YOLOv3 network is an algorithm proposed by Redmon et al. in 2018. The YOLOv4 network is an algorithm proposed by Bochkovskiy et al. in 2020. The YOLOv5 network is an algorithm proposed by Jocher et al. on GitHub in 2021, including YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x according to the network depth.
[0059] In this embodiment, the initial model file contains an initial calculation graph corresponding to the target model. The initial calculation graph includes nodes and directed edges, and the nodes are connected by directed edges.
[0060] In the initial calculation graph, each node is an operator, and the operator corresponds to a mathematical operation function. For example, the operator can correspond to addition, subtraction, multiplication, division, square, matrix rank conversion, etc. In other words, the mathematical operation function corresponding to the operator is the functional relationship between the input data and output data of the node.
[0061] The directed edge corresponds to the data transfer relationship between the nodes, showing the data flow direction of the target model during execution. For example, a directed edge from node A to node B indicates that the output data of node A is transferred to node B as the input data of node B.
[0062] Step 202: Use a preset inference acceleration engine to optimize the initial model file to obtain an automatically optimized calculation graph.
[0063] In this embodiment, the inference acceleration engine can read the initial calculation graph in the initial model file through the MLIR graph compilation technology, and convert the initial calculation graph in the initial model file into an intermediate representation; and can perform hardware-independent optimizations such as shape reasoning and constant propagation on the initial calculation graph of the intermediate representation, as well as hardware-related optimizations such as software pipelining. The automatically optimized calculation graph is the calculation graph after the inference acceleration engine optimizes the initial calculation graph in the initial model file. Among them, the intermediate representation is the abbreviation of multi-level intermediate representation, and its English name is Multi-Level Intermediate Representation, abbreviated as MLIR.
[0064] Among them, shape inference optimization refers to eliminating operators that obtain the shape of a tensor and / or operators that reset the shape of a tensor. For example, the operator that obtains the shape of a tensor may be a shape operator; the operator that resets the shape of a tensor may be a Reshape operator.
[0065] The inference acceleration engine can calculate the input scale of each operator in the initial calculation graph layer by layer according to the specific input scale or scale range that the user wants to use the target model for inference, and according to the mathematical operation function corresponding to the operator, without performing specific calculations, so as to eliminate the operators that obtain the tensor shape and / or reset the tensor shape in the initial calculation graph, so as to optimize the initial calculation graph. For operators that need to be calculated, the calculation logic inside the operator can be optimized through constant propagation.
[0066] Constant propagation optimization means simplifying variables that can calculate results into constants during the compilation process, and propagating the simplified constants to all places where the variables are used for further judgment.
[0067] When performing constant propagation optimization, the operators in the initial calculation graph need to support all preset input scales and all underlying hardware that the operator is planned to support. Therefore, the operators in the initial calculation graph need to be optimized differently for each combination of preset input scales and underlying hardware. This makes the operators in the initial calculation graph correspond to different mathematical operation functions at different preset input scales, resulting in code expansion. In the process of using an operator in a specific input model or a specific layer of an input model, only one or several code paths of the operator in the calculation library are required. The inference acceleration engine can pass the input scale of each operator obtained by shape reasoning into the operator calculation logic to obtain the calculation path required under the input scale, eliminating redundant code except for the code corresponding to the input scale.
[0068] Software pipelining optimization refers to the optimization of the operator's computational process.
[0069] During the operator calling process, since the capacity of the on-chip storage space is limited and is used to store the scalars, neurons, and weights of the target model, the input data and output data when calling the operator are stored in the memory of the accelerator card, illustratively, in the DDR of the accelerator card. Therefore, the calculation process of calling the operator is: read the input data from the accelerator card memory to the on-chip storage space; use the on-chip computing resources for calculation; store the output data in the corresponding location on the accelerator card memory. The calculation process of calling the operator includes two parts: input and output (IO for short) and calculation, and the hardware resources required for these two parts are different. IO needs to use IO resources, and calculation needs to use computing resources.
[0070] When the hardware architecture supports out-of-order execution, the IO and calculation parts can be run in parallel by adjusting the instruction order to save running time and optimize the operator calculation process. For example, for operators conv and relu, the input data required by relu can be read while performing conv calculations; while running relu calculations, the calculation results of conv can be stored. Software pipelining can optimize the calculation graph through instruction-level parallelism to ensure that both on-chip computing resources and IO resources are used.
[0071] In this embodiment, a preset inference acceleration engine can be used to perform any one or more of shape inference optimization, constant propagation optimization, and software pipeline optimization on the initial model file to obtain an automatically optimized calculation graph.
[0072] Step 203, analyze the automatically optimized computation graph to determine at least one sub-computation graph that meets the operator fusion condition; the operator in the sub-computation graph is the target operator.
[0073] In this embodiment, the automatically optimized computation graph includes nodes and directed edges, each node is an operator. The sub-computation graph includes at least two target operators. At least one sub-computation graph that meets the operator fusion condition can be determined based on at least one of the network structure between the operators in the automatically optimized computation graph, the input data feature, and the output data feature.
[0074] The network structure refers to the connection relationship between operators in the computation graph. Input data features include input data shape and scale. Output data features include output data shape and scale.
[0075] Among them, data shape refers to the dimension of the input data, the length of each dimension, and the connection order of each dimension. For example, when the input data is a tensor of 48×27×96×96, the dimension of the input data is 4, and the lengths of the first, second, third and fourth dimensions are 48, 27, 96 and 96 respectively, and the connection order of the four dimensions is the first dimension, the second dimension, the third dimension and the fourth dimension.
[0076] Exemplarily, at least two operators with the same input data shape may be determined as a sub-computation graph. Alternatively, directed edges and operators between input data and output data with the same shape may be determined as a sub-computation graph. Alternatively, directed edges and operators between input data and output data with the same size may be determined as a sub-computation graph.
[0077] Step 204: for at least one sub-computation graph, perform operator fusion processing on the target operator in the same sub-computation graph to obtain at least one fused operator.
[0078] In this embodiment, the input data features, output data features or mathematical operation functions corresponding to each target operator of the sub-computation graph can be analyzed to determine the function of the sub-computation graph. Then, the target operators in the same sub-computation graph are fused, and the function of the sub-computation graph is realized by a fused operator, and the input data features and output data features of the fused operator are the same as those of the sub-computation graph.
[0079] For example, the subcomputation graph includes operators slice0, slice1, and slice2. The input data shapes of slice0, slice1, and slice2 are all 48×3×96×96×9, the output data shapes of slice0 are 48×3×96×96×2, the output data shapes of slice1 are 48×3×96×96×2, and the output data shapes of slice2 are 48×3×96×96×5. The function of the graph is to split the input data of the shape of 48×3×96×96×9 into three parts of the shapes of 48×3×96×96×2, 48×3×96×96×2 and 48×3×96×96×5. Therefore, a fusion operator can be used to directly split the input data of the size of 48×3×96×96×9 into three parts of the sizes of 48×3×96×96×2, 48×3×96×96×2 and 48×3×96×96×5. Furthermore, in the process of target model inference, the functions implemented by the three operator calls of slice0, slice1 and slice2 can be realized by calling the fusion operator once.
[0080] Step 205: Optimize the reasoning performance of the target model based on at least one fusion operator.
[0081] In this embodiment, at least one sub-computation graph in the automatically optimized calculation graph can be replaced with a corresponding fusion operator. The calculation graph after the sub-computation graph is replaced with the corresponding fusion operator is the final optimized calculation graph. The inference acceleration engine can generate a binary model file based on the final optimized calculation graph to complete the optimization of the inference performance of the target model.
[0082] The method for optimizing model reasoning performance provided in this embodiment obtains an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model; optimizes the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph; parses the automatically optimized calculation graph to determine at least one sub-calculation graph that meets the operator fusion condition; the operator in the sub-calculation graph is the target operator; for at least one sub-calculation graph, performs operator fusion processing on the target operator in the same sub-calculation graph to obtain at least one fused operator; optimizes the reasoning performance of the target model based on at least one fused operator; and obtains At least one fusion operator, the target operator in the same sub-computation graph meets the operator fusion condition, therefore, the operator fusion processing at least fuses two target operators into one fusion operator, and then, based on at least one fusion operator, the inference performance of the target model is optimized, before the optimization, it is necessary to call each target operator in the same sub-computation graph, and after the optimization, it is only necessary to call the fusion operator corresponding to the sub-computation graph, which reduces the number of operator calls, and thus reduces the reading of data from the on-chip memory space each time the operator is called, and after the operator calculation is completed, the processed data is written to the on-chip memory space, that is, the number of reading and writing of data in the on-chip memory space is effectively reduced, so it can effectively avoid reducing the time for reading and writing data in the on-chip memory space, thereby reducing the time of the model inference process.
[0083] Embodiment 2
[0084] Figure 3 This is a flow chart of a method for optimizing model reasoning performance provided in Example 2 of the present application, such as Figure 3 As shown, the method for optimizing model reasoning performance provided in this embodiment is based on the first embodiment. For step 202, a preset reasoning acceleration engine is used to optimize the initial model file to obtain an automatically optimized calculation graph, and then the step 202 refinement includes steps 301 to 302.
[0085] Step 301, optimize the initial model file using the optimization strategy pre-configured in the preset reasoning acceleration engine.
[0086] In this embodiment, the pre-configured optimization strategy may include any one or more of shape inference optimization, constant propagation optimization, and software pipelining optimization.
[0087] Step 302: Check the optimization log after the model file is optimized, and obtain the automatically optimized calculation graph from the optimization log.
[0088] When optimizing the initial model file using the optimization strategy pre-configured in the preset inference acceleration engine, the inference acceleration engine generates an optimization log. The optimization log records the operator calls of the initial model file after optimization by the optimization strategy. By viewing the operator calls recorded in the optimization log, you can obtain the automatically optimized calculation graph.
[0089] The method for optimizing model reasoning performance provided in this embodiment optimizes the initial model file by adopting the optimization strategy pre-configured in the preset reasoning acceleration engine; checking the optimization log after the optimization of the model file, and obtaining the automatically optimized calculation graph from the optimization log; since the automatically optimized calculation graph is obtained from the optimization log, the optimization log records the operator calls after the optimization strategy optimizes the initial model file, and therefore, an accurate automatically optimized calculation graph can be obtained, ensuring the reliability and accuracy of subsequent steps such as parsing the automatically optimized calculation graph.
[0090] Embodiment 3
[0091] Figure 4 This is a flow chart of a method for optimizing model reasoning performance provided in Example 3 of the present application, such as Figure 4 As shown, the method for optimizing model reasoning performance provided by this embodiment, based on any one of the above embodiments, analyzes the automatically optimized calculation graph in step 203, determines at least one sub-calculation graph that meets the operator fusion condition, and refines it, then the refinement of step 203 includes steps 401 to 403.
[0092] Step 401, analyze the automatically optimized computation graph to obtain the network structure, input data features and output data features between operators before fusion.
[0093] In this embodiment, the operator before fusion refers to the operator in the automatically optimized calculation graph. The network structure between operators before fusion includes: the connection order of operators in the automatically optimized calculation graph before melting. Exemplarily, the operator conv is connected to the operator transpose through a directed line, the operator transpose is connected to the operator sigmoid through a directed line, and the output data of the operator conv is the output data of the operator transpose, and the output data of the operator transpose is the input data of the operator sigmoid, then the continuous order between the three operators is conv, transpose and sigmoid.
[0094] The input data features between the operators before fusion include input data shape features and input data scale. The output data between the operators before fusion include output data shape features and output data scale.
[0095] In this embodiment, the network structure between the operators before fusion can be determined by the direction of the directed lines in the automatically optimized calculation graph. In addition, the input data features and output data features between the operators before fusion can be determined according to the input data shape and input data scale of the automatically optimized calculation graph and the mathematical operation functions corresponding to the operators before fusion.
[0096] Exemplarily, the input data shape of the automatically optimized calculation graph is 3×9×3, and the input data of the automatically optimized calculation graph is respectively input into the first pre-fusion operator and the second pre-fusion operator, and the input data shape of the first pre-fusion operator and the second pre-fusion operator are both 3×9×3. The mathematical operation function corresponding to the first pre-fusion operator and the second pre-fusion operator is to take out the data of the first 4 lengths and the last 5 lengths from the second dimension of the input data, respectively, and the output data shapes of the first pre-fusion operator and the second pre-fusion operator are 3×4×3 and 3×5×3, respectively.
[0097] Step 402, determining at least one group of operators that meet operator fusion conditions according to the network structure between operators before fusion, input data characteristics, and output data characteristics.
[0098] In this embodiment, for a pre-fusion operator, the output data features of the pre-fusion operator can be compared with the input data features of any other pre-fusion operator or multiple fusion operators with the same operator name in the automatically optimized calculation graph. If the output data features of the pre-fusion operator are the same as the input data features of any other pre-fusion operator, and the directions of the directed lines between the two pre-fusion operators are the same, the two pre-fusion operators and the remaining pre-fusion operators connected between them can be determined as a group of operators that meet the operator fusion conditions.
[0099] It is understandable that there may be other ways to determine at least one group of operators that meet the operator fusion conditions based on the network structure between the operators before fusion, the input data characteristics, and the output data characteristics, which is not limited in this embodiment.
[0100] Step 403: Determine at least one sub-computation graph based on at least one group of operators that meet the operator fusion condition.
[0101] In this embodiment, when there are multiple groups of operators that meet the operator fusion conditions in the initial calculation graph, the multiple groups of operators that meet the operator fusion conditions can be determined as one sub-calculation graph, or the multiple groups of operators that meet the operator fusion conditions can be determined as multiple sub-calculation graphs respectively.
[0102] The method for optimizing the model reasoning performance provided in this embodiment analyzes the automatically optimized calculation graph to obtain the network structure, input data characteristics and output data characteristics between the operators before fusion; determines at least one group of operators that meet the operator fusion conditions according to the network structure, input data characteristics and output data characteristics between the operators before fusion; determines at least one sub-calculation graph based on at least one group of operators that meet the operator fusion conditions; because the operators that meet the operator fusion conditions are determined according to the network structure, input data characteristics and output data characteristics between the operators before fusion, it can be ensured that the operators that meet the fusion conditions will not affect the implementation function of the calculation graph after fusion, and at the same time, at least one group of operators that meet the operator fusion conditions can be flexibly determined, and then, the reasoning performance of the target model can be selectively optimized.
[0103] Embodiment 4
[0104] Figure 5 This is a flow chart of a method for optimizing model reasoning performance provided in Example 4 of the present application, such as Figure 5 As shown, the method for optimizing model reasoning performance provided by this embodiment, based on any one of the above embodiments, performs operator fusion processing on the target operator in the same sub-computation graph for step 204, for at least one sub-computation graph, to obtain at least one fused operator, and refines it, then the refinement of step 204 includes steps 501 to 502.
[0105] Step 501, for at least one sub-computation graph, obtain the network structure between each target operator in the same sub-computation graph, the implementation function of each target operator and the input and output data shape characteristics of the sub-computation graph.
[0106] In this embodiment, the implementation function of the target operator refers to the mathematical operation performed by the target operator to transform input data into output data.
[0107] Step 502: Generate at least one fusion operator based on the network structure, various implementation functions and input and output data shape characteristics of at least one sub-computation graph.
[0108] In this embodiment, each sub-computation graph can generate a fusion operator. If there are multiple sub-computation graphs in the automatically optimized calculation graph, at least one fusion operator can be generated. The mathematical operation functions performed by each target operator in the sub-computation graph on the input data of the sub-computation graph can be determined in sequence, and according to the connection order between the target operators in the sub-computation graph, all the target operators in the sub-computation graph can be generated into a fusion operator by function nesting or function composition to ensure that the implementation function of the fusion operator is the same as the implementation function of the sub-computation graph, and the input and output data characteristics of the fusion operator are the same as the input and output data characteristics of the sub-computation graph.
[0109] The method for optimizing model reasoning performance provided in this embodiment obtains the network structure between each target operator in the same sub-computation graph, the implementation function of each target operator and the input and output data shape characteristics of the sub-computation graph for at least one sub-computation graph; generates at least one fusion operator according to the network structure, each implementation function and the input and output data shape characteristics of the sub-computation graph in at least one sub-computation graph; since at least one fusion operator is generated according to the network structure, each implementation function and the input data shape characteristics of the sub-computation graph in at least one sub-computation graph, it can be ensured that the fusion operator implements the implementation function of the sub-computation graph, and the input and output data characteristics are the same as the sub-computation graph, thereby ensuring that the implementation function of the target operator in the sub-computation graph after fusion is the same as the sub-computation graph.
[0110] Embodiment 5
[0111] Figure 6 This is a flowchart of a method for optimizing model reasoning performance provided in Example 5 of the present application, such as Figure 6 As shown, the method for optimizing model reasoning performance provided by this embodiment, on the basis of the fourth embodiment, generates at least one fusion operator for step 502 according to the network structure in at least one sub-computation graph, each implementation function and the input and output data shape characteristics of the sub-computation graph, and refines the step 502, and the refinement of step 502 includes steps 601 to 603.
[0112] Step 601, generating kernel functions of fusion operators according to the network structure and various implementation functions in at least one sub-computation graph.
[0113] In this embodiment, the fusion operator includes a kernel function, a shape reasoning function, a scope, and an operator identifier.
[0114] The kernel function refers to the mathematical operation function corresponding to the fusion operator.
[0115] In this embodiment, the mathematical operation functions corresponding to the target operators in the sub-computation graph may be compounded according to the network structure between the target operators to generate a kernel function of the fusion operator.
[0116] Step 602: Generate a shape inference function of a fusion operator according to shape features of input and output data of at least one sub-computation graph.
[0117] The shape inference function refers to the functional relationship between the output data shape of the operator and the input data shape. The shape inference function can be used to determine the output data shape of the operator according to the input data shape of the operator.
[0118] In this embodiment, the shape reasoning function of each target operator in the sub-computation graph can be compounded according to the network structure between the target operators to generate the shape reasoning function of the fusion operator. The shape reasoning function corresponding to each target operator can be obtained from a preset calculation library. The preset calculation library stores the shape reasoning function corresponding to the target operator. The shape reasoning function of the target operator includes the functional relationship between the output data shape of the target operator and all preset input data shapes.
[0119] Step 603: Determine the scope and operator identifier of at least one fusion operator.
[0120] In this embodiment, the scope of the fusion operator can be determined according to the scope of each target operator in the sub-computation graph. For example, the scope of the fusion operator can be the sum of the scopes of all target operators in the sub-computation graph. The operator identifier of the fusion operator can be determined according to the implementation function of the sub-computation graph.
[0121] The method for optimizing model reasoning performance provided in this embodiment generates kernel functions of fusion operators according to the network structure and each implementation function in at least one sub-computation graph; generates a shape reasoning function of the fusion operator according to the shape characteristics of the input and output data of the sub-computation graph in at least one sub-computation graph; determines the scope and operator identification of at least one fusion operator; since the kernel function of the fusion operator is generated according to the network structure and implementation function in the sub-computation graph, the fusion operator can realize the implementation function of the sub-computation graph, and generates the shape reasoning function of the fusion operator according to the shape characteristics of the input and output data of the sub-computation graph. The shape characteristics of the input and output data of the fusion operator are the same as those of the sub-computation graph. Therefore, the fusion operator can replace the sub-computation graph in the optimized calculation graph to optimize the model reasoning performance.
[0122] Embodiment 6
[0123] Figure 7 This is a flowchart of a method for optimizing model reasoning performance provided in Example 6 of the present application, such as Figure 7 As shown, the method for optimizing model reasoning performance provided by this embodiment is based on any one of the above embodiments. Step 205 optimizes the reasoning performance of the target model based on at least one fusion operator, and also includes steps 701 to 702.
[0124] Step 701: register the scope and operator identifier of at least one fusion operator in a preset artificial intelligence framework.
[0125] In this embodiment, the program code implementing the fusion operator can be first compiled into a dynamic library using the inference acceleration engine, and then the custom operator registration function provided by the artificial intelligence framework can be used to load the compiled dynamic library of the fusion operator to register the scope and operator identifier of the fusion operator. After successful registration, the calculation graph in the model file exported from the artificial intelligence framework includes the fusion operator.
[0126] Step 702: register the scope, operator identifier, kernel function and shape reasoning function of at least one fusion operator in a preset reasoning acceleration engine, and compile and generate a dynamic library of at least one fusion operator.
[0127] In this embodiment, after determining the scope, operator identifier, kernel function and shape reasoning function of the fusion operator, the scope, operator identifier, kernel function and shape reasoning function of the fusion operator can be registered in a preset inference acceleration engine, and the inference acceleration engine can be used to compile the program code that implements the fusion operator into a dynamic library, so that the inference acceleration engine can call the kernel function and shape reasoning function of the fusion operator through the operator identifier of the fusion operator to execute the implementation function of the fusion operator.
[0128] The method for optimizing model reasoning performance provided in this embodiment is to register the scope and operator identifier of at least one fusion operator in a preset artificial intelligence framework; register the scope, operator identifier, kernel function and shape reasoning function of at least one fusion operator in a preset reasoning acceleration engine, and compile and generate a dynamic library of at least one fusion operator; since at least one fusion operator is registered in the preset artificial intelligence framework, the preset artificial intelligence framework can be used to export a computational graph including the fusion operator; at the same time, at least one fusion operator is registered in the preset reasoning acceleration engine, so that the reasoning acceleration engine can call the fusion operator through the operator identifier during reasoning.
[0129] Embodiment 7
[0130] Figure 8 This is a flow chart of a method for optimizing model reasoning performance provided in Example 7 of the present application, such as Figure 8 As shown, the method for optimizing model reasoning performance provided in this embodiment, on the basis of embodiment six, optimizes the reasoning performance of the target model based on at least one fusion operator in step 205, and the refinement of step 205 includes steps 801 to 804.
[0131] Step 801, obtaining a model file after operator fusion corresponding to a target model from a preset artificial intelligence framework, wherein the model file after operator fusion includes a computational graph having at least one fusion operator.
[0132] In this embodiment, after registering the fusion operator in the preset artificial intelligence framework, the model file after the operator fusion corresponding to the target model can be directly obtained from the preset artificial intelligence framework. The model file after the operator fusion includes a calculation graph with at least one fusion operator, and the number of calculation graphs with at least one fusion operator included corresponds to the number of fusion operators registered in the artificial intelligence framework.
[0133] Step 802: Use a preset inference acceleration engine to optimize the model file after the operator fusion to obtain a final optimized calculation graph.
[0134] In this embodiment, in the model file after the operator fusion, other operators except the fusion operator have not been optimized, and the fusion operator also needs to correspond to different mathematical operation functions at different preset input scales, and has codes corresponding to different preset input scales. Therefore, the preset reasoning acceleration engine can be used to optimize the model file after the operator fusion by shape reasoning, constant propagation, software pipelining, etc., to obtain the final optimized calculation graph.
[0135] Step 803: Generate a binary code file corresponding to the final optimized computation graph, and input the binary code file and a dynamic library of at least one fusion operator into the accelerator card.
[0136] In this embodiment, the inference acceleration engine can generate a corresponding binary code file based on the network structure between the operators in the final optimized calculation graph and the mathematical operation function corresponding to each operator. In addition, the binary code file and the dynamic library of at least one fusion operator can be input into the accelerator card. Exemplarily, the binary code file storage and the dynamic library of at least one fusion operator can be stored in the DDR of the accelerator card.
[0137] Step 804, controlling the accelerator card to perform the inference process of the target model according to the binary code file and the dynamic library of at least one fusion operator.
[0138] In this embodiment, the device for optimizing model reasoning performance can control the acceleration card to execute binary code and a dynamic library of at least one fusion operator to perform the reasoning process of the target model and output the reasoning result of the target model.
[0139] The optimization method of model reasoning performance provided in this embodiment is to obtain the operator fused model file corresponding to the target model from the preset artificial intelligence framework, and the operator fused model file includes a calculation graph with at least one fused operator; use the preset reasoning acceleration engine to optimize the operator fused model file to obtain the final optimized calculation graph; generate the binary code file corresponding to the final optimized calculation graph, and input the binary code file and the dynamic library of at least one fused operator into the acceleration card; control the acceleration card to perform the reasoning process of the target model according to the binary code file and the dynamic library of at least one fused operator; because the operator fused model file includes a calculation graph with at least one fused operator, the fused operator can reduce the time for reading and writing data to the on-chip memory space during the reasoning process. At the same time, the fused operator is an operator fused by multiple target operators. When the preset reasoning acceleration is used for optimization, the calculation graph with at least one fused operator can be optimized by shape reasoning, constant propagation and software pipelining, and the implementation function of the fused operator will not be changed. Therefore, the binary code corresponding to the final optimized computational graph generated by the inference acceleration engine can reduce the time for reading and writing data in the on-chip memory space during the model inference process. Furthermore, controlling the acceleration card to perform the inference process of the target model according to the binary code file and the dynamic library of at least one fusion operator can reduce the inference time of the target model.
[0140] Embodiment 8
[0141] Optionally, the optimization method of model reasoning performance provided by this embodiment is based on any of the above embodiments, the target model is a YOLOv5s network model or a model that improves the YOLOv5s network, and the target model is a target detection model; the sub-computation graph is a sub-computation graph corresponding to the final processing process of the prediction box generated after the multi-scale prediction box reasoning is completed. Specifically, Figure 9a This is a schematic diagram of the structure of a portion of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application, Figure 9b A schematic diagram of the structure of the first branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application; Fig.9c A schematic diagram of the structure of the second branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application; Figure 9d A schematic diagram of the structure of the third branch in a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application; Fig.9e A calculation diagram of the final optimization corresponding to a part of the target model in the method for optimizing model reasoning performance provided in Example 8 of the present application is shown.
[0142] Among them, part of the YOLOv5s network is: after the YOLOv5s network completes the reasoning of the three different multi-scale prediction boxes, it generates the final processing process of the prediction box. Part of the YOLOv5s network includes a first branch for processing the prediction box of the first scale, a second branch for processing the prediction box of the second scale, and a third branch for processing the prediction box of the third scale. Part of the YOLOv5s network includes operators conv, reshpae, transpose, sigmod, mul, sub, add, pow, concat and output, and the corresponding mathematical operation functions are convolution, reshape, rank conversion, activation, multiplication, subtraction, addition, square, string concatenation and output.
[0143] In this embodiment, the output data shape of the operator concat is 48×36288×9, the input data shape of the first branch is 48×3×96×96×9, the input data shape of the second branch is 48×3×48×48×9, and the input data shape of the first branch is 48×3×24×24×9. Since 36288=3×96×96+3×48×48+3×24×24, and the directed lines between the operators in the first branch, the second branch, and the third branch and the operator concat have only one direction, the operator concat and the operators in the first branch, the second branch, and the third branch meet the operator fusion condition. According to the network structure and input and output data characteristics between the operator concat and the operators in the first branch, the second branch, and the third branch, the operator concat and the operators in the first branch, the second branch, and the third branch are fused to obtain the final optimized calculation graph. The final optimized calculation graph is as shown in FIG. Fig.9e shown.
[0144] In this embodiment, the product specifications of the accelerator card may be shown in Table 1 below.
[0145] In this embodiment, the YOLOv5s network model can be in ONNX format. ONNX is a file format for storing trained models. The preset artificial intelligence framework can be PyTorch, which is an artificial intelligence framework developed by Paszke et al. in 2019. It can be used to read or store the parameters of the trained model and export the calculation graph of the trained model. The operating environment of the target model can be shown in Table 2 below. In this embodiment, no quantization operation is performed during the operation of the network, and the data type of all calculation processes is fp32.
[0146] Table 1 Accelerator card product specifications
[0147]
[0148] Table 2 Operating environment of the target model
[0149] Model / Version CPU Intel(R)Xeon(R)Gold 6140CPU@2.30GHz operating system Ubuntu 18.04.6LTS PyTorch 1.10.2 Inference Acceleration Engine 0.8.4
[0150] In this embodiment, an accelerator card with the product specifications shown in Table 1 and an operating environment shown in Table 2 are used to respectively run the binary codes corresponding to the initial calculation graph and the final optimized calculation graph of the YOLOv5s network. Part of the performance of the initial calculation graph and the final optimized calculation graph when processing the prediction frame is shown in Table 3 below.
[0151] Table 3 Partial performance of the initial calculation graph and the final optimized calculation graph
[0152]
[0153] In Table 3, the acceleration ratio of the accelerator card that executes the final optimized calculation graph is the ratio of the processing time of the final optimized calculation graph to the processing time of the initial calculation graph. From Table 3, it can be seen that the processing time of the final optimized calculation graph is shorter than that of the initial calculation graph. Therefore, it can be proved that calling fusion during the inference process of the target model can reduce the inference time of the target model.
[0154] In the method for optimizing model reasoning performance provided by this embodiment, the target model is a YOLOv5s network model or a model that improves the YOLOv5s network, and the target model is a target detection model; after the sub-computation graph is completed for the multi-scale prediction box reasoning, a sub-computation graph corresponding to the final separate processing process of the prediction box is generated; since the network structure of the three separate processing processes of the prediction boxes of the three scales of the YOLOv5s network model is the same, and the three separate processings of the prediction boxes of the three scales are followed by string concatenation by the operator concat, the operator concat does not change the data shape but only changes the data scale, therefore, the operators that need to be called in the three separate processing processes and the operator concat are subjected to operator fusion processing to obtain a fusion operator, and the implementation function of the fusion operator can replace the implementation function of the three separate processing processes and the operator concat, and the target model also needs to call the fusion operator once during the reasoning process, thereby reducing the reasoning time of the target model.
[0155] Embodiment 9
[0156] The method for optimizing model reasoning performance provided in this embodiment is based on the eighth embodiment, and refines step 804, controlling the acceleration card to perform the reasoning process of the target model according to the binary code file and the dynamic library of at least one fusion operator. Then step 804 is refined into the following scheme.
[0157] When the control accelerator card performs inference on the target model according to the binary code file and the dynamic library of at least one fusion operator, each processor core of the control accelerator card processes only the tensor data corresponding to one image at a time in the final separate processing of the generation frame, and splices the processed data in sequence at the output position on the DDR.
[0158] The final separate processing of the generated box is as follows: after the multi-scale prediction box inference is completed, the final separate processing of the generated box is completed. The generated box can be the detection box output by the target model inference.
[0159] In this embodiment, the input data in the YOLOv5s network model is an image, and the output data is the tensor data corresponding to the image. The tensor data corresponding to the image includes the number of prediction boxes generated on the image, the scale of the prediction box, and the attributes of each prediction box. The attributes of the prediction box include coordinate points, width and height values, and confidence. It can be understood that each image is independent of each other, so the tensor data corresponding to each image is also independent of each other.
[0160] During the inference process, the YOLOv5s network model will generate tensor data of the input image at three scales. One input image corresponds to tensor data of three scales. The scale refers to the scale of the prediction box. The width and height values of the prediction box generated in the tensor data of the three scales are different. Therefore, after the prediction box inference of the three scales is completed, the YOLOv5s network model will perform the final processing of the generated box on the prediction box of each scale. The final processing is to adjust the coordinates and length and width values of the center point of the prediction box to adjust the size of the corresponding size of the prediction box of each scale to the size suitable for the original image, including: slicing operation, vector binary product operation and splicing operation.
[0161] The final processing of the generated frame includes: extracting the center coordinate point, length and width values, and confidence of the predicted frame through slicing operations; performing different calculations on the center coordinate point and length and width values to adjust the size of the generated frame to the size suitable for the original image; and then splicing the calculated center coordinate point and length and width values with the confidence that is not calculated. It can be understood that in the final processing of the generated frame, each predicted frame does not need to rely on the data of other predicted frames, so the processing process is also independent for each predicted frame.
[0162] Since the accelerator card generally has multiple processor cores, and multiple processor cores cannot communicate directly, as long as the on-chip storage space of the processor core is greater than or equal to the number of elements corresponding to the prediction box attributes in the tensor data corresponding to the image, the final separate processing of the prediction box generated for an image can be completed through a loop. Here, for tensor data of different scales, the corresponding processing method can be determined according to the shape or scale of the input data. Control each processor core to process only the tensor data corresponding to one image at a time, and after the processing is completed, put the output data at the output position of the DDR for splicing, which can avoid the time waste caused by communication between processor cores. In addition, it is possible to process all prediction boxes in batches with each processor core, and make the output data of each processor core realize the splicing of the number of prediction boxes at the output position of the DDR.
[0163] In this embodiment, the product specifications of the accelerator card can be the same as those in the eighth embodiment, and the YOLOv5s network model can be in ONNX format. The preset artificial intelligence framework can be PyTorch, and the operating environment of the target model can be the same as that in the eighth embodiment.
[0164] It can be seen from Table 3 of Example 8 that when the number of image frames is 1, the processing time of the final optimized calculation graph is not much different from the processing time of the initial calculation graph. This is because the tensor data corresponding to each frame of the image are independent of each other. In the process of executing the target model by the acceleration card, it is simply split according to the number of image frames, and each core processes one image at a time. This makes it possible to have only one processor core working when the number of image frames is 1 frame, and the general acceleration card is a multi-processor core architecture, which wastes the hardware resources of the acceleration card. Therefore, as long as the amount of computation is evenly distributed to each processor core, and no more processing logic is introduced, the hardware resource utilization of the processor core can be improved. Therefore, the method provided in this embodiment can be used to make each processor core only process the tensor data corresponding to one image at a time, and the computing tasks are evenly distributed to each processor core. And the processed data can be stored in the output position on the DDR to achieve the splicing of the output data. The initial calculation graph, and the final optimized calculation graph, when each processor core only processes the tensor data corresponding to one image at a time, the corresponding YOLOv5s network model's overall transmission frames per second fps and optimization ratio ratio are shown in Table 4 below.
[0165] Table 4 Overall fps and optimization ratio of the model
[0166]
[0167] In this embodiment, if the overall running time of the YOLOv5s network is t milliseconds and the number of input frames is bn, the calculation formula for the number of fps transmitted per second is: If the fps of the YOLOv5s network model corresponding to the initial calculation graph is fps_auto, and the fps of the YOLOv5s network model corresponding to the final optimized calculation graph when each processor core processes only the tensor data corresponding to one image at a time is fps_manual, then the calculation formula for the optimization ratio ratio is:
[0168] The method for optimizing model reasoning performance provided in this embodiment controls the accelerator card to perform the reasoning process of the target model according to the binary code file and the dynamic library of at least one fusion operator, and controls each processor core of the accelerator card to process only the tensor data corresponding to one image at a time during the final separate processing of the generation frame, and splices the processed data in sequence at the output position on the DDR; since each processor core processes only the tensor data corresponding to one image at a time during the final separate processing of the generation frame, all prediction frames can be processed in batches, thereby improving the hardware resource utilization of the accelerator card, placing the processed parts in sequence in the memory of the accelerator card, realizing automatic splicing, and reducing the time for reading and writing data operations between the processor core and the memory of the accelerator card.
[0169] Embodiment 10
[0170] Fig.10 This is a schematic diagram of the structure of the device for optimizing model reasoning performance provided in Example 10 of the present application, such as Fig.10 As shown, the model reasoning performance optimization device 100 provided in this embodiment includes: an acquisition module 101, an automatic optimization module 102, an automatic optimization module 102, a fusion processing module 104 and a fusion optimization processing module 105.
[0171] An acquisition module 101 is used to acquire an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model;
[0172] The automatic optimization module 102 is used to optimize the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph;
[0173] A determination module 103 is used to analyze the automatically optimized computation graph and determine at least one sub-computation graph that meets the operator fusion condition; the operator in the sub-computation graph is the target operator;
[0174] A fusion processing module 104 is used to perform operator fusion processing on a target operator in the same sub-computation graph for at least one sub-computation graph, so as to obtain at least one fused operator;
[0175] The fusion optimization processing module 105 optimizes the reasoning performance of the target model based on at least one fusion operator.
[0176] As an optional implementation, the automatic optimization module 102 is specifically used to optimize the initial model file using the optimization strategy pre-configured in the preset inference acceleration engine; view the optimization log after the model file is optimized, and obtain the automatically optimized calculation graph from the optimization log.
[0177] As an optional implementation, the determination module 103 is specifically used to parse the automatically optimized computation graph to obtain the network structure, input data characteristics and output data characteristics between the operators before fusion; determine at least one group of operators that meet the operator fusion conditions based on the network structure, input data characteristics and output data characteristics between the operators before fusion; and determine at least one sub-computation graph based on at least one group of operators that meet the operator fusion conditions.
[0178] As an optional implementation, the fusion processing module 104 is specifically used to obtain, for at least one sub-computation graph, the network structure between each target operator in the same sub-computation graph, the implementation function of each target operator and the input and output data shape characteristics of the sub-computation graph; and generate at least one fusion operator based on the network structure, each implementation function and the input and output data shape characteristics of the sub-computation graph in at least one sub-computation graph.
[0179] As an optional implementation, the fusion processing module 104 is specifically used to generate kernel functions of the fusion operator according to the network structure and each implementation function described in at least one sub-computation graph; generate a shape inference function of the fusion operator according to the shape characteristics of the input and output data of the sub-computation graph described in at least one sub-computation graph; and determine the scope and operator identifier of at least one fusion operator.
[0180] As an optional implementation, the device 100 for optimizing model reasoning performance also includes a registration module, which is used to register the scope and operator identifier of at least one fusion operator in a preset artificial intelligence framework; register the scope, operator identifier, the kernel function and the shape reasoning function of at least one fusion operator in a preset reasoning acceleration engine, and compile and generate a dynamic library of at least one fusion operator.
[0181] As an optional implementation, the fusion optimization processing module 105 is specifically used to obtain a model file after operator fusion corresponding to the target model from a preset artificial intelligence framework, wherein the model file after operator fusion includes a calculation graph with at least one fusion operator; optimize the model file after operator fusion using a preset inference acceleration engine to obtain a final optimized calculation graph; generate a binary code file corresponding to the final optimized calculation graph, and input the binary code file and the dynamic library of at least one fusion operator into an acceleration card; control the acceleration card to perform the inference process of the target model according to the binary code file and the dynamic library of at least one fusion operator.
[0182] As an optional implementation, the target model is a YOLOv5s network model or a model that improves the YOLOv5s network, and the target model is a target detection model; the sub-computation graph is a sub-computation graph corresponding to the final processing process of the generated box after the multi-scale prediction box reasoning is completed.
[0183] As an optional implementation, the fusion optimization processing module 105 is also specifically used to control the acceleration card to perform inference on the target model according to the binary code file and the dynamic library of at least one fusion operator, and to control the acceleration card to process only the tensor data corresponding to one image at a time during the final separate processing of the generation frame, and to splice the processed data in sequence at the output position on the DDR.
[0184] Embodiment 11
[0185] Fig.11 A schematic diagram of the structure of an electronic device provided in Example 11 of the present application, such as Fig.11 As shown, the electronic device 110 provided in this embodiment includes: a processor 111, and a memory 112 communicatively connected to the processor.
[0186] The memory 112 stores computer executable instructions.
[0187] The processor 111 executes the computer-executable instructions stored in the memory 112 to implement a method for optimizing model reasoning performance as provided in any one of the above embodiments.
[0188] The processor 111 and the memory 112 are connected via a bus.
[0189] An embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement a method for optimizing model reasoning performance as provided in any of the above embodiments.
[0190] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements a method for optimizing model reasoning performance as provided in any of the above embodiments.
[0191] The foregoing may be better understood in view of the following clauses:
[0192] Clause 1. A method for optimizing model inference performance, comprising:
[0193] Obtain an initial model file of the target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model;
[0194] The initial model file is optimized using a preset inference acceleration engine to obtain an automatically optimized calculation graph;
[0195] Analyzing the automatically optimized computation graph to determine at least one sub-computation graph that satisfies the operator fusion condition; the operator in the sub-computation graph is the target operator;
[0196] For at least one sub-computation graph, perform operator fusion processing on a target operator in the same sub-computation graph to obtain at least one fused operator;
[0197] The inference performance of the target model is optimized based on at least one fusion operator.
[0198] Clause 2. The method according to clause 1, wherein the initial model file is optimized using a preset inference acceleration engine to obtain an automatically optimized computation graph, comprising:
[0199] The initial model file is optimized using an optimization strategy pre-configured in a preset reasoning acceleration engine;
[0200] Check the optimization log after optimizing the model file, and obtain the automatically optimized calculation graph from the optimization log.
[0201] Clause 3. The method according to clause 1, wherein the automatically optimized computation graph is parsed to determine at least one sub-computation graph that satisfies the operator fusion condition, comprising:
[0202] Analyze the automatically optimized computational graph to obtain the network structure, input data features, and output data features between operators before fusion;
[0203] Determine at least one group of operators that meet operator fusion conditions according to the network structure, input data characteristics, and output data characteristics between operators before fusion;
[0204] At least one sub-computation graph is determined based on at least one group of operators that satisfy an operator fusion condition.
[0205] Clause 4. The method according to clause 1, wherein for at least one sub-computation graph, performing operator fusion processing on a target operator in the same sub-computation graph to obtain at least one fused operator comprises:
[0206] For at least one sub-computation graph, obtain the network structure between target operators in the same sub-computation graph, the implementation function of each target operator, and the input and output data shape characteristics of the sub-computation graph;
[0207] At least one fusion operator is generated according to the network structure in at least one sub-computation graph, each of the implementation functions and the input and output data shape characteristics of the sub-computation graph.
[0208] Clause 5. The method according to clause 4, wherein generating at least one fusion operator according to the network structure, each of the implementation functions and the input and output data shape characteristics of at least one sub-computation graph comprises:
[0209] Generate kernel functions of fusion operators respectively according to the network structure and each implementation function in at least one sub-computation graph;
[0210] Generate a shape reasoning function of a fusion operator according to shape features of input and output data of the sub-computation graph in at least one sub-computation graph;
[0211] Determine the scope and operator identifier of at least one fusion operator.
[0212] Clause 6. The method according to clause 5, before optimizing the reasoning performance of the target model based on at least one fusion operator, further comprising:
[0213] Registering the scope and operator identifier of at least one fusion operator in a preset artificial intelligence framework;
[0214] The scope, operator identifier, kernel function and shape reasoning function of at least one fusion operator are registered in a preset reasoning acceleration engine, and a dynamic library of at least one fusion operator is compiled and generated.
[0215] Clause 7. The method according to clause 6, wherein the optimizing the inference performance of the target model based on at least one fusion operator comprises:
[0216] Obtaining a model file after operator fusion corresponding to a target model from a preset artificial intelligence framework, wherein the model file after operator fusion includes a computational graph having at least one fusion operator;
[0217] Using a preset inference acceleration engine to optimize the model file after the operator fusion to obtain a final optimized calculation graph;
[0218] Generate a binary code file corresponding to the final optimized computation graph, and input the binary code file and a dynamic library of at least one fusion operator into an accelerator card;
[0219] The accelerator card is controlled to perform an inference process of a target model according to the binary code file and a dynamic library of at least one fusion operator.
[0220] Clause 8. A method according to any one of clauses 1 to 7, wherein the target model is a YOLOv5s network model or a model that improves the YOLOv5s network, and the target model is a target detection model;
[0221] The sub-computation graph is a sub-computation graph corresponding to the final processing process of the generated box after the multi-scale prediction box reasoning is completed.
[0222] Clause 9. The method according to clause 8, wherein the step of controlling the accelerator card to perform the inference process of the target model according to the binary code file and the dynamic library of at least one fusion operator comprises:
[0223] In the process of controlling the accelerator card to perform the inference of the target model according to the binary code file and the dynamic library of at least one fusion operator, each processor core of the accelerator card is controlled to process only the tensor data corresponding to one image at a time during the final separate processing of the generation frame, and the processed data is sequentially spliced at the output position on the DDR.
[0224] Clause 10. A device for optimizing model reasoning performance, comprising:
[0225] An acquisition module, used to acquire an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model;
[0226] An automatic optimization module, used to optimize the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph;
[0227] A determination module, used to analyze the automatically optimized computation graph and determine at least one sub-computation graph that meets the operator fusion condition; the operator in the sub-computation graph is the target operator;
[0228] A fusion processing module, used for performing operator fusion processing on a target operator in the same sub-computation graph for at least one sub-computation graph, so as to obtain at least one fused operator;
[0229] The fusion optimization processing module optimizes the reasoning performance of the target model based on at least one fusion operator.
[0230] Clause 11. An electronic device comprising: a processor, and a memory communicatively connected to the processor;
[0231] The memory stores computer-executable instructions;
[0232] The processor executes the computer-executable instructions stored in the memory to implement the method as described in any of clauses 1-9.
[0233] Clause 12. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to implement the method as described in any one of Clauses 1-9 when executed by a processor.
[0234] Clause 13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of clauses 1 to 9.
[0235] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0236] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0237] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0238] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0239] If the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the artificial intelligence processor can be any appropriate hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM (Resistive Random Access Memory), dynamic random access memory DRAM (Dynamic Random Access Memory), static random access memory SRAM (Static Random-Access Memory), enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), high-bandwidth memory HBM (High-Bandwidth Memory), hybrid memory cube HMC (Hybrid Memory Cube), etc.
[0240] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0241] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
Claims
1. A method for optimizing model reasoning performance, characterized in that: include: Obtain an initial model file of the target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model; The initial model file is optimized using a preset inference acceleration engine to obtain an automatically optimized calculation graph; Analyze the automatically optimized computation graph to determine at least one sub-computation graph that meets the operator fusion condition, where the operator in the sub-computation graph is the target operator; For at least one sub-computation graph, perform operator fusion processing on a target operator in the same sub-computation graph to obtain at least one fused operator; Optimizing the inference performance of the target model based on at least one fusion operator; The optimizing process of the initial model file by using a preset inference acceleration engine to obtain an automatically optimized calculation graph includes: The initial model file is optimized using a pre-configured optimization strategy in a preset reasoning acceleration engine; the pre-configured optimization strategy includes any one or more of shape reasoning optimization, constant propagation optimization, and software pipelining optimization; Viewing an optimization log after optimizing the model file, and obtaining an automatically optimized calculation graph from the optimization log; The step of parsing the automatically optimized computation graph to determine at least one sub-computation graph that satisfies the operator fusion condition includes: The automatically optimized calculation graph is parsed to obtain the network structure, input data characteristics and output data characteristics between the operators before fusion; the network structure between the operators before fusion includes the connection order of the operators before fusion in the automatically optimized calculation graph; the input data characteristics between the operators before fusion include input data shape characteristics and input data scale; the output data characteristics between the operators before fusion include output data shape characteristics and output data scale; Determine at least one group of operators that meet operator fusion conditions according to the network structure, input data characteristics, and output data characteristics between the operators before fusion; Determine at least one sub-computation graph based on at least one group of operators that meet the operator fusion condition; The optimizing the reasoning performance of the target model based on at least one fusion operator includes: Generate a binary code file corresponding to the final optimized calculation graph, and input the binary code file and a dynamic library of at least one fusion operator into the accelerator card.
2. The method according to claim 1, characterized in that: The step of performing operator fusion processing on a target operator in the same sub-computation graph to obtain at least one fused operator includes: For at least one sub-computation graph, obtain the network structure between target operators in the same sub-computation graph, the implementation function of each target operator, and the input and output data shape characteristics of the sub-computation graph; At least one fusion operator is generated according to the network structure in at least one sub-computation graph, each of the implementation functions and the input and output data shape characteristics of the sub-computation graph.
3. The method according to claim 2, characterized in that Generating at least one fusion operator according to the network structure, each implementation function and input and output data shape characteristics of at least one sub-computation graph includes: Generate kernel functions of fusion operators respectively according to the network structure and each implementation function in at least one sub-computation graph; Generate a shape reasoning function of a fusion operator according to shape features of input and output data of the sub-computation graph in at least one sub-computation graph; Determine the scope and operator identifier of at least one fusion operator.
4. The method according to claim 3, characterized in that Before optimizing the reasoning performance of the target model based on at least one fusion operator, the method further includes: Registering the scope and operator identifier of at least one fusion operator in a preset artificial intelligence framework; The scope, operator identifier, kernel function and shape reasoning function of at least one fusion operator are registered in a preset reasoning acceleration engine, and a dynamic library of at least one fusion operator is compiled and generated.
5. The method according to claim 4, characterized in that The optimizing the reasoning performance of the target model based on at least one fusion operator further includes: Obtaining a model file after operator fusion corresponding to a target model from a preset artificial intelligence framework, wherein the model file after operator fusion includes a computational graph having at least one fusion operator; Using a preset inference acceleration engine to optimize the model file after the operator fusion to obtain a final optimized calculation graph; The accelerator card is controlled to perform an inference process of a target model according to the binary code file and a dynamic library of at least one fusion operator.
6. The method according to any one of claims 1 to 5, characterized in that: The target model is a YOLOv5s network model or a model that improves the YOLOv5s network, and the target model is a target detection model; The sub-computation graph is a sub-computation graph corresponding to the final processing process of the generated box after the multi-scale prediction box reasoning is completed.
7. The method according to claim 5, characterized in that The process of controlling the accelerator card to perform the inference process of the target model according to the binary code file and the dynamic library of at least one fusion operator includes: In the process of controlling the accelerator card to perform the inference of the target model according to the binary code file and the dynamic library of at least one fusion operator, each processor core of the accelerator card is controlled to process only the tensor data corresponding to one image at a time during the final separate processing of the generation frame, and the processed data is sequentially spliced at the output position on the DDR.
8. A device for optimizing model reasoning performance, characterized in that: include: An acquisition module, used to acquire an initial model file of a target model from a preset artificial intelligence framework, wherein the initial model file has an initial calculation graph corresponding to the target model; An automatic optimization module, used to optimize the initial model file using a preset reasoning acceleration engine to obtain an automatically optimized calculation graph; A determination module, used to analyze the automatically optimized computation graph and determine at least one sub-computation graph that meets the operator fusion condition; the operator in the sub-computation graph is the target operator; A fusion processing module, used for performing operator fusion processing on a target operator in the same sub-computation graph for at least one sub-computation graph, so as to obtain at least one fused operator; A fusion optimization processing module, which optimizes the reasoning performance of the target model based on at least one fusion operator; The automatic optimization module is specifically used to optimize the initial model file using the optimization strategy pre-configured in the preset reasoning acceleration engine; the pre-configured optimization strategy includes any one or more of shape reasoning optimization, constant propagation optimization, and software pipeline optimization; check the optimization log after the optimization of the model file, and obtain the automatically optimized calculation graph from the optimization log; The determination module is specifically used to analyze the automatically optimized calculation graph to obtain the network structure, input data characteristics and output data characteristics between the operators before fusion; the network structure between the operators before fusion includes the connection order of the operators before melting in the automatically optimized calculation graph; the input data characteristics between the operators before fusion include input data shape characteristics and input data scale; the output data characteristics between the operators before fusion include output data shape characteristics and output data scale; determine at least one group of operators that meet the operator fusion condition according to the network structure, input data characteristics and output data characteristics between the operators before fusion; determine at least one sub-calculation graph based on at least one group of operators that meet the operator fusion condition; The fusion optimization processing module is specifically used to generate a binary code file corresponding to the final optimized calculation graph, and input the binary code file and a dynamic library of at least one fusion operator into the acceleration card.
9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
