Model optimization method and device, computer equipment and storage medium

Abstract

The invention relates to a model optimization method and device, computer equipment and a storage medium. The method comprises the steps that a to-be-optimized model is converted into an intermediate model, at least two to-be-optimized operators in the intermediate model are recognized through a first-class optimizer according to a pre-configured reference operator structure chart, and the reference operator structure chart is configured according to time delay information fused by operators in historical data; plug-in operators are inserted into connection links of the adjacent to-be-optimized operators through the first-class optimizers to obtain a first optimization result, the plug-in operators are used for blocking fusion between the to-be-optimized operators, and an optimized model is determined according to the first optimization result. By adopting the method, the problem of time delay increase caused by operator fusion of an inference engine in a model deployment stage can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a model optimization method and device, computer equipment and storage medium. BACKGROUND

[0002] Model optimization refers to a process of improving model performance and reducing resource consumption under the premise of ensuring model accuracy. The core goal is to make the model more suitable for actual application scenarios, and the specific optimization directions include reducing inference latency, reducing memory occupancy, and improving throughput, etc.

[0003] Model deployment is the whole process of converting the optimized model after training into a form that can run efficiently on the target hardware, and integrating it into the actual application system.

[0004] In the model deployment process, the inference engine needs to parse the model first, and the parsing will automatically perform operator fusion to improve efficiency. However, when encountering specific operator connection patterns, hardware adaptation problems or engine bugs, the calculation after fusion will produce additional overhead, resulting in a sharp rise in latency. SUMMARY

[0005] Therefore, it is necessary to provide a model optimization method, device, computer equipment and storage medium capable of solving the latency problem caused by operator fusion to solve the above technical problems.

[0006] In a first aspect, the present application provides a model optimization method, comprising: Converting a to-be-optimized model into an intermediate model; Identifying, by a first type of optimizer, to-be-optimized operators in the intermediate model according to a pre-configured benchmark operator structure diagram, the to-be-optimized operators including at least two, and the benchmark operator structure diagram being configured according to latency information of operator fusion in historical data; Inserting, by the first type of optimizer, a plug-in operator in a connection link between adjacent to-be-optimized operators to obtain a first optimization result, the plug-in operator being used to block fusion between the to-be-optimized operators; According to the first optimization result, determining an optimized model.

[0007] In one of the embodiments, the model optimization method further comprises: Obtaining, by a second type of optimizer, operator specifications of each operator in the intermediate model, adjusting the operator specifications according to a parsing rule when the operator specifications do not match the parsing rule, obtaining a second optimization result, the inference engine being an inference engine to which the optimized model is deployed, and the operator specifications including attribute information, input quantity and operator type of the operator; When identifying, by the first type of optimizer, the to-be-optimized operators in the intermediate model, identifying the to-be-optimized operators from the second optimization result.

[0008] In one of the embodiments, the process of adjusting the operator type according to the resolution rule comprises: creating a new operator according to the operator type specified in the resolution rule and the function of the operator before adjustment, and replacing the operator before adjustment with the new operator to adjust the operator type.

[0009] In one of the embodiments, the benchmark operator structure diagram comprises a plurality of benchmark operator structure diagrams, each of which corresponds to a group of to-be-optimized operators, and the first optimization result is obtained by inserting a plug-in operator into the connection link between adjacent to-be-optimized operators through the first type of optimizer, comprising: obtaining the first optimization result by inserting a plug-in operator into the connection link between adjacent operators of each group of to-be-optimized operators through the first type of optimizer.

[0010] In one of the embodiments, the model optimization further comprises: obtaining the time delay difference value of each operator of the intermediate model before and after fusion with other operators according to the time delay information of operator fusion in the historical data; configuring the benchmark operator structure diagram according to the operators whose time delay difference values are greater than the preset threshold and the corresponding operator connection relationship.

[0011] In one of the embodiments, before the first optimization result is obtained by inserting a plug-in operator into the connection link between adjacent to-be-optimized operators through the first type of optimizer, the method further comprises: detecting each operator of the intermediate model through the third type of optimizer, and performing operator merging on two or more operators on the same connection link which are positionally continuous, functionally identical, and of the same type; and / or performing operator fusion on two or more operators on the same connection link which are positionally continuous through the third type of optimizer by means of mathematical transformation or parameter injection; and / or detecting whether the attribute information and input dimension of each operator are respectively consistent with the target attribute and target input dimension pre-configured through the third type of optimizer, and if not, adjusting the attribute information and input dimension of the to-be-optimized operator to the target attribute and target input dimension respectively to obtain a third optimization result; identifying the to-be-optimized operator from the third optimization result when identifying the to-be-optimized operator in the intermediate model through the first type of optimizer.

[0012] In one of the embodiments, the model optimization method further comprises: traversing each operator on the same link in the calculation graph of the intermediate model through the third type of optimizer; when there are identical operators on the same link, judging whether the output result of the calculation graph remains unchanged before and after positionally interchanging each identical operator with the adjacent operator on the input side through the third type of optimizer, and if so, performing the position interchanging.

[0013] In a second aspect, the present application provides a model optimization apparatus, comprising: a conversion module configured to convert a model to be optimized into an intermediate model; an identification module configured to identify, by a first type of optimizer, to-be-optimized operators in the intermediate model according to a pre-configured benchmark operator structure diagram, the to-be-optimized operators including at least two, and the benchmark operator structure diagram being configured according to time delay information of operator fusion in historical data; an insertion module configured to insert a plug-in operator in a connection link between adjacent to-be-optimized operators to obtain a first optimization result, the plug-in operator being configured to block fusion between the to-be-optimized operators; a determination module configured to determine an optimized model according to the first optimization result.

[0014] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the model optimization method provided in any of the embodiments of the first aspect when executing the computer program.

[0015] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the steps of the model optimization method provided in any of the embodiments of the first aspect.

[0016] The above model optimization method, apparatus, computer device, and storage medium, by configuring a benchmark operator structure diagram to identify to-be-optimized operators in an intermediate model, wherein the benchmark operator structure diagram is configured based on time delay information before and after operator fusion in historical data, and is a structure diagram template configured according to time delay problems caused by historical operator fusion. The to-be-optimized operators in the intermediate model that may trigger time delay problems are identified by the benchmark operator structure diagram, and a plug-in operator is further inserted between the to-be-optimized operators to physically block fusion between the to-be-optimized operators and their adjacent operators, thereby avoiding the problem of excessively high time delay caused by fusion of some operators in the model deployment stage. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 Flowchart of the model optimization method in some embodiments; Figure 2 Flowchart of the model optimization method in some other embodiments; Figure 3 Flowchart of the model optimization method in some other embodiments; Figure 4 Structure block diagram of the model optimization apparatus in some embodiments; Figure 5An internal structure diagram of a computer device in some embodiments. DETAILED DESCRIPTION

[0018] For the purposes of the present application, the technical solutions and advantages thereof are more clearly apparent, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0019] In a first aspect, the present application provides a model optimization method, as shown in the formula: Figure 1 The method is applied to a server as an example, and includes the following steps: Step S11, converting a to-be-optimized model into an intermediate model.

[0020] The to-be-optimized model is a model trained by using a deep learning training framework (such as Pytorch).

[0021] The intermediate model is a standardized model representation format independent of specific training frameworks and inference engines. Its core role is to convert the model output by the training framework into a general format that can be parsed and adapted by the inference engine, solving the format compatibility problem between different frameworks and engines, and providing a unified operation carrier for model optimization. The intermediate model can be an ONNX (Open Neural Network Exchange) model.

[0022] The present application can traverse the computation graph of the original model, map each native operator to a standard operator of the intermediate model, and extract and save the model weights and other parameters to the intermediate model file, finally realizing the conversion of the to-be-optimized model into the intermediate model.

[0023] The reason why the present application optimizes the intermediate model instead of the original model is that: The computation graph of the original model is not necessarily a completely static graph, and model optimization needs to determine the type, connection order, and input-output relationship of all operators in the model, therefore, the computation graph of the original model is not suitable for direct optimization; while the computation graph of the intermediate model is completely static and suitable for the structural analysis required for optimization.

[0024] Step S12, identifying to-be-optimized operators in the intermediate model according to a pre-configured benchmark operator structure diagram by a first type of optimizer, the to-be-optimized operators including at least two, and the benchmark operator structure diagram being configured according to the latency information of operator fusion in historical data.

[0025] The first type of optimizer is a type of optimizer for repairing the latency bug introduced by the inference engine when performing operator fusion.

[0026] An operator is a mapping from a function space to a function space. The benchmark operator structure diagram is a connection diagram between two or more operators configured according to the history record of the time delay problem introduced when the inference engine operator fusion, and is a local operator computation diagram. The benchmark operator structure diagram configures the to-be-optimized operator that may cause the time delay problem and the adjacent operator that may cause the time delay problem when fused with the to-be-optimized operator.

[0027] The to-be-optimized operator refers to an operator that matches the benchmark operator structure diagram and is prone to cause a time delay problem after the inference engine performs operator fusion.

[0028] The benchmark operator structure diagram in the present application is a kind of operator structure diagram mode configured in advance according to the time delay information of operator fusion in the historical data, and is used as the basis for screening the to-be-optimized operator. Specifically, a group of operators that may cause a high time delay before and after fusion is configured as a benchmark operator structure diagram, and the to-be-optimized operator is screened based on the benchmark operator structure diagram. Each time the to-be-optimized operator screened at least includes two, and the plug-in operator is inserted between the connection links of each adjacent to-be-optimized operator to block the fusion between the to-be-optimized operators.

[0029] For example, when it is concluded from the historical data that the fusion of operator A and operator B may cause a time delay problem, the connection relationship position of operator A connecting operator B is taken as the benchmark operator structure diagram.

[0030] It should be noted that the benchmark operator structure diagram can include multiple, each benchmark operator structure diagram can adapt to the local operator structure diagram at different positions of the same model, or can adapt to the operator structure diagram of different models. In the present application, the operator connection relationship of each part in the computation graph of the intermediate model is matched with the benchmark operator structure diagram respectively to identify the to-be-optimized operator. When a new model is replaced, the computation graph of the new model is also matched with the benchmark operator structure diagram to obtain the to-be-optimized operator in the new model, so as to adapt to multiple different models.

[0031] For example, the benchmark operator structure diagram is configured as operator A connecting operator B in advance, if operator A connecting operator B appears at multiple different positions in the computation graph of model M1, then the operator A and the operator B at these positions are to-be-optimized operators. For another example, when operator A connecting operator B appears in the computation graphs of model M2 and model M3, then the operator A and the operator B in model M2 are to-be-optimized operators, and the operator A and the operator B in model M3 are also to-be-optimized operators.

[0032] Further, the first type of optimizer can obtain the entire computation graph of the intermediate model, extract the operator connection relationship of each operator from the computation graph, match the operator connection relationship of each operator with the benchmark operator structure diagram, and obtain the to-be-optimized operator when the matching is successful.

[0033] Specifically, when the inference engine fuses the operator A and the operator B, the time delay anomaly occurs multiple times. Based on this, the operator A, the operator B, and the serial connection relationship of the two are configured into the benchmark operator structure diagram, and are pre-configured into the first type of optimizer.

[0034] Further, the first type of optimizer obtains a complete calculation graph of the intermediate model, uses the pre-configured benchmark operator structure diagram as a template, and searches in the calculation graph. When a local structure that completely matches the template is found, the operator A and the operator B are determined as the to-be-optimized operators.

[0035] After the to-be-optimized operators are identified, the adjacent operators of the to-be-optimized operators that have direct input-output connections with the to-be-optimized operators are extracted from the calculation graph according to the pre-configured benchmark operator structure diagram.

[0036] In step S13, the first optimization result is obtained by inserting the plug-in operator into the connection link of the adjacent to-be-optimized operators through the first type of optimizer, and the plug-in operator is used to block the fusion between the to-be-optimized operators.

[0037] The connection link refers to the path for transmitting data, instructions, or states between two or more operators.

[0038] The plug-in operator is an operator that is inserted into the connection link to break the original data transmission logic, thereby preventing the fusion of the upstream and downstream operators. Specifically, the plug-in operator needs to meet the following conditions: the plug-in operator cannot be fused with the to-be-optimized operators and the adjacent operators of the to-be-optimized operators, and the plug-in operator does not change the data content transmitted between the to-be-optimized operators and the adjacent operators.

[0039] The fusion between operators refers to the process of combining multiple operators with closely related functions and data streams into a single operator by the compiler or framework. The purpose is to reduce the data interaction overhead between operators and improve execution efficiency.

[0040] In this application, since the inference engine may have a time delay problem when deploying the intermediate model, resulting in excessively high time delay, the operators that often have a time delay problem after fusion and their operator connection relationship in the historical data are configured into a template, i.e., the above-mentioned benchmark operator structure diagram. Based on the benchmark operator structure diagram, the to-be-optimized operators in the intermediate model are identified, and operator insertion is performed on the to-be-optimized operators to block the fusion between the to-be-optimized operators, so as to avoid subsequent time delay problems.

[0041] For example, when deploying the model, TensorRT fuses some operators to reduce the overall delay of the model. However, in some cases, the fused operators have a time delay problem, which causes the delay to skyrocket. In this case, the operator fusion needs to be destroyed by inserting plug-in operators before and after the to-be-optimized operators.

[0042] In this application, because a plug-in operator is inserted, the plug-in operator cannot be fused with the operator to be optimized. Therefore, the operator to be optimized and its corresponding adjacent operators cannot meet the operator fusion conditions, and thus operator fusion cannot be performed, thereby blocking the fusion between the operator to be optimized and its adjacent operators.

[0043] For example, when the baseline operator structure diagram is configured with operator E connecting operator F connecting operator D, when the first type of optimizer recognizes the same connection relationship in the intermediate model, that is, operator E connecting operator F connecting operator D, the first type of optimizer will insert plug-in operators between each pair of operators E, operator F and operator D to block the fusion process between them.

[0044] Step S14: Determine the optimized model based on the first optimization result.

[0045] This application can determine the optimized model based on the optimized operator computation graph in the first optimization result. Specifically, it can parse the type, parameters, and attributes of the operators in the optimized computation graph, and determine the specific function of each operator; then it can sort out the data flow between nodes to determine the execution order and dependencies of the operators; finally, it can encapsulate the parsed operator combination logic and parameter configuration according to the specification of the target framework to generate a model file containing a complete structure and running logic.

[0046] In one embodiment, the model optimization method may further include: obtaining the operator specifications of each operator in the intermediate model through a second type of optimizer; when the operator specifications do not match the parsing rules of the inference engine, adjusting the operator specifications according to the parsing rules to obtain a second optimization result; the inference engine is the inference engine deployed on the optimized model; the operator specifications include the operator's attribute information, the number of inputs, and the operator type; when the first type of optimizer identifies the operator to be optimized in the intermediate model, the operator to be optimized is identified from the second optimization result.

[0047] The second type of optimizer is designed to ensure that the intermediate model can be correctly parsed by the inference engine. The inference engine is the inference engine deployed on the optimized model, and can be TensorRT, a deep learning model inference engine developed by NVIDIA.

[0048] The inference engine has special requirements for the attribute information of some operators of the intermediate model, such as the requirement of TensorRT 8.4 that the Reshape operator must have the allowzero attribute, otherwise the parsing will fail, in which case a default value needs to be filled for these attributes. In addition, some operators are not supported under a specific version of TensorRT, in which case the computation graph of the intermediate model needs to be modified, and one or more equivalent operators are used to replace the unsupported operators. Based on this, the second type of optimizer is designed to solve the problem of better parsing of the intermediate model by the inference engine.

[0049] The operator specification is a parameter set describing the basic information of the operator, mainly including the attribute information of the operator, the number of inputs, and the type of the operator.

[0050] The inference engine is a software framework for deploying a trained deep learning model to actual hardware and efficiently performing inference computation.

[0051] The attribute information of the operator is a key parameter that defines the behavior of the operator in addition to the input and output, such as the stride of the convolution operator, the padding method, and the threshold of the activation function. The number of inputs is the number of inputs received by the operator.

[0052] The type of the operator is the functional label of the operator, which clearly defines the specific operation type performed by the operator, such as "convolution" and "matrix multiplication". The inference engine needs to call the corresponding execution logic according to the type.

[0053] Specifically, the second type of optimizer scans the intermediate model and extracts the complete specification of each operator in the model, including the operator type, the number of inputs, and various attribute information. The extracted operator specification is compared with the parsing rules of the inference engine (such as TensorRT) one by one to check for any incompatible items, such as the inference engine not supporting a certain type of operator or the number of operator inputs not meeting the format required by the engine. For the mismatched problems found, the operator specification is modified according to the parsing rules of the inference engine. After completing the adaptation and adjustment of all operator specifications, the second optimization result is obtained, in which the operators can be normally parsed by the inference engine.

[0054] Exemplarily, the number of inputs of the operator is compared with the number specified in the parsing rules. When it is detected that an input is missing, the missing input is supplemented according to the default value in the parsing rules, so that the number of inputs of the operator is adapted to the parsing rules, and thus the subsequent inference engine can correctly parse the corresponding operator.

[0055] The beneficial effects of this embodiment are that by adjusting the operator specification, the parsing obstacles between the intermediate model and the target inference engine are eliminated, ensuring that the model can be correctly recognized and parsed by the inference engine, and avoiding errors caused by specification mismatch during deployment.

[0056] In one embodiment, the process of adjusting the operator type according to the parsing rule includes: creating a new operator according to the operator type specified in the parsing rule and the function of the operator before adjustment, replacing the operator before adjustment with the new operator to adjust the operator type.

[0057] In this embodiment, the application can obtain the operator types supported in the parsing rule of the inference engine, and based on the function of the operator before adjustment, a new operator with consistent function is created in the operator types supported in the parsing rule to reproduce the original function. In the computational graph of the intermediate model, the operator before adjustment is removed, the created new operator is inserted into the position of the original operator, and the connection link, input and output data flow of the new operator and the upstream and downstream operators remain completely unchanged, ensuring the overall logical coherence of the computational graph.

[0058] The beneficial effects of this embodiment are that the problem of the inference engine not supporting the operator type of the original model is solved, and the model is adapted to the operator library of the target inference engine through functionally equivalent replacement, ensuring that the model can be normally parsed.

[0059] In one embodiment, the benchmark operator structure diagram includes multiple benchmark operator structure diagrams, each benchmark operator structure diagram corresponds to a group of identified optimization operators, and a first optimization result is obtained by inserting a plug-in operator into the connection link between adjacent optimization operators through a first type of optimizer, including: a first optimization result is obtained by inserting a plug-in operator into the connection link between adjacent operators of each group of optimization operators through a first type of optimizer.

[0060] In this embodiment, the benchmark operator structure diagram is configured as a comparison template. The benchmark operator structure diagram can include multiple benchmark operator structure diagrams, and each benchmark operator structure diagram can correspond to a group of identified optimization operators. For example, please refer to Table 1 below, which is a configuration table of benchmark operator structure diagrams in one embodiment.

[0061] Table 1

[0062] As shown in Table 1 above, three benchmark operator structure diagrams are configured in the table, each benchmark operator structure diagram corresponds to a group of optimization operators, and the first type of optimizer compares the entire computational graph of the intermediate model with each benchmark operator structure diagram. When a structural connection relationship is matched in the computational graph, it is determined that the optimization operators are operator G4, operator G5 and operator G6, and a plug-in operator is inserted between the two adjacent operators of operator G4, operator G5 and operator G6 to block the operator fusion process between operator G4, operator G5 and operator G6. Figure 2

[0063] ​In the present application, when matching with the reference operator structure diagram, the operator identifier of each operator in the calculation graph is compared with the operator identifier of each operator in the configured reference operator structure diagram. When the operator identifier is successfully matched, the connection relationship between the operators is further compared. When the connection relationship between the operators is also successfully matched, it is determined that the matching is successful.

[0064] The beneficial effect of this embodiment is that by traversing each operator in the model, each operator and its connection relationship are compared with the reference operator structure diagram, and the to-be-optimized operator is accurately located.

[0065] In one of the embodiments, the model optimization method can further include: obtaining the time delay difference value of each operator of the intermediate model before and after fusion with other operators according to the time delay information of the operator fusion in the historical data, and configuring the reference operator structure diagram according to the operators whose time delay difference value is greater than a preset threshold and the corresponding operator connection relationship.

[0066] The time delay difference value refers to the difference between the time delay caused by the execution of one or more fused operators as a single composite operator and the time delay caused by the execution of these operators before fusion. If the difference is positive, it means that the time delay after fusion is greater.

[0067] Specifically, the present application can extract relevant data of the fusion between operators from the historical data of the intermediate model, further extract the time delay information of each operator before and after fusion with other operators, and calculate the time delay difference value before and after fusion. The combination of the fusion operators whose time delay difference value is greater than a preset threshold is screened out. The combination of these operators is configured into a reference operator structure diagram.

[0068] The beneficial effect of this embodiment is that the operator result diagram that is prone to cause time delay problems is configured based on the historical data, and the to-be-optimized operator in the calculation graph is identified based on the reference operator structure diagram, so that the time delay can be reduced after optimization.

[0069] In one of the embodiments, before the first optimization result is obtained by inserting the plug-in operator into the connection link of the adjacent to-be-optimized operators by the first type of optimizer, the method further comprises: detecting the operators of the intermediate model by the third type of optimizer, and performing operator merging on two or more operators that are positionally continuous, functionally identical, and of the same type on the same connection link, and / or performing operator fusion on two or more operators that are positionally continuous on the same connection link by mathematical transformation or parameter injection by the third type of optimizer, and / or detecting the attribute information of each operator and whether the input dimensions are respectively consistent with the pre-configured target attribute and target input dimension by the third type of optimizer, and if not, adjusting the attribute information of the to-be-optimized operator and the input dimensions to the target attribute and the target input dimension respectively to obtain a third optimization result, and identifying the to-be-optimized operator from the third optimization result when identifying the to-be-optimized operator in the intermediate model by the first type of optimizer.

[0070] The third type of optimizer is an optimizer designed to optimize the model latency, and the difference between the third type of optimizer and the first type of optimizer is that the first type of optimizer solves the latency problem caused by operator fusion when the inference engine performs subsequent operator fusion, and the third type of optimizer solves how to make the latency lower without operator fusion.

[0071] The functions of the third type of optimizer include: First, performing operator merging. Some continuous operators are merged into one under the condition of result equivalence, such as a plurality of continuous Transpose operators that can be fused into one Transpose operator.

[0072] Second, performing operator fusion. Some operators can fuse some nodes before and after the operators into the operator weights or parameters of the operators. For example, if a previous node of a Conv (convolution) operator is a Mul (multiplication) operator, and an input of the Mul operator is a constant input, we can update the kernel weight value in the Conv to fuse the calculation of the Mul operator into the Conv, so that one Mul operator can be saved to reduce the model latency. The condition for operator fusion is that two or more operators that are positionally continuous on the same connection link can be converted by mathematical transformation or parameter injection.

[0073] Third, adjusting the specifications of the operators to make them run more efficiently. Operators can have different running efficiencies when the input dimensions or attributes are different, such as a Reduce (reduction) operator that has a large difference in latency under different dimension quantities, and the latency is optimal under four dimensions, so a Reshape operator can be added before the Reduce operator to change the dimension to four before the Reduce operator to adjust the dimension of the operator.

[0074] Specifically, the adjustment of the input dimension can include adjustment of the number of operator dimensions to the most efficient dimension number. The adjustment of the attribute can include adjustment of the calculation dimension of the operator. For example, the application can obtain the calculation efficiency of a certain operator under a certain input dimension according to historical data, and adjust the calculation dimension of the operator to the dimension with the highest calculation efficiency, so that the operator calculates in the dimension.

[0075] For another example, the application can detect whether the input data of the current operator satisfies the specified conversion condition according to the protocol of the intermediate model. When the condition is satisfied, the operator type of the current operator is converted to the operator type specified in the protocol.

[0076] For example, the conversion condition can be: only when all the dimensions corresponding to the input shape whose dim is greater than 1 in the repeats input of the Tile operator are equal to 1, the Tile can be converted to the Expand operator. For example, Tile: input shape = [1,2,1,3], repeats = [10,1,10,1] == > Expand: input_shape = [1,2,1,3], shape = [10,2,10,3].

[0077] It means that only when all the dimensions whose values are greater than 1 in the repeats parameter of the Tile operator (i.e. the dimensions that need to be copied) are equal to 1 in the input shape, the Tile can be safely converted to the Expand. The above example defines a safe conversion rule to replace the relatively time-consuming Tile operator with the more efficient Expand operator. The key to the conversion is to ensure that the replication behavior of the Tile is mathematically equivalent to the broadcast behavior of the Expand.

[0078] The beneficial effects of this embodiment are that the number of operators is reduced by operator merging and fusion, the data transfer and calculation overhead between operators is reduced, and the running efficiency of the intermediate model is directly improved. The operator is in an efficient running state by adjusting the operator specification. By optimizing first by the third type of optimizer and then by the first type of optimizer, the latency caused by the optimization of the third type of optimizer is further solved by the first type of optimizer, so that better optimization effect is finally achieved.

[0079] In one of the embodiments, the model optimization method can further include: traversing each operator on the same link in the computational graph of the intermediate model by the third type of optimizer, and when there are same operators on the same link, determining by the third type of optimizer whether the output result of the computational graph remains unchanged before and after each same operator is respectively positionally interchanged with the adjacent operator on the input side, and if so, performing the positional interchanging.

[0080] The computational graph refers to a complete connection relationship graph constituted by the connection relationship of each operator of the intermediate model.

[0081] In the present application, multiple same operators are moved together as much as possible to facilitate merging. The specific moving manner is as follows: assuming that the same operator is a target operator, it is determined whether each target operator can be positionally interchanged with the adjacent node on the input side without affecting the final output result, and if not, the positional interchanging is performed to make each target operator as much forward as possible.

[0082] Or in another embodiment, it can also be determined whether each target operator can be positionally interchanged with the adjacent node on the output side without affecting the final output result, and if not, the positional interchanging is performed to make each target operator as much backward as possible.

[0083] In another design, each target operator can also move towards the output side, i.e., each target operator is positionally interchanged with the adjacent operator on the output side without affecting the final output result.

[0084] The beneficial effect of this embodiment is that when there are multiple same operators on the same link, the same operators can be moved to the continuous positions as much as possible through the positional interchanging to facilitate the operator fusion.

[0085] In one of the embodiments, the plug-in operator and the to-be-optimized operator do not satisfy the operator fusion condition.

[0086] The plug-in operator needs to achieve the change of the data content, and the to-be-optimized operator and the plug-in operator do not satisfy the operator fusion condition, so as to better block the operator fusion between the to-be-optimized operators without bringing new fusion.

[0087] Specifically, the operator fusion condition can include: two or more operators on the same connection link are converted through mathematical transformation or parameter injection, the hardware supports the fusion operation, the operator type and parameter meet the pre-defined fusion rule, the attribute matches, etc.

[0088] In one of the embodiments, the model optimization method further comprises: monitoring the latency information of the to-be-optimized model before and after the optimization by the first-type optimizer, and sending the latency information to the terminal for display; receiving the opening or closing instruction of the first-type optimizer submitted by the terminal, and corresponding to opening or closing the first-type optimizer.

[0089] The latency information refers to the time consumed by the to-be-optimized model for completing a complete calculation before and after the optimization by the first-type optimizer, and is a core index for measuring the running efficiency of the model.

[0090] Specifically, the application can set an optimizer pipeline, and for the to-be-optimized model, the optimizer pipeline monitors the latency of the model before and after the optimization by each optimizer, and feeds back to the user in the form of a table. After the user views the table, it can be known which optimizer has an optimization effect on the model and which optimizer has a reverse optimization effect. At the same time, each optimizer is pluggable, and the user can select to open or close which optimizer at any time. The user can select the optimizer to be opened or closed according to the table configuration given by the optimizer pipeline to optimize the model latency.

[0091] The application allows the user to manually decide to turn on or off the optimizer according to the latency result displayed, and flexibly adapts to the needs of different models or hardware scenarios.

[0092] In order to further illustrate the model optimization method of the application, please refer to Figure 2 In the Figure 2 , the model optimization method of the application can comprise: performing model training by using training data, converting the trained model into an intermediate model, optimizing the intermediate model by using a first-type optimizer, a second-type optimizer and a third-type optimizer, and deploying the optimized model on an inference engine.

[0093] Please continue to refer to Figure 3 , Figure 3 is a flowchart of the model optimization method in one of the embodiments. In Figure 3 , the converter is a Torch converter, i.e., a PyTorch model converter, and the export function is a torch.onnx.export function used for exporting the PyTorch model into an intermediate model. After the exported original intermediate model is sequentially subjected to the second-type optimizer, the third-type optimizer and the first-type optimizer, the final optimized model is obtained.

[0094] In a second aspect, the application provides a model optimization device, as shown in Figure 4 The model optimization device comprises a conversion module 41, an identification module 42, an insertion module 43 and a determination module 44, wherein: The conversion module 41 is configured to convert the model to be optimized into an intermediate model, the identification module 42 is configured to identify, by the first type of optimizer, to-be-optimized operators in the intermediate model according to a pre-configured benchmark operator structure diagram, the to-be-optimized operators include at least two, and the benchmark operator structure diagram is configured according to time delay information of operator fusion in historical data, the insertion module 43 is configured to insert a plug-in operator in a connection link between adjacent to-be-optimized operators to obtain a first optimization result, the plug-in operator is used to block fusion between the to-be-optimized operators, and the determination module 44 is configured to determine an optimized model according to the first optimization result.

[0095] In some embodiments, the optimization further includes an adjustment module (not shown in the figure) configured to acquire, by the second type of optimizer, operator specifications of the operators in the intermediate model, adjust the operator specifications according to parsing rules when the operator specifications do not match the parsing rules, and obtain a second optimization result, the inference engine is an inference engine to which the optimized model is deployed, the operator specifications include attribute information, input quantity, and an operator type of the operator, and the identification module 42 is configured to identify, from the second optimization result, the to-be-optimized operators when identifying the to-be-optimized operators in the intermediate model by the first type of optimizer.

[0096] In some embodiments, the adjustment module can create a new operator according to the operator type specified in the parsing rules and the function of the operator before adjustment, and replace the operator before adjustment with the new operator to adjust the operator type.

[0097] In some embodiments, the benchmark operator structure diagram includes multiple benchmark operator structure diagrams, each benchmark operator structure diagram corresponds to a group of to-be-optimized operators, and the insertion module 43 can insert plug-in operators in connection links between adjacent operators of each group of to-be-optimized operators to obtain the first optimization result by the first type of optimizer.

[0098] In some embodiments, the identification module 42 can acquire, according to time delay information of operator fusion in historical data, a time delay difference between each operator in the intermediate model and other operators after fusion and before fusion, and configure the benchmark operator structure diagram according to operators whose time delay difference is greater than a preset threshold and corresponding operator connection relationships.

[0099] In some embodiments, the determining module 44 can further detect each operator of the intermediate model by the third type of optimizer, perform operator merging on two or more operators that are positionally continuous, functionally identical, and of the same type on the same connection link, and / or perform operator fusion on two or more operators that are positionally continuous on the same connection link by mathematical transformation or parameter injection by the third type of optimizer, and / or detect attribute information of each operator and whether input dimensions of each operator are respectively consistent with preconfigured target attribute information and target input dimensions, and if not, adjust the attribute information and the input dimensions of the operator to be optimized to the target attribute information and the target input dimensions respectively to obtain a third optimization result, and the identifying module is configured to identify the operator to be optimized from the third optimization result when the first type of optimizer identifies the operator to be optimized in the intermediate model.

[0100] In some embodiments, the determining module 44 can further: traverse each operator on the same link in the computation graph of the intermediate model by the third type of optimizer, and when there are identical operators on the same link, determine whether the output result of the computation graph remains unchanged before and after position interchanging of each identical operator with a neighboring operator on the input side by the third type of optimizer, and if so, perform the position interchanging.

[0101] In a third aspect, the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the model optimization method provided in any one of the embodiments of the first aspect of the present application when executing the computer program.

[0102] In one embodiment, the computer device can be a server, and its internal structure diagram can be as shown in Figure 5 The computer device comprises a processor, a memory, a network interface, and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the model optimization method.

[0103] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the model optimization method provided in any one of the embodiments of the first aspect of the present application.

[0104] The computer readable storage medium can be Figure 5 the computer readable storage medium in the computer device shown in

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus DRAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0106] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0107] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A model optimization method, characterized by, The method comprises: Converting a model to be optimized into an intermediate model; Identifying, by a first type of optimizer, operators to be optimized in the intermediate model according to a pre-configured benchmark operator structure diagram, the operators to be optimized comprising at least two, and the benchmark operator structure diagram being configured according to time delay information of operator fusion in historical data; Inserting, by the first type of optimizer, a plug-in operator into a connection link between adjacent operators to be optimized to obtain a first optimization result, the plug-in operator being used to block fusion between the operators to be optimized; Determining an optimized model according to the first optimization result.

2. The method of claim 1, wherein, The method further comprises: Obtaining, by a second type of optimizer, operator specifications of each operator in the intermediate model, and when the operator specifications do not match parsing rules of an inference engine, adjusting the operator specifications according to the parsing rules to obtain a second optimization result, the inference engine being an inference engine to which the optimized model is deployed, and the operator specifications comprising attribute information, input quantity, and operator type of the operator; When identifying, by the first type of optimizer, the operators to be optimized in the intermediate model, identifying the operators to be optimized from the second optimization result.

3. The method of claim 2, wherein, The process of adjusting the operator type according to the parsing rules comprises: Creating a new operator according to the operator type specified in the parsing rules and the function of the operator before adjustment, and replacing the operator before adjustment with the new operator to adjust the operator type.

4. The method of claim 1, wherein, The benchmark operator structure diagram comprises a plurality of benchmark operator structure diagrams, each of which corresponds to a group of operators to be optimized, and the first optimization result is obtained by inserting, by the first type of optimizer, a plug-in operator into a connection link between adjacent operators to be optimized, which comprises: Inserting, by the first type of optimizer, a plug-in operator into a connection link between adjacent operators to be optimized in each group to obtain a first optimization result.

5. The method of claim 1, wherein, The method further comprises: Obtaining, according to time delay information of operator fusion in the historical data, a time delay difference value of each operator in the intermediate model before and after fusion with other operators; Configuring the benchmark operator structure diagram according to operators whose time delay difference values are greater than a preset threshold and corresponding operator connection relationships.

6. The method of claim 1, wherein, Before the first optimization result is obtained by inserting, by the first type of optimizer, a plug-in operator into a connection link between adjacent operators to be optimized, the method further comprises: Detecting, by a third type of optimizer, each operator in the intermediate model, and performing operator merging on two or more operators that are positionally continuous, functionally identical, and of the same type on the same connection link; and / or Performing, by the third type of optimizer, operator fusion on two or more operators that are positionally continuous on the same connection link through mathematical transformation or parameter injection; and / or Detecting, by the third type of optimizer, whether attribute information and input dimensions of each operator are respectively consistent with pre-configured target attribute and target input dimensions, and if not, adjusting the attribute information and input dimensions of the operators to be optimized to the target attribute and the target input dimensions respectively to obtain a third optimization result; When identifying, by the first type of optimizer, the operators to be optimized in the intermediate model, identifying the operators to be optimized from the third optimization result.

7. The method of claim 1, wherein, The method further comprises: traversing, by the third type of optimizer, each operator on the same link in a computation graph of the intermediate model; when there are same operators on the same link, judging, by the third type of optimizer, whether the output result of the computation graph remains unchanged before and after position interchanging of each of the same operators respectively with adjacent operators on the input side, and if yes, performing the position interchanging.

8. A model optimization apparatus characterized by comprising: The device comprises: a conversion module configured to convert a model to be optimized into an intermediate model; an identification module configured to identify, by a first type of optimizer, to-be-optimized operators in the intermediate model according to a pre-configured benchmark operator structure diagram, the to-be-optimized operators comprising at least two, and the benchmark operator structure diagram being configured according to time delay information of operator fusion in historical data; an insertion module configured to insert, by the first type of optimizer, a plug-in operator in a connection link between adjacent to-be-optimized operators to obtain a first optimization result, the plug-in operator being used to block fusion between the to-be-optimized operators; a determination module configured to determine an optimized model according to the first optimization result.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images