Model conversion method

Abstract

The present invention provides a model conversion method which can be applied to the processing of a source model in artificial intelligence. The model conversion method comprises the following steps: exporting the source model into a model to be processed, wherein the model to be processed has an intermediate format; generating a first model conversion tool and a first equivalent conversion operator database according to a specification of a first deployment end; and using the first model conversion tool and the first equivalent conversion operator database to convert the model to be processed into a first target model so that the first target model can run on the first deployment end.

Description

Technical Field

[0001] The present invention relates to a model conversion method, a computer program including instructions, and a computer-readable storage medium, and in particular to a model conversion method applied to an artificial intelligence model, a computer program including instructions, and a computer-readable storage medium. Background Art

[0002] In the current field of artificial intelligence technology, there are already many deep learning frameworks launched by different organizations, including PyTorch, TensorFlow, Caffe, and Keras. The different types of basic models established by the above-mentioned different types of deep learning frameworks will of course have differences in content between the source models completed after training. For example, the structures and operators (operators, referred to as ops) contained in their contents will have different contents. Moreover, the source model after the above training is highly likely to be deployed to the artificial intelligence chip at the edge (or deployment end) for application. However, the artificial intelligence chips used at the edge are likely to have different types and versions, such as different models of chips launched by common companies such as Hailo, Qualcomm, Cambrian Technology, Rockchip Electronics, or even different versions of chips launched by the same company. Therefore, when the source model after training does not completely correspond to the artificial intelligence chip (AI Chip) used at the edge, the source model usually cannot be directly deployed to the edge. At this time, a model conversion tool corresponding to the artificial intelligence chip at the edge is needed (such as the Snapdragon Neural Processing Engine (SNPE) launched by Qualcomm for its own chips or the Hailo tool launched by Hailo for its own chips, etc.) to convert the source model into a target model (Target Model) corresponding to the artificial intelligence chip, and then the target model can be successfully deployed to the edge.

[0003] However, there are usually two ways to convert the source model to the target model today: the first conversion method is direct conversion, that is, converting the source model directly to the target model. But in this way, the artificial intelligence chip manufacturer must produce different types of model conversion tools in response to different types of deep learning frameworks. So a second conversion method was developed, which first converts the source model to an intermediate format. This intermediate format can be an open file format developed and designed for machine learning (such as Open Neural Network Exchange, referred to as ONNX), which can be used to store trained model data and allow the stored trained model data to be exchanged between different platforms. This intermediate format allows different artificial intelligence frameworks to store model data in the same format. In this way, the artificial intelligence chip manufacturer only needs to produce a model conversion tool in response to the intermediate format, and the model conversion tool can smoothly convert the intermediate format into the corresponding target model.

[0004] Therefore, after obtaining the type information of a source model (for example, belonging to a framework such as PyTorch, TensorFlow, Caffe or Keras) and the type and version information of the artificial intelligence chip at the edge (or deployment end), the above two conversion methods can determine the conversion method (direct conversion or conversion through an intermediate format) and the model conversion tool (for example, the above SNPE or Hailo tool, etc.) from the source model to the target model. However, the inventors found that this type of prior art does not take into account the problem that the model conversion tool does not support certain structures and operators (operators, abbreviated as ops) in the source model to be processed.

[0005] The inventor believes that the main reason for the above problems is that with the development of technology, the model iteration speed is getting faster and faster (for example, the version of the YOLO (You Only Look Once) object detection model has developed from v1 in 2015 to v9 today), and many new versions of the model have structures and operators that have never appeared in the old versions of the model. Therefore, when the update speed of the artificial intelligence chip and its model conversion tool at the edge cannot keep up with the iteration speed of the model (or when the chip purchased by the company's products is an older model), the model conversion tool will not support some new structures and operators, and thus cannot effectively meet future needs. Therefore, the present invention proposes the following technical means to solve the deficiencies of the previous solution. Summary of the invention

[0006] The present invention is a model conversion method, which is applied to the processing of a source model in artificial intelligence. The model conversion method includes the following steps: exporting the source model into a model to be processed, and the model to be processed has an intermediate format; generating a first model conversion tool and a first equivalent conversion operator database according to a specification of a first deployment end; and using the first model conversion tool and the first equivalent conversion operator database to convert the model to be processed into a first target model, so that the first target model can run on the first deployment end.

[0007] According to the above conception, the model conversion method described in the present invention, wherein the intermediate format is an open file format developed and designed for machine learning, used to store trained model data, and the specification of the first deployment end is a first execution environment specification.

[0008] According to the above concept, the model conversion method of the present invention further includes the following steps: splitting the model to be processed into a first small model and a second small model;

[0009] According to the above concept, the model conversion method described in the present invention also includes the following steps: using a simplification tool program to simplify the model to be processed, so as to remove redundant nodes in the model to be processed and generate a simplified model to be processed; and using the first model conversion tool and the first equivalent conversion operator database to convert the simplified model to be processed into the first target model, so that the first target model can run on the first deployment end.

[0010] The first model conversion tool and the first equivalent conversion operator database are used to convert the split first small model and the second small model respectively, and then converted into a first target small model and a second target small model respectively, and the first target model includes the first target small model and the second target small model.

[0011] According to the above conception, the model conversion method described in the present invention, wherein the first small model includes a backbone module and a neck module, and the method of using the first model conversion tool to convert the first small model includes the following steps: when the process of the first model conversion tool converting the first small model into the first target small model is interrupted and an error report information is generated for an operator, it is determined whether the operator in the error report information has a corresponding equivalent conversion operator in the first equivalent conversion operator database; and if the operator in the error report information has a corresponding equivalent conversion operator in the first equivalent conversion operator database, the operator is subjected to an equivalent conversion, and then the first model conversion tool is used to continue converting the first small model that has undergone the equivalent conversion.

[0012] According to the above conception, the model conversion method described in the present invention, wherein the equivalent conversion includes the following steps: deleting the operator in the first small model; inserting the equivalent operator at the corresponding position in the first small model, the parameters of the equivalent operator are exactly the same as the parameters of the operator; and modifying the input name of the next operator connected to the operator to a name corresponding to the equivalent operator.

[0013] According to the above conception, the model conversion method described in the present invention, wherein if it is determined that the operator in the error report information does not have a corresponding equivalent conversion operator in the equivalent conversion operator database, the operator is added to an unsupported operator list for recording; a non-equivalent conversion is performed on the operator; and the first model conversion tool is used to continue converting the first small model that has undergone the non-equivalent conversion.

[0014] According to the above conception, the model conversion method described in the present invention, wherein the non-equivalent conversion includes the following steps: using a parameterless operator or an operator with a fixed parameter to perform the non-equivalent conversion; and continuing to use the first model conversion tool to perform model conversion after the non-equivalent conversion is completed until the first target small model is successfully output and the list of unsupported operators is output; wherein the script of the source model is used to obtain an input number, an input dimension, an output number and an output dimension of the operator, and the fixed parameter is calculated based on the input number, the input dimension, the output number and the output dimension of the operator.

[0015] According to the above conception, the model conversion method described in the present invention, wherein the second small model includes a header module and a post-processing module, and the method of using the first model conversion tool to convert the second small model includes the following steps: using the first model conversion tool to convert the second small model into a second target small model; if the process is interrupted and an error report information is generated for an operator, then judging whether the operator in the error report information has a corresponding equivalent conversion operator in an equivalent conversion operator database; if it is judged to exist, then performing an equivalent conversion on the operator; and if it is judged not to exist, then performing a deletion operation on the operator.

[0016] According to the above conception, the model conversion method described in the present invention outputs the second target small model that has completed the conversion when the conversion is successfully completed, and then ends the conversion process.

[0017] According to the above conception, the model conversion method described in the present invention, wherein the deletion operation includes the following steps: deleting the operator and all subsequently connected operators; specifying the output of the previous operator connected to the operator as a new output; and if the output of a related operator subsequently connected to the operator and the output of another side-branch operator are used together as the input of a multi-input operator, then deleting the multi-input operator and all subsequently connected operators, and specifying the output of the side-branch operator as a new output.

[0018] Another aspect of the present invention is a computer program including instructions, which, when executed on a computer, control the computer to complete a model conversion method, the model conversion method comprising the following steps: exporting an artificial intelligence source model into a model to be processed, the model to be processed having an intermediate format; generating a first model conversion tool and a first equivalent conversion operator database corresponding to a specification of a first deployment end; and using the first model conversion tool and the first equivalent conversion operator database to convert the model to be processed into a first target model, so that the first target model can run on the first deployment end.

[0019] Still another aspect of the present invention is a computer-readable storage medium storing the above-mentioned computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A This is a schematic diagram of the structure fragment of a model to be processed in ONNX format.

[0021] Figure 1B It is a schematic diagram of the structure fragment of the model to be processed in ONNX format, indicating the redundant nodes.

[0022] Figure 2 This is a simplified schematic diagram of the structure fragment of the model to be processed in ONNX format.

[0023] Figure 3A It is a schematic diagram of the present invention splitting the simplified ONNX format source model into two small models.

[0024] Figure 3B The present invention uses a model conversion tool to perform model conversion on the split first small model and the second small model respectively, and then converts them into a schematic diagram of a first target small model and a second target small model respectively.

[0025] Figure 4A It is a flow chart of the present invention using a model conversion tool to convert the split first small model.

[0026] Figure 4B It is a flowchart illustrating an example of the equivalent conversion steps described above according to the present invention.

[0027] Figure 4C It is a flowchart diagram of an example of the present invention for the above-mentioned non-equivalent conversion steps.

[0028] Figure 5A It is a flow chart of the present invention using a model conversion tool to convert the split second small model.

[0029] Figure 5B It is a flowchart diagram of an example of the operation steps of the deletion operator according to the present invention.

[0030] Figure 6 It is a flow chart of the method for directly converting the retrained model of the present invention.

[0031] The reference numerals are as follows: 10: label; 30: source model; 301: backbone module; 302: neck module; 303: head module; 304: post-processing module; 31: first small model; 32: second small model; 40: model conversion tool; 41: first target small model; 42: second target small model; 43: model fragment to be equivalently converted; 430: input; 431: first operator; 432: second operator; 433: third operator; 434: output; 44: model fragment with completed equivalent conversion Segment; 45: Model fragment to be non-equivalently converted; 450: Input; 451: First operator; 452: Second operator; 453: Output; 46: Model fragment that has completed non-equivalent conversion; 53: Model fragment to be processed; 530: Input; 531: Model structure; 532: Output; 5311: First structure; 5312: Second structure; 5313: Third structure; 5314: Fourth structure; 54: Model fragment that has completed deletion processing; 540: New program code; 61: New source model; 62: Target model. DETAILED DESCRIPTION

[0032] Some specific embodiments are selected below to illustrate the implementation of the present invention. Those skilled in the art can clearly understand the technical features and effects of the present invention from the contents disclosed in this specification. However, it should be noted that the schematic diagrams of the embodiments shown in the attached drawings in this specification are only for the convenience of description and are not used to limit the scope of the present invention. The changes or adjustments in their relative relationships shall also be regarded as the scope of the present invention without substantially changing the technical contents.

[0033] It can be known from the above-mentioned prior art that in the current technical means, there are usually two ways to convert the source model (Source Model) to the target model. In order to facilitate the description of the technical means developed by the present invention, this embodiment is mainly described by the second conversion method, that is, first converting the source model to a method of expressing and storing a model to be processed in an intermediate format. For convenience, the following intermediate format example uses the common Open Neural Network Exchange (hereinafter referred to as ONNX) in the open file format for description, and the model to be processed stored in the intermediate format is processed by the method developed by the present invention, and then it can be properly converted into a target model that can be smoothly executed on the edge end (or deployment end) through the corresponding model conversion tool. The corresponding model conversion tool can be determined according to the deployment end to be deployed by the target model and an execution environment specification related to the deployment end. Because the previous technology can only get an unsupported operator at a time when executing the model conversion process, and then perform a manual equivalent conversion operation on this operator or modify the training code and retrain the model operation before starting the second conversion. If the second model conversion encounters another unsupported operator, the conversion process will be interrupted again, and then the unsupported operator must be manually converted again or the training code must be modified and the model operation must be retrained before the next model conversion can begin. Therefore, the prior art requires multiple runs of the model conversion process, multiple manual EC operations, and multiple modifications of the training code to retrain the model before the model conversion can be successful. The entire process is very cumbersome and time-consuming. The present invention is aimed at the shortcomings of traditional model conversion tools, and proposes an automatic and incremental model conversion method and related computer program that can improve its known deficiencies. By performing an equivalent conversion on unsupported operators that can be equivalently converted at one time or recording a list of operators that can be non-equivalently converted and modifying the training code to retrain the model, in addition to successfully and correctly converting the model to the structure required by the chip used at the edge, the non-equivalent conversion operator database that performs non-equivalent conversion operators on unsupported operators can also be automatically expanded as the types and number of artificial intelligence chips for model conversion increase.

[0034] First, the inventors observed that the model structure of the model to be processed in the intermediate format (such as ONNX format) usually has some redundant Identity and Expand nodes, but the model conversion tools for subsequent processing generally do not support these redundant Identity and Expand nodes. Therefore, the present invention can choose to first convert the model structure of a model to be processed in ONNX format (the content of its fragment instance can be as follows Figure 1AFor example, a simplification tool program (such as a tool program named onnxsim.simplify) can be used to simplify the structure of the model to be processed. The ONNX format model before simplification may contain some redundant nodes (such as Figure 1B As shown in the circle marked with 10 in the figure, the characteristics of these redundant nodes are, for example, that they have no parent node and are not supported by the conversion tool), while the simplified ONNX format model (for example Figure 2 The simplified model to be processed does not contain redundant nodes, making the content of the simplified model to be processed simpler and clearer. Figure 1A , Figure 1B ,as well as Figure 2 Show only a portion of the model.

[0035] Next, the inventor observed that an artificial intelligence model usually includes multiple modules. Taking the common object detection model as an example, it includes at least three modules, namely the backbone module, the neck module and the head module. An artificial intelligence model used for object detection usually extracts the features required for the task from different hidden layers, then encodes them into the backbone module, and then the neck module performs feature organization, and finally the head module makes a decision. The main purpose of the backbone module is to extract features; the neck module is a connection layer for processing and fusing features; and the head module is used to generate the final prediction. Some artificial intelligence models also include a post-processing module, and the post-processing functions include processing procedures such as decoding and non-maximum suppression (NMS). Since the role and function of each module are different, the technical means developed by the present invention can split the simplified ONNX format model according to the classification of the above modules, and then form multiple small models. Then, the method of the present invention can automatically convert the split small models into target models according to the characteristics of the models.

[0036] For example Figure 3AAs shown, the method of the present invention can split the simplified ONNX format source model (the file name can be recorded as model.onnx) into two small models. In this example, the backbone module 301+neck module 302 in the source model (model.onnx) 30 are separated as the first small model 31, and the file name can be recorded as model_1.onnx, and the remaining head module 303+post-processing module 304 are used as the second small model 32, and the file name can be recorded as model_2.onnx. Of course, the technical means developed by the present invention can also directly group the above-mentioned ONNX format model before simplification into multiple small models according to the classification of the above-mentioned modules. For example, the ONNX format model before simplification is also split into a first small model (backbone module+neck module) and a second small model (head module+post-processing module).

[0037] And as Figure 3B As shown in the functional block diagram, the model conversion tool 40 can be used to convert the split first small model 31 (model_1.onnx) and the second small model 32 (model_2.onnx) into a first target small model (model_1.target) 41 and a second target small model (model_2.target) 42. The first target small model (model_1.target) 41 and the second target small model (model_2.target) 42 are also similar in form. Figure 2 The content shown, Figure 2 It is a specific form of the simplified ONNX format source model (model.onnx) 30, but due to limited space, the present invention Figure 2 Only part of the model can be displayed. The filename extensions (target) of the first and second target small models (model_1.target, model_2.target) need to be determined according to the chip model. For example, on a Qualcomm chip, model.target is model.dlc, and on a Hailo chip, model.target is model.hef. Then, the model conversion tool 40 can be used to simultaneously start the model conversion process for the split first small model 31 and the second small model 32, so as to convert the two separately. The process details can be referred to as follows. Figure 4A and Figure 5AThe process example diagrams shown respectively have no strict regulations on the time sequence of the model conversion process of the first small model 31 and the model conversion process of the second small model 32. The order of conversion can be determined according to the actual situation, or they can be converted simultaneously.

[0038] First, the details of the conversion process of the first small model (model_1.onnx) 31 are explained. Figure 4A The method represented by the flow chart can process the first small model (model_1.onnx) 31, but it should be emphasized that the first small model 31 in this example is disassembled from the backbone module 301 + the neck module 302 of the source model (model.onnx) 30. As the main structure of a model, in view of this characteristic, the present invention adopts a strategy of not being able to delete the operators (operators, abbreviated as ops) in the first small model 31 (or specifically the backbone module 301 included therein) that are not supported by the model conversion tool corresponding to the artificial intelligence chip at the edge end (or deployment end). The preferred approach is to use the operation of replacing operators. In detail, since the main purpose of the backbone module 301 is to extract features, the method of the present invention adopts a strategy of not deleting operators (operators, abbreviated as ops) in the first small model 31 (or specifically the backbone module 301 contained therein) that are not supported by the model conversion tool corresponding to the artificial intelligence chip at the edge end (or deployment end), mainly to avoid serious errors in feature extraction. In addition, the reason for not deleting the unsupported operators in the first small model 31 (especially the backbone module 301) is to retain the original architecture of the first small model 31 (or specifically the backbone module 301 contained therein), because if the operators in the first small model 31 (especially the backbone module 301) are deleted, not only the modified artificial intelligence model needs to be retrained, but also the reduced architecture size of the backbone module 301 may have a negative impact on the model prediction effect.

[0039] So if Figure 4AAs shown, the method of this example first uses the model conversion tool 40 to convert the first small model 31 into the first target small model (model_1.target) 41 (step 401), and then determines whether the conversion can be successfully completed during the conversion process (step 402). If so, it means that the model conversion tool 40 can support the structure and operators in the first small model (model_1.onnx) 31, and then the conversion process can be terminated by directly outputting the converted first target small model (model_1.target) 41 in step 403. However, if the conversion is not successfully completed (for example, the process is interrupted and an error report is generated for an unsupported operator), step 404 is entered to determine whether the operator in the error report information has a corresponding equivalent conversion operator in an equivalent conversion operator database. If it is determined to exist, the operator is equivalently converted (hereinafter referred to as EC) (step 405). The above-mentioned equivalent conversion operator database is constructed based on the operation manual of the chip used by the edge end (or deployment end), and then obtains a list of known unsupported operators, and then finds operators that can be equivalently converted (EC) for these unsupported operators. This equivalent conversion operator database can be shared in the model conversion process of the first small model 31 and the model conversion process of another second small model 32. The parameters of these equivalent conversion (EC) operators (such as the number of inputs, input dimensions, number of outputs, output dimensions, etc.) are exactly the same as the parameters of the corresponding unsupported operators and correspond one to one. Because they are highly compatible with the source model, the first target small model 41 that only completes the equivalent conversion does not need to be retrained. For example, if SQRT(A1) is used to calculate the square root of A1, and the operator on the deployment side is SQROOT(A1), then an equivalent conversion can be performed; however, if the operator used on the deployment side to calculate the square root of A1 is SQRT(A1,B1), and the processing program stores the square root of A1 in B1, then SQRT(A1) and SQRT(A1,B1) cannot be equivalently converted, and non-equivalent conversion must be used.

[0040] In addition, if Figure 4AStep 404 in the above is judged to be non-existent (that is, the operator in the error report information is not in the equivalent conversion operator database mentioned above), indicating that the operator cannot be equivalently converted (EC). At this time, the operator can be added to a non-supported operator list (not_support_op_list) to update the record (step 406), and then the operator is non-equivalently converted (Non-Equivalent Conversion, hereinafter referred to as NEC) (step 407), and then returns to step 401 to convert the next operator until all operator conversions are successfully completed. Step 403 outputs the first target small model 41 that has completed the conversion and the latest version of the unsupported operator list. As for the non-equivalent conversion of the above step 407, the non-equivalent conversion operator database selects a parameter-free operator or an operator with randomly initialized parameters to replace the unsupported operator based on all the supported layers / operators (supported layers / ops) listed in the operation manual of the chip used by the edge end (or deployment end). In detail, the processing of non-equivalent conversion (NEC) can be that the non-equivalent conversion operator database obtains a non-equivalent conversion operator corresponding to the unsupported operator for replacement, and the selection process of the non-equivalent conversion operator can be that the non-equivalent conversion operator database first obtains the input number, input dimension, output number and output dimension of the unsupported operator (which can be obtained by script), and then the non-equivalent conversion operator database selects suitable parameter-free operators (such as Mul, Add, Concat, Sigmoid, etc.) or fixed parameter operators (calculated according to the input number, input dimension, output number and output dimension of the operator) (such as conv, pool, etc.) for replacement from all supporting layers / operators listed in the chip operation manual used by the edge end based on this information. It can be seen from the above steps that the method of the present invention will perform equivalent conversion (EC) on the unsupported operators but with corresponding equivalent conversion operators. However, if the unsupported operators do not have corresponding equivalent conversion operators, then the non-equivalent conversion operator database will be used to select parameter-free operators or operators with randomly initialized parameters to perform non-equivalent conversion (NEC). Since the non-equivalent conversion operation will cause deviations in the feature extraction results, it is necessary to retrain the modified model (the first target small model 41 that has undergone non-equivalent conversion). For example, when it is found that the unsupported operator list generated by step 403 is not empty or there are unsupported operators written in the list, it means that the chip used at the edge end contains non-equivalent conversion operators. In order to prevent deviations in the feature extraction results, the retraining will be performed. Figure 4A The model produced by the process.

[0041] For the example method step description flow chart related to the above equivalent conversion (step 405), please refer to Figure 4B , the model fragment 43 to be equivalently converted on the left side of the figure is a fragment in the first small model 31, which includes input 430, first operator 431, second operator 432, third operator 433 and output 434, and the equivalent conversion process usually includes the following three steps: deleting the original first operator 431 (step 4051); inserting the first equivalent operator 441 at the corresponding position (step 4052) and modifying the input name (input_name) of the next operator connected to the first operator (the second operator in this case) (step 4053), for example, modifying the input name A to the input name B, and finally forming a model fragment 44 that has completed the equivalent conversion, and then continue to use the model conversion tool to perform model conversion. The parameters of these equivalent conversion (EC) operators (such as the number of inputs, input dimensions, number of outputs, output dimensions, etc.) are exactly the same as the parameters of the corresponding unsupported operators. Therefore, if it is determined that it does not exist (for example, the operator in the above error report information is not in the above equivalent conversion operator database), step 406 and step 407 are entered to add the operator to a list of unsupported operators (not_support_op_list) for recording, and then perform non-equivalent conversion (NEC) on the operator.

[0042] For the example flow diagram of this non-equivalent conversion, please refer to Figure 4C, the non-equivalent conversion model fragment 45 on the left side of the figure is a fragment in the first small model 31, which includes an input 450, a first operator 451, a second operator 452 and an output 453, which includes the following steps: wherein step 4071 is to obtain the number of inputs, input dimensions, number of outputs and output dimensions of the operator (which can be obtained using the script of the source model), and then use parameter-free operators (such as Mul, Add, Concat, Sigmoid, etc.) or fixed parameters (calculated according to the number of inputs, input dimensions, number of outputs and output dimensions of the operator) according to these values ​​(such as conv, pool, etc.) for replacement (step 4072), insert the first non-equivalent operator 461 at the corresponding position to complete the replacement, and modify the input name (input_name) of the next operator connected to the first operator (the second operator in this case) (step 4073), for example, modify the input name C to the input name D, and after forming the model fragment 46 that has completed the non-equivalent conversion, continue to use the model conversion tool to convert the model until all the content has been converted, then enter the above step 403, and then output the first target small model 41 that has completed the conversion and the list of unsupported operators (all operators that have not completed the equivalent conversion are included). It is emphasized again that since the operation of non-equivalent conversion will cause deviations in the result of feature extraction, the first target small model 41 that has undergone non-equivalent conversion needs to be retrained (for example, when the list of unsupported operators (not_support_op_list) is not empty or there are unsupported operators written in the list).

[0043] At this point, in addition to obtaining a list of unsupported operators at this stage (the list contains all operators supported by the model conversion tool that cannot be equivalently converted), the present invention also obtains the first target small model 41 that has completed the conversion (these unsupported operators have been replaced with non-equivalent operators). In this way, the first target small model 41 that has completed the conversion obtained at this stage can run normally. However, some operators in the first target small model 41 are replaced with non-equivalent operators, so there may be performance differences. In detail, after the non-equivalent conversion is completed, the model conversion tool 40 can continue to be used to perform model conversion until the source model (model_1.onnx) successfully outputs the first target small model 41 (model_1.target) and the unsupported operator list (not_support_op_list) after passing through the model conversion tool. At this point, we can obtain all operators that cannot be equivalently converted and get the unsupported operator list (not_support_op_list), and when the unsupported operator list is not empty or there are unsupported operators written in the list, the first target small model is retrained.

[0044] Therefore, when the technician finds that the edge artificial intelligence chip has an unsupported operator, and finds that the unsupported operator has a corresponding equivalent conversion operator, the program code of the equivalent conversion operator is incorporated into the equivalent conversion operator database, so that the unsupported operators of the first target small model and the second target small model can be equivalently converted. On the other hand, for unsupported operators for which no corresponding equivalent conversion operators are found, the non-equivalent operator database is allowed to select operators that can perform non-equivalent conversions on the unsupported operator from the operators in the chip operation manual used at the edge, and perform non-equivalent conversions on the unsupported operator. Therefore, the non-equivalent conversion operator database can automatically perform non-equivalent conversions on the unsupported operators of the first target small model, and the non-equivalent conversion operator database can be automatically expanded as the types and number of artificial intelligence chips processed by the model are increased. Currently, commonly used equivalent conversions include changing resize to convtranspose, changing hardsigmoid to sigmoid, changing averagepool to conv, etc. (while non-equivalent replacements are completed using parameter-free operators such as Mul, Add, Concat, Sigmoid, etc.). The content differences between the above-mentioned different model conversion tools and multiple equivalent conversion operator databases or non-equivalent conversion operator databases are mainly different according to the execution environment specifications of the deployment end. Therefore, when building the multiple model conversion tools and multiple equivalent conversion operator databases, you can also consider naming them according to the execution environment specifications of the deployment end to facilitate subsequent search.

[0045] In addition, since the second small model 32 in this example is separated from the head module 303 + post-processing module 304 in the source model (model.onnx) 30 as a non-main structure of a model, in this conversion process, for an unsupported operator that cannot complete an equivalent conversion, the operation of deleting the unsupported operator can be selected. The model conversion tool can also be used to convert the second small model 32 into the second target small model (model_2.target) 42, for example Figure 5AIn the example flow chart shown, for the second small model 32, the model conversion tool 40 is first used to convert it into the second target small model (model_2.target) 42 (step 501), and then it is determined whether the conversion can be successfully completed (step 502). If so, it means that the model conversion tool 40 can support the structure and operators in the second small model 32, and then it enters step 503 to output the second target small model 42 that has completed the conversion, and then ends the conversion process. However, if the conversion cannot be completed successfully (for example, the process is interrupted and an error report information is generated for a certain operator), it enters step 504 to determine whether the operator in the error report information has a corresponding equivalent conversion operator in the equivalent conversion operator database. If it is determined to exist, the operator is equivalently converted (step 505). Therefore, if an operator appears in both the first small model 31 and the second small model 32 and the corresponding equivalent conversion operator can be found in the equivalent conversion operator database, it will be in the equivalent conversion operator database. Figure 4A and Figure 5A Each of them is equivalently converted (EC). In addition, if it is judged that it does not exist (for example, the operator in the error report information is not in the above-mentioned equivalent conversion operator database), the operator is deleted and a new output is set (step 506). It should be emphasized here that for unsupported operators that cannot complete equivalent conversion, you can choose to delete the unsupported operator and record the deleted operator. However, because the operator deletion operation may cause deviations in the feature fusion at the deployment end, the second target small model (model_2.target) 42 that has performed the operator deletion operation needs to implement the function of the deleted operator with code and then re-train the model. Because these operations occur in the model conversion / deployment / application stage, for the unsupported operators deleted in the second small model 32 (model_2.onnx), the original structure of the second small model 32 (model_2.onnx) can be used for training during model training. The training process will be described later through Figure 6 In addition, those skilled in the art know how to use the code to implement the function of the deleted operator, so it will not be described here.

[0046] The specific details of step 506 can be implemented by the following steps: first, delete the operator and all subsequent operators connected to it; specify the output of the previous operator connected to the operator as a new output; if the output of a related operator subsequently connected to the operator and the output of another side branch (side_branch) operator are used together as the input of a multi-input operator (multiple_input_op), then it is also necessary to delete the multi-input operator and all subsequent operators connected to it, and specify the output of the side branch (side_branch) operator as a new output. After the above operations are completed, continue to use the model conversion tool to perform model conversion until the second small model 32 successfully outputs the second target small model 42 after passing through the model conversion tool. An example of the above method can be found in Figure 5B As shown in the figure, the model fragment 53 to be processed shown on the left side is a fragment in the second small model 32, which includes input 530, model structure 531 and output 532, and the model structure 531 includes a first structure 5311, a second structure 5312, a third structure 5313 and a fourth structure 5314. In this example, since the fourth structure 5314 contains operators that are not supported by the conversion tool and cannot be equivalently converted, we delete the fourth structure 5314 in the deployment phase to form Figure 5B The model fragment 54 on the right side that has completed the deletion process includes: deleting the fourth structure 5314 and all subsequent connected operators; renaming the outputs of the upper-level operators (the first structure 5311 and the third structure 5313) connected to the fourth structure 5314 as new output names (such as the output name E and the output name F in the example); and then using the new program code 540 to re-implement the function of the deleted fourth structure 5314, so that the final functional output of the right block is consistent with the output of the left block.

[0047] Finally, since the first target small model 41 will use non-equivalent operators for replacement, there may be performance differences (such as causing deviations in feature extraction) and the model needs to be retrained; on the other hand, when the unsupported operators that cannot complete the equivalent conversion are directly deleted and the deleted functions are implemented by code, the second target small model 42 also needs to be retrained (because feature fusion may cause deviations). For the retrained model, it is converted back to a new source model (model.onnx), and then without splitting, it is retrained. Figure 6The method flow chart shown in the figure is used to directly convert the source model (model.target) to obtain the final target model (model.target). Since the new source model (model.onnx) 61 has undergone equivalent conversion and non-equivalent conversion, and even some non-equivalent conversion operators have been replaced by additionally modifying the training program code, it can be successfully processed by the model conversion tool 40 (step 601) and directly converted to a target model (model.target) 62 to end the conversion process. In addition, if Figure 6 When it is found during the conversion process that the retrained model still has unsupported operators, the equivalent conversion operator database can be used to perform equivalent conversion on the unsupported operators.

[0048] In summary, the present invention develops an automatic and incremental open file format conversion method for the model to be processed. The present invention can first simplify and split the original open file format, and then automatically convert the target model to the split small model according to the model characteristics. The advantages of the present invention are as follows: First: For the small model of the backbone module 301 + the neck module 302, the present invention can find all the operators that are not supported by the model conversion tool at one time. If these operators are in the equivalent conversion operator database, equivalent conversion (EC) is performed. If the unsupported operator does not have a corresponding equivalent conversion operator, it is added to the unsupported operator list (all operators that have not completed equivalent conversion are included) and can be changed to the non-equivalent conversion operator database to select a parameterless operator or an operator that can randomly initialize parameters to perform a non-equivalent conversion (NEC). Second: For the small model of the head module 303 + post-processing module 304, the present invention can also find all operators that are not supported by the model conversion tool at one time. If these operators are in the equivalent conversion operator database, equivalent conversion is performed, otherwise deletion and new output node operation are performed. Third: Splitting the model into at least two small models can avoid deleting too many model structures when using the deletion method. As can be seen from the above steps, the method of the present invention will perform equivalent conversion (EC) on the part of the unsupported operator but with the corresponding equivalent conversion operator, but if the unsupported operator does not have a corresponding equivalent conversion operator, it will go to the non-equivalent conversion operator database to select a parameter-free operator or an operator with randomly initialized parameters to perform non-equivalent conversion (NEC). Since the operation of non-equivalent conversion (NEC) will cause the result of feature extraction to be biased, it is necessary to retrain the modified model (the first target small model 41 with non-equivalent conversion).

[0049] The model conversion method developed by the present invention can be implemented using a computer program containing instructions. When the instructions are executed on a computer or a server, they are used to control the computer or the server to complete the model conversion method. The computer program can be stored in a computer-readable storage medium.

[0050] The above is only a preferred embodiment of the present invention, but it is not intended to limit the scope of the present invention. Any person familiar with this technology can make further improvements and changes on this basis without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope defined by the claims of this application.

Claims

1. A model conversion method is applied to the processing of a source model in artificial intelligence, and the model conversion method comprises the following steps: Exporting the source model into a to-be-processed model, where the to-be-processed model has an intermediate format; Corresponding to the first model conversion tool and the first equivalent conversion operator database according to the specification of the first deployment end; and The first model conversion tool and the first equivalent conversion operator database are used to convert the model to be processed into a first target model, so that the first target model can run on the first deployment end.

2. The model conversion method according to claim 1, wherein: The intermediate format is an open file format developed and designed for machine learning, and is used to store trained model data. The specification of the first deployment end is the first execution environment specification.

3. The model conversion method according to claim 1, wherein: The following steps are also included: Splitting the model to be processed into a first small model and a second small model; and The first model conversion tool and the first equivalent conversion operator database are used to convert the split first small model and the second small model respectively, and then converted into a first target small model and a second target small model respectively. The first target model includes the first target small model and the second target small model.

4. The model conversion method according to claim 3, wherein: The following steps are also included: Using a simplification tool program to simplify the model to be processed, so as to remove redundant nodes in the model to be processed and generate a simplified model to be processed; and The first model conversion tool and the first equivalent conversion operator database are used to convert the simplified model to be processed into the first target model, so that the first target model can run on the first deployment end.

5. The model conversion method according to claim 3, wherein: The first small model includes a backbone module and a neck module, and the method for converting the first small model using the first model conversion tool includes the following steps: When the process of converting the first small model to the first target small model by the first model conversion tool is interrupted and an error report information is generated for the operator, determining whether the operator in the error report information has a corresponding equivalent conversion operator in the first equivalent conversion operator database; and If the operator in the error report information has a corresponding equivalent conversion operator in the first equivalent conversion operator database, the operator is equivalently converted, and then the first model conversion tool is used to continue converting the first small model that has undergone the equivalent conversion.

6. The model conversion method according to claim 5, wherein: The equivalent conversion includes the following steps: Deleting the operator in the first small model; Inserting an equivalent operator at a corresponding position in the first small model, wherein the parameters of the equivalent operator are exactly the same as those of the operator; and The input name of the next operator connected to the operator is modified to a name corresponding to the equivalent operator.

7. The model conversion method according to claim 5, wherein: If it is determined that the operator in the error report information does not have a corresponding equivalent conversion operator in the equivalent conversion operator database, then the operator is added to the unsupported operator list for recording; Performing a non-equivalence transformation on the operator; and The first model conversion tool is used to continue converting the first small model that has undergone the non-equivalent conversion.

8. The model conversion method according to claim 7, wherein: This non-equivalence conversion includes the following steps: Using a parameterless operator or a fixed-parameter operator to perform the non-equivalent conversion; and After the non-equivalent conversion is completed, the first model conversion tool is continued to be used for model conversion until the first target small model is successfully output and the unsupported operator list is output; Among them, the script of the source model is used to obtain the input number, input dimension, output number and output dimension of the operator, and the fixed parameter is calculated based on the input number, input dimension, output number and output dimension of the operator.

9. The model conversion method according to claim 3, wherein: The second small model includes a head module and a post-processing module, and the method for converting the second small model using the first model conversion tool includes the following steps: Using the first model conversion tool to convert the second small model into a second target small model; If the process is interrupted and an error report is generated for the operator, it is determined whether the operator in the error report has a corresponding equivalent conversion operator in the equivalent conversion operator database; If it is judged to exist, the operator is converted to an equivalent state; and If it is determined not to exist, the operator is deleted.

10. The model conversion method according to claim 9, wherein: The deletion operation includes the following steps: Delete the operator and all subsequent connected operators; Assign the output of the previous operator connected to this operator as a new output; and If the output of a related operator subsequently connected to the operator and the output of another side-branch operator are used together as the input of a multi-input operator, the multi-input operator and all operators subsequently connected are deleted, and the output of the side-branch operator is designated as a new output.

Images