Transform model-based precision tuning method, system and device, and storage medium

Abstract

The invention discloses a precision adjusting and optimizing method, system and equipment based on a transformer model and a storage medium, the precision of the transformer model is quickly recovered through a heuristic iterative search mixing precision strategy, an FP32 + FP16 mixing model basically aligned with the precision of a full-precision model is obtained, meanwhile, it can be guaranteed that the reasoning performance is close to that of an original FP16 model, and the accuracy of the whole-precision model is improved. The method is a mixed precision configuration method based on heuristic iterative search, is used for optimizing the precision and reasoning performance of a transformer model, and can keep the reasoning performance close to that of a full FP16 model on FP16 precision deployment while the precision is adjusted and optimized; a heuristic strategy of an FP32 network layer is selected according to the layer type so as to maintain the precision of the model; a cos similarity index and a model precision and performance evaluation mechanism of a preset similarity threshold are combined to ensure a precision tuning result; and meanwhile, the method can be operated on different NVIDIA chip platforms, and the compatibility is good.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to deep learning GPU inference optimization technology, and specifically to a precision tuning method, system, device and storage medium based on a transformer model. Background Art

[0002] Since the Transformer model was proposed in 2017, it has been widely used and has achieved remarkable results in many fields. However, it still faces many challenges in engineering deployment, especially when using FP16 precision. Since the Transformer model has a complex network structure compared to the CNN model, some operators are prone to anomalies under FP16 precision, which eventually leads to serious precision loss of the entire model. Since the Transformer model is composed of a large number of operators and the model has random dropout locations, the method of repairing the precision by checking one by one is inefficient and cannot guarantee the quality of the precision tuning results. The above problems limit the application effect of the Transformer model under FP16 precision. Summary of the invention

[0003] In view of the above problems, the main purpose of the present invention is to design a precision tuning method, system, device and storage medium based on the Transformer model to solve the technical problems of serious point dropout and low precision repair efficiency of the Transformer model.

[0004] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A precision tuning method based on the transformer model, which runs on the NVIDIA chip platform and uses Python scripts combined with the TensorRT tool for optimization. The specific steps include:

[0006] Full-precision model reasoning to obtain true values: Use the ONNXRuntime library to load the trained ONNX source model in Python, input unlabeled sample data into the model, obtain the model's reasoning results through reasoning, and use the reasoning results as the true value of the full-precision model, recorded as GT;

[0007] Initialize FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize FP32 layer configuration, determine the network layers that maintain high precision and the network layers that are reduced to low precision;

[0008] Model conversion to deployment platform: Based on the configuration of network layer accuracy, use TensorRT's trtexec tool to convert the ONNX source model to a format supported by the deployment platform to obtain an optimized hybrid model;

[0009] Inference of the hybrid model to obtain prediction values: Using the above unlabeled sample data as input, the optimized hybrid model is inferred using the TensorRT tool to obtain the prediction values of the hybrid model for each input, denoted as Pred;

[0010] Model accuracy evaluation: Compare the output results of the true values GT and the prediction values Pred for the corresponding same inputs one by one, calculate the accuracy evaluation result S, and compare S with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and the next model inference performance evaluation is carried out. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and the FP32 layer configuration is adjusted;

[0011] Model inference performance evaluation: Use the TensorRT tool to test the inference time of the optimized hybrid model, calculate the average inference time Tavg, and compare Tavg with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; otherwise, it does not meet the requirements, and the FP32 layer configuration is adjusted;

[0012] Adjust the FP32 layer configuration: Randomly sample and select high-precision network layers, configure the obtained high-precision network layers as low-precision network layers, convert the adjusted hybrid model to the deployment platform again, and perform the next iteration;

[0013] Output the result model: After the above iterative process, when both the model accuracy and inference performance meet the requirements, output the target model to complete the accuracy tuning process of the transformer model in low-precision deployment.

[0014] As a further description of the present invention, in the initialization of the FP32 layer configuration, it includes the network layers in the set L_fp32 and the network layers in the set L_fp16;

[0015] Adopt a heuristic strategy to extract a subset L_fp32 from the set L, and denote the set of all remaining network layers as L_fp16, where L = L_fp32 + L_fp16.

[0016] As a further description of the present invention, the heuristic strategy is to traverse each network layer in the set L, analyze and select according to the type of the network layer. All network layers involved in numerical calculations and having an impact on the final accuracy, including convolutional layers and fully connected layers, are denoted as FP32, and all network layers not involved in numerical calculations or having a small impact on the final accuracy, including transpose layers and reshaping layers, are denoted as FP16;

[0017] Obtain the network layers with high precision maintained and the network layers reduced to low precision in the network layers.

[0018] As a further description of the present invention, during the process of converting the model to the deployment platform, the network layers included in the set L_fp32 are set to FP32 precision, and the network layers included in the set L_fp16 are set to FP16 precision.

[0019] As a further description of the present invention, in the above model precision evaluation, the precision evaluation result S is carried out by using the cos similarity index between tensor data, and its value range is 0 to 1, where 0 means completely similar and 1 means completely dissimilar.

[0020] As a further description of the present invention, the above model inference performance evaluation includes the following steps:

[0021] Randomly generate float-type test data that meets the model input requirements;

[0022] Use the TensorRT tool to load the optimized model, perform multiple inferences on the generated test data, and record the time consumption of each inference;

[0023] Calculate the total time consumption Tsum based on the time consumption of each inference, and obtain the average inference time Tavg as the model performance data of the current iteration. The expression is:

[0024] Tavg = Tsum / x,

[0025] where x is the number of inferences of the test data;

[0026] Compare Tavg with the preset performance threshold Tth. The preset performance threshold Tth uses the inference performance value of the full FP16 model as the benchmark;

[0027] If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, re-adjust the FP32 layer configuration, and perform the next iteration;

[0028] Record the comparison results of Tavg and Tth for each iteration.

[0029] An accuracy tuning system based on the transformer model, which is used to execute the above accuracy tuning method, including a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model accuracy evaluation module, a model inference performance evaluation module, and an output module;

[0030] The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model;

[0031] The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layer that maintains high precision and the network layer that is reduced to low precision in the network layer of the ONNX source model, where the high precision is FP32 and the low precision is FP16;

[0032] The model conversion module is used to convert the ONNX source model into a format supported by the deployment platform according to the configuration of the network layer accuracy of the full-precision model to obtain an optimized hybrid model;

[0033] The hybrid model reasoning module is used to load the optimized hybrid model and obtain the reasoning result through data input, and use the reasoning result as the predicted value of the hybrid model;

[0034] The model accuracy evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the accuracy evaluation result, and compares the accuracy evaluation result with a preset similarity threshold, and judges whether the accuracy of the hybrid model meets the target requirements through the comparison result. For the hybrid model that meets the target requirements, the model reasoning performance evaluation module is entered for evaluation, and for the hybrid model that does not meet the target requirements, the full-precision model configuration module is returned to reconfigure and the next iteration is performed;

[0035] The model reasoning performance evaluation module calculates the reasoning time of the hybrid model, and compares the reasoning time with a preset performance threshold, and determines whether the performance of the hybrid model meets the requirements through the comparison result. For the hybrid model that meets the requirements, the module enters the output module for output, and for the hybrid model that does not meet the requirements, the module returns to the full-precision model configuration module for reconfiguration and performs the next iteration;

[0036] The output module is used to output a model that meets the target requirements and the reasoning performance requirements, thereby realizing the tuning of the transformer model.

[0037] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the memory is used to store a computer program;

[0038] The processor is used to execute the above-mentioned accuracy tuning method by running the computer program stored in the memory.

[0039] A computer-readable storage medium stores a computer program, wherein the computer program implements the above-mentioned accuracy tuning method when executed by a processor.

[0040] Compared with the prior art, the technical effects of the present invention are:

[0041] The present invention provides a precision tuning method, system, device and storage medium based on a transformer model. The precision of the transformer model is quickly restored by heuristically iteratively searching a mixed precision strategy, and an FP32+FP16 mixed model that is basically aligned with the full precision model precision is obtained. At the same time, the reasoning performance can also be guaranteed to be close to the original FP16 model. The method is a mixed precision configuration method based on heuristic iterative search, which is used to optimize the precision and reasoning performance of the transformer model. While the precision is tuned, the reasoning performance can be maintained close to that of the full FP16 model on FP16 precision deployment; the precision of the model is maintained by adopting a heuristic strategy of selecting FP32 network layers according to layer types; the model precision and performance evaluation mechanism combining the cos similarity index and the preset similarity threshold is guaranteed to obtain the result of precision tuning; at the same time, the method can run on different NVIDIA chip platforms and has good compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the overall method flow of the present invention; DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings:

[0044] In one embodiment of the present invention, a precision tuning method based on a transformer model is disclosed, referring to Figure 1 As shown in the figure, by heuristically iteratively searching for a mixed precision strategy, the accuracy of the transformer model is quickly restored, and an FP32+FP16 hybrid model that is basically aligned with the full-precision model accuracy is obtained, while also ensuring that the inference performance is close to that of the original FP16 model.

[0045] Specifically, in this embodiment, the above method is run on the NVIDIA chip platform, and is optimized using Python scripts combined with the TensorRT tool, specifically including the following steps:

[0046] Full-precision model reasoning to obtain true values: Use the ONNXRuntime library to load the trained ONNX source model in Python, input unlabeled sample data into the model, obtain the model's reasoning results through reasoning, and use the reasoning results as the true value of the full-precision model, recorded as GT;

[0047] Initialize FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize FP32 layer configuration, determine the network layers that maintain high precision and the network layers that are reduced to low precision;

[0048] Model conversion to the deployment platform: According to the configuration of the network layer accuracy, use the trtexec tool of TensorRT to convert the ONNX source model to the format supported by the deployment platform, and obtain the optimized hybrid model;

[0049] Inference of the hybrid model to obtain prediction values: Use the above unlabeled sample data as input, and use the TensorRT tool to perform inference on the optimized hybrid model to obtain the prediction values of the hybrid model for each input, denoted as Pred;

[0050] Model accuracy evaluation: Compare the output results of the corresponding same input in the true value GT and the prediction value Pred one by one, calculate the accuracy evaluation result S, and compare S with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and proceed to the next model inference performance evaluation. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and adjust the FP32 layer configuration;

[0051] Model inference performance evaluation: Use the TensorRT tool to perform an inference time-consuming test on the optimized hybrid model, calculate the average inference time Tavg, and compare Tavg with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; otherwise, it does not meet the requirements, and adjust the FP32 layer configuration;

[0052] Adjust the FP32 layer configuration: Randomly sample and select high-precision network layers, configure the obtained high-precision network layers as low-precision network layers, convert the adjusted hybrid model to the deployment platform again, and perform the next iteration;

[0053] Output the result model: After the above iterative process, when both the model accuracy and inference performance meet the requirements, output the target model to complete the accuracy tuning process of the transformer model in low-precision deployment.

[0054] In this embodiment, the process of the above accuracy tuning method specifically includes the following content:

[0055] 1. Application scenario description: In this embodiment, the application scenarios where the accuracy tuning method is applied include, but are not limited to, models such as image classification, object detection, and semantic segmentation, nor are they limited to the algorithm implementation scenarios in the autonomous driving industry. Any other fields or applications involving accuracy tuning of the transformer model can use the method of this embodiment to achieve accuracy tuning. The method of this embodiment is a general accuracy tuning strategy for the transformer model.

[0056] 2. Operating platform description: The precision tuning method in this embodiment runs on NVIDIA chip platforms such as orin\xavier, and the entire optimization process can be implemented using python scripts and TensorRT tools.

[0057] 3. Full-precision model reasoning to obtain the true value: The ONNX source model is inferred on the Python side using the ONNXRuntime library to obtain the model reasoning result as the true value of the full-precision model, denoted as GT. In this embodiment, a certain amount of unlabeled sample data is used as the input of the model.

[0058] 4. Initialize the FP32 layer configuration: For all network layer sets L in the ONNX source model, a set L_fp32 is extracted using an heuristic strategy, and all remaining network layer sets are recorded as L_fp16, L = L_fp32 + L_fp16.

[0059] The heuristic strategy is to traverse each network layer in the set L, analyze and select according to the type of network layer, all network layers involving numerical calculations and affecting the final accuracy, including but not limited to convolutional layers and fully connected layers, are recorded as FP32, and all network layers that do not involve numerical calculations or have little impact on the final accuracy, including but not limited to transposition layers and reshaping layers, are recorded as FP16; the network layers that maintain high precision and the network layers that are reduced to low precision are obtained.

[0060] 5. Model conversion to the deployment platform: Based on the L_fp32 and L_fp16 information obtained above, continue to use TensorRT's trtexec tool to convert the ONNX source model to the format supported by the deployment platform to obtain the optimized hybrid model; during the model conversion process, set the network layers contained in the set L_fp32 to FP32 precision, and set the network layers contained in the set L_fp16 to FP16 precision.

[0061] It should be noted that the above process of converting the ONNX source model on the NVIDIA chip platform uses the trtexec model conversion tool of the tensorrt software package under the NVIDIA chip platform; this part of the conversion process is existing technology.

[0062] 6. Hybrid model reasoning to obtain prediction values: Continue to use the TensorRT tool to reason about the optimized hybrid model, using the above-mentioned unlabeled sample data as input, and collect the reasoning results as the hybrid model prediction values ​​of this iteration, recorded as Pred.

[0063] 7. Model accuracy evaluation: Compare the output results of the true value GT and the predicted value Pred corresponding to the same input sample one by one, and calculate the accuracy evaluation result S. In this embodiment, the accuracy evaluation result S is performed using the cos similarity index between tensor data. The value range of S is 0 to 1, where 0 represents complete similarity and 1 represents complete dissimilarity.

[0064] In this embodiment, the cos similarity calculation formula between the above tensor data is:

[0065]

[0066] Among them, A and B are two tensors, and the cos similarity is equal to the dot product of AB divided by the modulus of the vector.

[0067] The accuracy evaluation result S is compared with a preset similarity threshold Sth. In this embodiment, Sth is generally set to 0.98, or is set according to actual needs.

[0068] If S>Sth, the prediction accuracy of the model meets the target requirements, and the next step of model reasoning performance evaluation is carried out. If S≤Sth, the prediction accuracy of the model does not meet the target requirements, and the FP32 layer configuration is adjusted for the next round of iteration.

[0069] 8. Model reasoning performance evaluation: Use the TensorRT tool to test the reasoning time of the hybrid model in the deployment platform format. Specifically, the model reasoning performance evaluation includes the following steps:

[0070] Randomly generate float type test data that meets the model input requirements;

[0071] Use the TensorRT tool to load the optimized model, perform multiple inferences on the generated test data, and record the time consumed for each inference.

[0072] The total time consumption Tsum is calculated based on the time consumption of each inference, and the average inference time consumption Tavg is obtained as the model performance data of the current iteration. The expression is:

[0073] Tavg = Tsum / x,

[0074] Wherein, x is the number of inferences of the test data. In this embodiment, the number of inferences is set to 1000 times, and the average inference time Tavg = Tsum / 1000;

[0075] Use Tavg to compare with the preset performance threshold Tth, which uses the inference performance value of the full FP16 model as a benchmark;

[0076] If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, then re-adjust the FP32 layer configuration and perform the next iteration;

[0077] Record the comparison results of Tavg and Tth for each iteration.

[0078] 9. Re-adjust the FP32 layer configuration: When the accuracy or performance evaluation result of the hybrid model on the deployment platform does not meet the requirements, randomly sample the L_fp32 set. For example, extract 5% of the network layers and put them into L_fp16, then continue to convert the model to the deployment platform and perform the next iteration.

[0079] 10. Output the result model: After the above heuristic hybrid precision configuration iteration process, finally, a target platform model that meets the requirements of both model accuracy and inference performance can be obtained for output, thus completing the accuracy tuning process of the transformer model in low-precision deployment and obtaining better model inference performance at the same time.

[0080] Through the above content, the accuracy tuning method of the present invention is disclosed. Compared with the prior art, the present invention has the following advantages:

[0081] 1. The present invention is a hybrid precision configuration method based on heuristic iterative search, which is used to optimize the accuracy and inference performance of the transformer model, and can maintain an inference performance close to that of the full FP16 model during accuracy tuning when deployed at FP16 precision;

[0082] 2. The present invention adopts a heuristic strategy of selecting FP32 network layers according to layer types to maintain the accuracy of the model;

[0083] 3. The present invention combines a model accuracy and performance evaluation mechanism of cos similarity index and a preset similarity threshold to ensure the result of accuracy tuning;

[0084] 4. The accuracy optimization method of the present invention can run on different NVIDIA chip platforms and has good compatibility.

[0085] In another embodiment of the present invention, an accuracy tuning system based on the transformer model is disclosed. This system is used to execute the above method and includes a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model accuracy evaluation module, a model inference performance evaluation module, and an output module;

[0086] The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model;

[0087] The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layer that maintains high precision and the network layer that is reduced to low precision in the network layer of the ONNX source model, where the high precision is FP32 and the low precision is FP16;

[0088] The model conversion module is used to convert the ONNX source model into a format supported by the deployment platform according to the configuration of the network layer accuracy of the full-precision model to obtain an optimized hybrid model;

[0089] The hybrid model reasoning module is used to load the optimized hybrid model and obtain the reasoning result through data input, and use the reasoning result as the predicted value of the hybrid model;

[0090] The model accuracy evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the accuracy evaluation result, and compares the accuracy evaluation result with a preset similarity threshold, and judges whether the accuracy of the hybrid model meets the target requirements through the comparison result. For the hybrid model that meets the target requirements, the model reasoning performance evaluation module is entered for evaluation, and for the hybrid model that does not meet the target requirements, the full-precision model configuration module is returned to reconfigure and the next iteration is performed;

[0091] The model reasoning performance evaluation module calculates the reasoning time of the hybrid model, and compares the reasoning time with a preset performance threshold, and determines whether the performance of the hybrid model meets the requirements through the comparison result. For the hybrid model that meets the requirements, the module enters the output module for output, and for the hybrid model that does not meet the requirements, the module returns to the full-precision model configuration module for reconfiguration and performs the next iteration;

[0092] The output module is used to output a model that meets the target requirements and the reasoning performance requirements, thereby realizing the tuning of the transformer model.

[0093] In another embodiment of the present invention, an electronic device is also included. The electronic device may include a processor and a memory storing computer program instructions.

[0094] Specifically, in this embodiment, the processor may include a central processing unit (CPU), or a specific integrated circuit, or may be configured as one or more integrated circuits of this embodiment; the memory may include a large-capacity memory for data or instructions, including but not limited to a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these; where appropriate, the memory may include a removable or non-removable (or fixed) medium; in a specific embodiment, the memory is a non-volatile solid-state memory. In a specific embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0095] The processor implements the precision tuning method disclosed above in the present invention by reading and executing computer program instructions stored in the memory.

[0096] It should also be noted that the electronic device of this embodiment may also include a communication interface and a communication bus. Among them, the processor, memory, and communication interface are connected through the communication bus and complete the communication between them. The communication interface is mainly used to realize the communication between the various units, modules, devices or equipment in the embodiment of the present invention.

[0097] The communication bus mentioned above includes hardware, software or a combination of hardware and software, coupling the components of the online data flow device to each other. Where appropriate, the communication bus may include one or more buses.

[0098] In addition, in combination with the power grid electricity theft detection method in the above embodiment, the embodiment of the present invention can be implemented by providing a computer storage medium, on which computer program instructions are stored; the computer program instructions are executed by a processor to implement the above accuracy tuning method.

[0099] It should be clear that the present invention is not limited to the methods, systems, and devices disclosed above, but also includes various changes, modifications, and additions made by those skilled in the art based on the ideas of the present invention, or changes in the order of steps.

[0100] When the present invention is implemented in hardware, it can be an electronic circuit, a dedicated integrated circuit, appropriate firmware, plug-in, function card, etc.; when implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks, and the programs or code segments can be stored in a machine-readable medium, or uploaded on a transmission medium or communication link through a data signal carried in a carrier wave. "Machine-readable medium" may include any medium capable of storing or transmitting information, such as: electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, optical disks, hard disks, optical fiber media, radio frequency links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0101] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A precision tuning method based on a transformer model, characterized in that: This method runs on the NVIDIA chip platform and is optimized using Python scripts in combination with TensorRT tools. The specific steps are as follows: Full-precision model inference to obtain the true value: Use the ONNXRuntime library to load the pre-trained ONNX source model in Python, input the unlabeled sample data into the model, obtain the inference result of the model through inference, and use the inference result as the true value of the full-precision model, denoted as GT; Initialize the FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize the FP32 layer configuration to determine the network layers that maintain high precision and the network layers that are reduced to low precision in the network layer; Convert the model to the deployment platform: According to the network layer precision configuration, use the trtexec tool of TensorRT to convert the ONNX source model to the format supported by the deployment platform to obtain the optimized hybrid model; Hybrid model inference to obtain the predicted value: Use the above unlabeled sample data as the input, and use the TensorRT tool to perform inference on the optimized hybrid model to obtain the predicted value of the hybrid model for each input, denoted as Pred; Model accuracy evaluation: Compare the output results of the true value GT and the predicted value Pred for the same input one by one, calculate the accuracy evaluation result S, and compare S with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and the next model inference performance evaluation is performed. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and the FP32 layer configuration is adjusted; Model inference performance evaluation: Use the TensorRT tool to perform an inference time-consuming test on the optimized hybrid model, calculate the average inference time Tavg, and compare Tavg with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements. Otherwise, it does not meet the requirements, and the FP32 layer configuration is adjusted; Adjust the FP32 layer configuration: Randomly sample and select the network layers with high precision, configure the obtained high-precision network layers as low-precision network layers, convert the adjusted hybrid model to the deployment platform again, and perform the next iteration; Output the result model: After the above iteration process, when both the model accuracy and the inference performance meet the requirements, output the target model to complete the accuracy tuning process of the transformer model in low-precision deployment.

2. According to claim 1, a precision tuning method based on a transformer model is characterized in that: In the initialization of the FP32 layer configuration, it includes the network layers of the set L_fp32 and the network layers of the set L_fp16; Adopt a heuristic strategy to extract the subset L_fp32 from the set L, and denote the remaining all network layer sets as L_fp16, L = L_fp32 + L_fp16.

3. The precision tuning method based on the transformer model according to claim 2, characterized in that: The heuristic strategy is to traverse each network layer in the set L, analyze and select according to the type of the network layer. All network layers involved in numerical calculation and having an impact on the final accuracy, including convolutional layers and fully connected layers, are denoted as FP32, and all network layers not involved in numerical calculation or having a small impact on the final accuracy, including transpose layers and reshaping layers, are denoted as FP16; Obtain the network layers in the network layer that maintain high precision and the network layers that are reduced to low precision.

4. The precision tuning method based on the transformer model according to claim 2, characterized in that: During the process of converting the model to the deployment platform, set the network layers included in the set L_fp32 to FP32 precision and the network layers included in the set L_fp16 to FP16 precision.

5. The precision tuning method based on the transformer model according to claim 1, characterized in that: In the above model precision evaluation, the precision evaluation result S is carried out using the cos similarity index between tensor data, and its value range is 0 to 1. 0 indicates complete similarity, and 1 indicates complete dissimilarity.

6. The precision tuning method based on the transformer model according to claim 1, characterized in that: The above model inference performance evaluation includes the following steps: Randomly generate float-type test data that meets the model input requirements; Use the TensorRT tool to load the optimized model, perform multiple inferences on the generated test data, and record the time consumption of each inference; Calculate the total time consumption Tsum based on the time consumption of each inference, and obtain the average inference time Tavg as the model performance data for the current iteration. The expression is: Tavg = Tsum / x, where x is the number of inferences of the test data; Compare Tavg with the preset performance threshold Tth. The preset performance threshold Tth uses the inference performance value of the full FP16 model as the benchmark; If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, re-adjust the FP32 layer configuration, and perform the next iteration; Record the comparison results of Tavg and Tth for each iteration.

7. A precision tuning system based on a transformer model according to any one of claims 1 to 6, characterized in that: The system includes a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model precision evaluation module, a model inference performance evaluation module, and an output module; The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model; The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layers in the ONNX source model that maintain high precision and the network layers that are reduced to low precision. The high precision is FP32, and the low precision is FP16; The model conversion module is used to convert the ONNX source model to the format supported by the deployment platform according to the configuration of the network layer precision of the full-precision model, and obtain the optimized hybrid model; The hybrid model inference module is used to load the optimized hybrid model and obtain the inference result through data input, and use the inference result as the predicted value of the hybrid model; The model accuracy evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the accuracy evaluation result, and compares the accuracy evaluation result with a preset similarity threshold, and judges whether the accuracy of the hybrid model meets the target requirements through the comparison result, and enters the model reasoning performance evaluation module for evaluation for the hybrid model that meets the target requirements, and returns to the full-precision model configuration module for reconfiguration and performs the next iteration for the hybrid model that does not meet the target requirements; the model reasoning performance evaluation module calculates the reasoning time of the hybrid model, and compares the reasoning time with a preset performance threshold, and judges whether the performance of the hybrid model meets the requirements through the comparison result, and enters the output module for output for the hybrid model that meets the target requirements, and returns to the full-precision model configuration module for reconfiguration and performs the next iteration for the hybrid model that does not meet the target requirements; The output module is used to output a model that meets the target requirements and the reasoning performance requirements, thereby realizing the tuning of the transformer model.

8. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein the memory is used to store a computer program; The processor is configured to execute the method according to any one of claims 1 to 6 by running the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program implements the method described in any one of claims 1 to 6 when executed by a processor.

Images