A precision tuning method, system, device and storage medium based on a transformer model

Abstract

The application discloses a precision tuning method, system and device based on a transformer model, and a storage medium. The precision of the transformer model is quickly recovered through heuristic iterative search of a mixed precision strategy, an FP32+FP16 mixed model with precision basically aligned with that of a full-precision model is obtained, and meanwhile, the inference performance can be ensured to be close to that of an original FP16 model. The method is a mixed precision configuration method based on heuristic iterative search, is used for optimizing the precision and inference performance of the transformer model, and can keep the inference performance close to that of the full FP16 model in the FP16 precision deployment while tuning the precision. The precision of the model is maintained through the heuristic strategy of selecting an FP32 network layer according to layer types. The model precision and performance evaluation mechanism combining a cos similarity index and a preset similarity threshold value is used to ensure the result of the precision tuning. Meanwhile, the method can run on different NVIDIA chip platforms and has good compatibility.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology and relates to deep learning GPU inference optimization technology, specifically to a precision tuning method, system, device and storage medium based on the transformer model. Background Technology

[0002] Since its introduction in 2017, the Transformer model has demonstrated widespread application and significant effects across various fields. However, its engineering deployment, especially when using FP16 precision, still faces numerous challenges. Due to its more complex network structure compared to CNN models, certain operators are prone to anomalies at FP16 precision, ultimately leading to a significant drop in overall model accuracy. Furthermore, because the Transformer model consists of a large number of operators, and the locations of these drops are random, using a step-by-step approach to fix accuracy issues is inefficient and cannot guarantee high-quality precision tuning results. These problems limit the effectiveness of the Transformer model at FP16 precision. Summary of the Invention

[0003] To address the aforementioned problems, the main objective of this invention is to design a method, system, device, and storage medium for precision tuning based on the Transformer model, thereby resolving the technical issues of severe point loss and low precision repair efficiency in the Transformer model.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A method for accuracy tuning based on a transformer model, running on an NVIDIA chip platform, uses Python scripts in conjunction with the TensorRT tool for optimization, and includes the following steps:

[0006] Obtaining the true value through full-precision model inference: The ONNXRuntime library is used to load the pre-trained ONNX source model in Python, input the unlabeled sample data into the model, obtain the inference result of the model through inference, and use the inference result as the true value of the full-precision model, denoted as GT;

[0007] Initialize FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize the FP32 layer configuration to determine the network layers that maintain high precision and the network layers that are reduced to low precision.

[0008] Model conversion to deployment platform: Based on the network layer accuracy configuration, the ONNX source model is converted to a format supported by the deployment platform using TensorRT's trtexec tool to obtain an optimized hybrid model;

[0009] Obtaining prediction values through hybrid model inference: Using the above unlabeled sample data as input, the optimized hybrid model is inferred using the TensorRT tool to obtain the prediction values of the hybrid model for each input, denoted as Pred;

[0010] Model accuracy evaluation: The output results of the true value GT and the prediction value Pred corresponding to the same input are compared one by one, and the accuracy evaluation result S is calculated. S is compared with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and the next model inference performance evaluation is carried out. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and the FP32 layer configuration is adjusted;

[0011] Model inference performance evaluation: The TensorRT tool is used to test the inference time consumption of the optimized hybrid model, and the average inference time Tavg is calculated. Tavg is compared with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; otherwise, it does not meet the requirements, and the FP32 layer configuration is adjusted;

[0012] Adjusting the FP32 layer configuration: Randomly sample and select high-precision network layers, and configure the obtained high-precision network layers as low-precision network layers. The adjusted hybrid model is converted to the deployment platform again, and the next iteration is carried out;

[0013] Output result model: After the above iterative process, when both the model accuracy and inference performance meet the requirements, the target model is output, and the accuracy tuning process of the transformer model in low-precision deployment is completed.

[0014] As a further description of the present invention, in the initialization of the FP32 layer configuration, it includes the network layers in the set L_fp32 and the network layers in the set L_fp16;

[0015] A heuristic strategy is used to extract a subset L_fp32 from the set L, and the remaining set of all network layers is denoted as L_fp16, L = L_fp32 + L_fp16.

[0016] As a further description of the present invention, the heuristic strategy is to traverse each network layer in the set L, and analyze and select according to the type of the network layer. All network layers involved in numerical calculations and having an impact on the final accuracy, including convolutional layers and fully connected layers, are denoted as FP32, and all network layers not involved in numerical calculations or having a small impact on the final accuracy, including transpose layers and reshaping layers, are denoted as FP16;

[0017] The network layers with high precision and the network layers reduced to low precision in the network layer are obtained.

[0018] As a further description of the present invention, during the process of converting the model to the deployment platform, the network layers included in the set L_fp32 are set to FP32 precision, and the network layers included in the set L_fp16 are set to FP16 precision.

[0019] As a further description of the present invention, in the above model precision evaluation, the precision evaluation result S is carried out by using the cosine similarity index between tensor data, and its value range is 0 to 1, where 0 indicates complete similarity and 1 indicates complete dissimilarity.

[0020] As a further description of the present invention, the above model inference performance evaluation includes the following steps:

[0021] Randomly generate float-type test data that meets the model input requirements;

[0022] Use the TensorRT tool to load the optimized model, perform multiple inferences on the generated test data, and record the time consumption of each inference;

[0023] Calculate the total time consumption Tsum based on the time consumption of each inference, and obtain the average inference time Tavg as the model performance data of the current iteration. The expression is:

[0024] Tavg = Tsum / x,

[0025] where x is the number of inferences of the test data;

[0026] Compare Tavg with the preset performance threshold Tth. The preset performance threshold Tth uses the inference performance value of the full FP16 model as the benchmark;

[0027] If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, re-adjust the FP32 layer configuration, and perform the next iteration;

[0028] Record the comparison results of Tavg and Tth for each iteration.

[0029] An accuracy tuning system based on a transformer model, which is used to execute the above accuracy tuning method, including a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model accuracy evaluation module, a model inference performance evaluation module, and an output module;

[0030] The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model;

[0031] The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layers in the ONNX source model that maintain high precision, and the network layers that are reduced to low precision, wherein the high precision is FP32 and the low precision is FP16.

[0032] The model conversion module is used to convert the ONNX source model to a format supported by the deployment platform according to the configuration of the network layer precision of the full-precision model, so as to obtain an optimized hybrid model.

[0033] The hybrid model inference module is used to load the optimized hybrid model and obtain inference results through data input, and use the inference results as the predicted values ​​of the hybrid model.

[0034] The model accuracy evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the accuracy evaluation result, and compares the accuracy evaluation result with a preset similarity threshold. Based on the comparison result, it determines whether the accuracy of the hybrid model meets the target requirements. For hybrid models that meet the requirements, they are entered into the model inference performance evaluation module for evaluation. For hybrid models that do not meet the requirements, they are returned to the full-precision model configuration module for reconfiguration and the next iteration is performed.

[0035] The model inference performance evaluation module calculates the inference time of the hybrid model and compares the inference time with a preset performance threshold. Based on the comparison result, it determines whether the performance of the hybrid model meets the requirements. For hybrid models that meet the requirements, they are output to the output module. For hybrid models that do not meet the requirements, they are returned to the full-precision model configuration module for reconfiguration and the next iteration is performed.

[0036] The output module is used to output a model that meets the target requirements and inference performance requirements, thereby achieving the optimization of the transformer model.

[0037] An electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus, and the memory is used to store computer programs;

[0038] The processor is configured to execute the aforementioned precision tuning method by running the computer program stored in the memory.

[0039] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the above-described precision tuning method.

[0040] Compared with the prior art, the technical advantages of the present invention are as follows:

[0041] This invention provides a method, system, device, and storage medium for precision tuning based on the Transformer model. It rapidly restores the precision of the Transformer model by heuristically iteratively searching a hybrid precision strategy, obtaining an FP32+FP16 hybrid model with precision essentially aligned with the full-precision model, while also ensuring inference performance close to the original FP16 model. This method is a heuristic iterative search-based hybrid precision configuration approach used to optimize the precision and inference performance of the Transformer model. It maintains inference performance close to the full FP16 model in FP16 precision deployment while simultaneously tuning precision. It maintains model precision through a heuristic strategy of selecting FP32 network layers by layer type. A model precision and performance evaluation mechanism combining cosine similarity metrics and preset similarity thresholds ensures the accuracy tuning results. Furthermore, it can run on different NVIDIA chip platforms, exhibiting good compatibility. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Detailed Implementation

[0043] The present invention will now be described in detail with reference to the accompanying drawings:

[0044] In one embodiment of the present invention, a method for accuracy tuning based on a transformer model is disclosed, with reference to... Figure 1 As shown, a heuristic iterative search strategy for mixed precision is used to quickly restore the precision of the transformer model, resulting in a hybrid FP32+FP16 model whose precision is basically aligned with that of the full-precision model, while also ensuring that the inference performance is close to that of the original FP16 model.

[0045] Specifically, in this embodiment, the above method runs on an NVIDIA chip platform and is optimized using Python scripts in conjunction with the TensorRT tool. The specific steps include:

[0046] Obtaining the true value through full-precision model inference: The ONNXRuntime library is used to load the pre-trained ONNX source model in Python, input the unlabeled sample data into the model, obtain the inference result of the model through inference, and use the inference result as the true value of the full-precision model, denoted as GT;

[0047] Initialize FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize the FP32 layer configuration to determine the network layers that maintain high precision and the network layers that are reduced to low precision.

[0048] Model conversion to the deployment platform: According to the configuration of the network layer precision, use the trtexec tool of TensorRT to convert the ONNX source model to the format supported by the deployment platform, and obtain an optimized hybrid model;

[0049] Inference of the hybrid model to obtain prediction values: Use the TensorRT tool to perform inference on the optimized hybrid model with the above unlabeled sample data as the input, and obtain the prediction values of the hybrid model for each input, denoted as Pred;

[0050] Model accuracy evaluation: Compare the output results of the true value GT and the prediction value Pred corresponding to the same input one by one, calculate the accuracy evaluation result S, and compare S with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and proceed to the next model inference performance evaluation. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and adjust the FP32 layer configuration;

[0051] Model inference performance evaluation: Use the TensorRT tool to perform an inference time-consuming test on the optimized hybrid model, calculate the average inference time Tavg, and compare Tavg with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements. Otherwise, it does not meet the requirements, and adjust the FP32 layer configuration;

[0052] Adjust the FP32 layer configuration: Randomly sample and select high-precision network layers, configure the obtained high-precision network layers as low-precision network layers, convert the adjusted hybrid model to the deployment platform again, and perform the next iteration;

[0053] Output the result model: After the above iterative process, when both the model accuracy and inference performance meet the requirements, output the target model, and complete the accuracy tuning process of the transformer model in low-precision deployment.

[0054] In this embodiment, the process of the above accuracy tuning method specifically includes the following contents:

[0055] 1. Application scenario description: In this embodiment, the application scenarios where the accuracy tuning method is applied include, but are not limited to, models such as image classification, object detection, semantic segmentation, etc., and are also not limited to the algorithm implementation scenarios in the autonomous driving industry. Any other fields or applications involving accuracy tuning of the transformer model can use the method of this embodiment to achieve accuracy tuning. The method of this embodiment is a general accuracy tuning strategy for the transformer model.

[0056] 2. Operating Platform Description: The precision tuning method in this embodiment runs on NVIDIA chip platforms such as Orin and Xavier. The entire optimization process can be achieved using Python scripts and the TensorRT tool.

[0057] 3. Obtaining the true value through full-precision model inference: The ONNX source model is inferred using the ONNXRuntime library in Python to obtain the model inference result, which is used as the true value of the full-precision model, denoted as GT. In this embodiment, a certain amount of unlabeled sample data is used as the input of the model.

[0058] 4. Initialize FP32 layer configuration: For all network layer sets L in the ONNX source model, extract a set L_fp32 using a heuristic strategy. The remaining set of all network layers is denoted as L_fp16, and L = L_fp32 + L_fp16.

[0059] The heuristic strategy described above involves traversing each network layer in the set L and analyzing and selecting layers based on their type. All network layers that involve numerical computation and affect the final accuracy, including but not limited to convolutional layers and fully connected layers, are denoted as FP32. All network layers that do not involve numerical computation or have a small impact on the final accuracy, including but not limited to transposed layers and remodeled layers, are denoted as FP16. This process yields network layers that maintain high accuracy and those that are reduced to low accuracy.

[0060] 5. Model conversion to deployment platform: Based on the L_fp32 and L_fp16 information obtained above, the ONNX source model is converted to a format supported by the deployment platform using TensorRT's trtexec tool to obtain an optimized hybrid model. During the model conversion process, the network layers contained in the L_fp32 set are set to FP32 precision, and the network layers contained in the L_fp16 set are set to FP16 precision.

[0061] It should be noted that the above process of converting ONNX source models on the NVIDIA chip platform uses the trtexec model conversion tool of the tensorrt package under the NVIDIA chip platform; this part of the conversion process is existing technology.

[0062] 6. Obtaining Predicted Values ​​from Hybrid Model Inference: Continue to use the TensorRT tool to infer the optimized hybrid model. The above unlabeled sample data is required as input. The inference results are collected and recorded as the hybrid model predicted values ​​for this iteration, denoted as Pred.

[0063] 7. Model accuracy evaluation: The output results of the same input samples in the true value GT and the predicted value Pred are compared one by one to calculate the accuracy evaluation result S. In this embodiment, the accuracy evaluation result S is based on the cosine similarity index between tensor data. The value of S ranges from 0 to 1, where 0 indicates complete similarity and 1 indicates complete dissimilarity.

[0064] In this embodiment, the formula for calculating the cosine similarity between the above tensor data is:

[0065]

[0066] Here, A and B are two tensors, and the cosine similarity is equal to the dot product of A and B divided by the magnitude of the vector.

[0067] The accuracy assessment result S is compared with the preset similarity threshold Sth. In this embodiment, Sth is generally set to 0.98, or it can be set according to actual needs.

[0068] If S > Sth, the model's prediction accuracy meets the target requirement, and the next step of model inference performance evaluation is performed. If S ≤ Sth, the model's prediction accuracy does not meet the target requirement, the FP32 layer configuration is adjusted, and the next iteration is performed.

[0069] 8. Model Inference Performance Evaluation: The TensorRT tool is used to test the inference time of the hybrid model in the deployment platform format. Specifically, the model inference performance evaluation includes the following steps:

[0070] Randomly generate test data of type float that meets the model input requirements;

[0071] The optimized model was loaded using the TensorRT tool, and multiple inferences were performed on the generated test data, with the time taken for each inference recorded.

[0072] The total inference time Tsum is calculated based on the time spent in each inference iteration, and the average inference time Tavg is taken as the model performance data for the current iteration. The expression is as follows:

[0073] Tavg = Tsum / x,

[0074] Where x is the number of inferences for the test data. In this embodiment, the number of inferences is set to 1000, so the average inference time Tavg = Tsum / 1000.

[0075] The Tavg is compared with a preset performance threshold Tth, which uses the inference performance value of the full FP16 model as a benchmark.

[0076] If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, then re-adjust the FP32 layer configuration and perform the next iteration;

[0077] Record the comparison results of Tavg and Tth for each iteration.

[0078] 9. Re-adjust the FP32 layer configuration: When the accuracy or performance evaluation result of the hybrid model on the deployment platform does not meet the requirements, randomly sample the L_fp32 set, for example, extract 5% of the network layers and put them into L_fp16, then continue to convert the model to the deployment platform for the next iteration.

[0079] 10. Output the result model: After the above heuristic hybrid precision configuration iteration process, finally, a target platform model that meets the requirements of both model accuracy and inference performance can be obtained for output, thus completing the accuracy tuning process of the transformer model in low-precision deployment and obtaining better model inference performance at the same time.

[0080] Through the above content, the accuracy tuning method of the present invention is disclosed. Compared with the prior art, the present invention has the following advantages:

[0081] 1. The present invention is a hybrid precision configuration method based on heuristic iterative search, which is used to optimize the accuracy and inference performance of the transformer model, and can maintain an inference performance close to that of the full FP16 model in FP16 precision deployment while tuning the accuracy.

[0082] 2. The present invention adopts a heuristic strategy of selecting FP32 network layers according to the layer type to maintain the accuracy of the model.

[0083] 3. The present invention combines a model accuracy and performance evaluation mechanism of cosine similarity index and preset similarity threshold to ensure the result of accuracy tuning.

[0084] 4. The accuracy optimization method of the present invention can run on different NVIDIA chip platforms and has good compatibility.

[0085] In another embodiment of the present invention, an accuracy tuning system based on the transformer model is disclosed. This system is used to execute the above method and includes a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model accuracy evaluation module, a model inference performance evaluation module, and an output module;

[0086] The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model;

[0087] The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layers in the ONNX source model that maintain high precision, and the network layers that are reduced to low precision, wherein the high precision is FP32 and the low precision is FP16.

[0088] The model conversion module is used to convert the ONNX source model to a format supported by the deployment platform according to the configuration of the network layer precision of the full-precision model, so as to obtain an optimized hybrid model.

[0089] The hybrid model inference module is used to load the optimized hybrid model and obtain inference results through data input, and use the inference results as the predicted values ​​of the hybrid model.

[0090] The model accuracy evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the accuracy evaluation result, and compares the accuracy evaluation result with a preset similarity threshold. Based on the comparison result, it determines whether the accuracy of the hybrid model meets the target requirements. For hybrid models that meet the requirements, they are entered into the model inference performance evaluation module for evaluation. For hybrid models that do not meet the requirements, they are returned to the full-precision model configuration module for reconfiguration and the next iteration is performed.

[0091] The model inference performance evaluation module calculates the inference time of the hybrid model and compares the inference time with a preset performance threshold. Based on the comparison result, it determines whether the performance of the hybrid model meets the requirements. For hybrid models that meet the requirements, they are output to the output module. For hybrid models that do not meet the requirements, they are returned to the full-precision model configuration module for reconfiguration and the next iteration is performed.

[0092] The output module is used to output a model that meets the target requirements and inference performance requirements, thereby achieving the optimization of the transformer model.

[0093] In another embodiment of the invention, an electronic device is also included, which may include a processor and a memory storing instructions from a computer program.

[0094] Specifically, in this embodiment, the processor may include a central processing unit (CPU), a specific integrated circuit, or one or more integrated circuits that can be configured in this embodiment; the memory may include a mass storage device for data or instructions, including but not limited to a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these; where appropriate, the memory may include removable or non-removable (or fixed) media; in a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0095] The processor described above implements the precision tuning method disclosed in this invention by reading and executing computer program instructions stored in the memory.

[0096] It should also be noted that the electronic device in this embodiment may further include a communication interface and a communication bus. The processor, memory, and communication interface are connected via the communication bus to complete communication between them. The communication interface is mainly used to realize communication between the various units, modules, devices, or equipment in this embodiment of the invention.

[0097] The aforementioned communication bus comprises hardware, software, or a combination of both, coupling components of online data traffic devices together. Where appropriate, the communication bus may include one or more buses.

[0098] In addition, in conjunction with the power grid theft detection method in the above embodiments, the embodiments of the present invention can be implemented by providing a computer storage medium, on which computer program instructions are stored; the computer program instructions are executed by a processor to perform the above-mentioned precision tuning method.

[0099] It should be clarified that the present invention is not limited to the methods, systems, and devices disclosed above, but also includes various changes, modifications, and additions made by those skilled in the art based on the ideas of the present invention, or changes in the order of steps.

[0100] When implemented in hardware, this invention can be electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. These programs or code segments can be stored on a machine-readable medium or transmitted via a data signal carried on a carrier wave through a transmission medium or communication link. "Machine-readable medium" can include any medium capable of storing or transmitting information, such as electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, optical disks, hard disks, fiber optic media, radio frequency links, etc. The code segments can be downloaded via computer networks such as the Internet or intranets.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A precision tuning method based on the transformer model, characterized in that, This method runs on the NVIDIA chip platform and is optimized using Python scripts in combination with TensorRT tools. The specific steps are as follows: Full-precision model inference to obtain the true value: Use the ONNXRuntime library to load the trained ONNX source model in Python, input the unlabeled sample data into the model, obtain the inference result of the model through inference, and use the inference result as the true value of the full-precision model, denoted as GT; Initialize the FP32 layer configuration: For all network layer sets L in the ONNX source model, initialize the FP32 layer configuration to determine the network layers that maintain high precision and the network layers that are reduced to low precision in the network layer; Convert the model to the deployment platform: According to the configuration of the network layer precision, use the trtexec tool of TensorRT to convert the ONNX source model to the format supported by the deployment platform to obtain the optimized hybrid model; Hybrid model inference to obtain the predicted value: Use the TensorRT tool to perform inference on the optimized hybrid model with the above unlabeled sample data as the input, and obtain the predicted value of the hybrid model for each input, denoted as Pred; Model accuracy evaluation: Compare the output results of the true value GT and the predicted value Pred for the same input one by one, calculate the accuracy evaluation result S, and compare S with the preset similarity threshold Sth. If S > Sth, the prediction accuracy of the model meets the target requirements, and proceed to the next model inference performance evaluation. If S ≤ Sth, the prediction accuracy of the model does not meet the target requirements, and adjust the FP32 layer configuration; Model inference performance evaluation: Use the TensorRT tool to perform an inference time-consuming test on the optimized hybrid model, calculate the average inference time Tavg, and compare Tavg with the preset performance threshold Tth. If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements. Otherwise, it does not meet the requirements, and adjust the FP32 layer configuration; Adjust the FP32 layer configuration: Randomly sample and select the network layers with high precision, configure the obtained high-precision network layers as low-precision network layers, convert the adjusted hybrid model to the deployment platform again, and perform the next iteration; Output the result model: After the above iterative process, when both the model accuracy and inference performance meet the requirements, output the target model to complete the accuracy tuning process of the transformer model in low-precision deployment.

2. The accuracy tuning method based on the transformer model according to claim 1, characterized in that: In the initialization of the FP32 layer configuration, it includes the network layers in the set L_fp32 and the network layers in the set L_fp16; Adopt a heuristic strategy to extract the subset L_fp32 from the set L, and denote the remaining all network layer sets as L_fp16, L = L_fp32 + L_fp16.

3. The accuracy tuning method based on the transformer model according to claim 2, characterized in that: The heuristic strategy is to traverse each network layer in the set L, analyze and select according to the type of the network layer. All network layers involved in numerical calculation and affecting the final accuracy, including convolutional layers and fully connected layers, are denoted as FP32, and all network layers not involved in numerical calculation or having a small impact on the final accuracy, including transpose layers and reshaping layers, are denoted as FP16; Obtain the network layers in the network layer that maintain high precision and the network layers that are reduced to low precision.

4. The accuracy tuning method based on the transformer model according to claim 2, characterized in that: During the process of converting the model to the deployment platform, set the network layers included in the set L_fp32 to FP32 precision and the network layers included in the set L_fp16 to FP16 precision.

5. The accuracy tuning method based on the transformer model according to claim 1, characterized in that: In the above model precision evaluation, the precision evaluation result S is carried out using the cosine similarity index between tensor data, and its value range is 0 to 1, where 0 means completely similar and 1 means completely dissimilar.

6. The accuracy tuning method based on the transformer model according to claim 1, characterized in that: The above model inference performance evaluation includes the following steps: Randomly generate float-type test data that meets the model input requirements; Use the TensorRT tool to load the optimized model, perform multiple inferences on the generated test data, and record the time consumption of each inference; Calculate the total time consumption Tsum based on the time consumption of each inference, and obtain the average inference time Tavg as the model performance data for the current iteration. The expression is: Tavg = Tsum / x, where x is the number of inferences of the test data; Compare Tavg with the preset performance threshold Tth. The preset performance threshold Tth uses the inference performance value of the full FP16 model as the benchmark; If (Tavg - Tth) < Tth * 0.95, the inference performance meets the requirements; if (Tavg - Tth) ≥ Tth * 0.95, the inference performance does not meet the requirements, re-adjust the FP32 layer configuration, and perform the next iteration; Record the comparison results of Tavg and Tth for each iteration.

7. A precision tuning system based on a transformer model, used to implement the method described in any one of claims 1-6, characterized in that: The system includes a full-precision model inference module, a full-precision model configuration module, a model conversion module, a hybrid model inference module, a model precision evaluation module, a model inference performance evaluation module, and an output module; The full-precision model inference module is used to load the ONNX source model and obtain the inference result through data input, and use the inference result as the true value of the full-precision model; The full-precision model configuration module is used to initialize the precision of the full-precision model, determine the network layers in the ONNX source model that maintain high precision and the network layers that are reduced to low precision. The high precision is FP32 and the low precision is FP16; The model conversion module is used to convert the ONNX source model to the format supported by the deployment platform according to the configuration of the network layer precision of the full-precision model, and obtain the optimized hybrid model; The hybrid model inference module is used to load the optimized hybrid model and obtain the inference result through data input, and use the inference result as the predicted value of the hybrid model; The model precision evaluation module compares the true value of the full-precision model with the predicted value of the hybrid model, calculates the precision evaluation result, and compares the precision evaluation result with the preset similarity threshold. Judge whether the precision of the hybrid model meets the target requirements through the comparison result. For the hybrid model that meets the requirements, enter the model inference performance evaluation module for evaluation. For the hybrid model that does not meet the requirements, return to the full-precision model configuration module for re-configuration and perform the next iteration; The model inference performance evaluation module calculates the inference time of the hybrid model and compares the inference time with a preset performance threshold. Based on the comparison result, it determines whether the performance of the hybrid model meets the requirements. For hybrid models that meet the requirements, they are output to the output module. For hybrid models that do not meet the requirements, they are returned to the full-precision model configuration module for reconfiguration and the next iteration is performed. The output module is used to output a model that meets the target requirements and inference performance requirements, thereby achieving the optimization of the transformer model.

8. An electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, the communication interface, and the memory communicate with each other via the communication bus, wherein the memory is used to store computer programs; The processor is configured to perform the method of any one of claims 1-6 by running the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of claims 1-6.

Images