Transform model-based GPU reasoning performance optimization method, system and equipment and storage medium
By optimizing the key operators in the Transformer model and using iterative search and pattern matching technology, the inference performance of the Transformer model on the GPU is improved, the problem of low inference performance is solved, and faster response speed and lower resource consumption are achieved.
Patent Information
- Application Number
- CN202411963194.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-25
Smart Images
Figure CN119886347A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology and relates to deep learning GPU inference optimization technology, and specifically to a GPU inference performance optimization method, system, device and storage medium based on a transformer model. Background Art
[0002] With the continuous advancement of deep learning technology, the Transformer model has gained wide attention in the fields of natural language processing (NLP) and computer vision (CV) due to its excellent performance. The core of the Transformer model is the self-attention mechanism, which can efficiently capture contextual information in sequence data. However, the inference performance of Transformer on GPU has always been a bottleneck for its application, especially in fields with limited computing resources such as intelligent driving. In these scenarios, the speed of inference is directly related to the response time of the system and user experience. Therefore, it is particularly important to improve the inference performance of Transformer on GPU. By optimizing the inference process on GPU, the response speed of the model can be accelerated and the latency can be reduced, thereby meeting the needs of real-time applications and further broadening the application scope of the Transformer model. Summary of the invention
[0003] In view of the above problems, the main purpose of the present invention is to design a GPU reasoning performance optimization method, system, device and storage medium based on the transformer model, which adopts iterative search, pattern matching, operator replacement and fusion to improve the reasoning performance of Transformer on GPU.
[0004] In order to achieve the above purpose, the present invention adopts the following technical solutions:
[0005] A GPU inference performance optimization method based on the transformer model. This method runs on the NVIDIA chip platform and uses Python scripts combined with TensorRT and Cuda tools to optimize inference performance. The specific steps include:
[0006] Get the original ONNX model Model_0; use the input data to run inference through the ONNX model and record the initial performance indicator, recorded as P_0;
[0007] Optimize the Attention operator in the original ONNX model Model_0 to obtain the new model Model_1; evaluate the inference performance of Model_1, denoted as P_1;
[0008] Optimize the LayerNorm operator in the model Model_1 to obtain the new model Model_2; evaluate the inference performance of Model_2, denoted as P_2;
[0009] Optimize the GBR operator in model Model_2 to obtain a new model Model_3; evaluate the inference performance of Model_3, denoted as P_3;
[0010] Optimize the GB operator in model Model_3 to obtain a new model Model_4; evaluate the inference performance of Model_4, denoted as P_4;
[0011] Compare the performance of the original model P_0, the performance after optimizing the Attention operator P_1, the performance after optimizing the LayerNorm operator P_2, the performance after optimizing the GBR operator P_3, and the performance after optimizing the GB operator P_4 to find the model version with the best performance, recorded as Model_best, which is the optimal model and the best performance optimization result of the above method.
[0012] As a further description of the present invention, optimizing the Attention operator in the original ONNX model Model_0 includes the following steps:
[0013] Iteratively search the network graph of the original ONNX model Model_0, traverse the network layers in the model Model_0 layer by layer, and perform network layer context pattern matching;
[0014] Among them, the Attention operator consists of 2 Matmul layers + 1 softmax layer;
[0015] After finding all the Attention operators in the model Model_0, replace them with the custom fused AttentionFusionOp operator. AttentionFusionOp is a custom operator obtained by re-integrating the Attention operator structure in the model Model_0.
[0016] After the conversion is completed, the new ONNX model is recorded as Model_1.
[0017] As a further description of the present invention, the LayerNorm operator in the optimization model Model_1 includes the following steps:
[0018] Iterate the network graph of model Model_1, traverse the network layers in model Model_1 layer by layer, and perform network layer context pattern matching;
[0019] Among them, the LayerNorm operator includes 2 Add operators, 2 reduceMean operators, a sub operator, a pow operator, a Sqrt operator, a Div operator and a mul operator;
[0020] After finding all LayerNorm operators in model Model_1, replace them with custom fused LNFusionOp operators. LNFusionOp is a custom operator obtained by reintegrating the LayerNorm operator structure in model Model_1.
[0021] After the replacement is completed, the new ONNX model is recorded as Model_2.
[0022] As a further description of the present invention, the GBR operator in the optimization model Model_2 includes the following steps:
[0023] Iteratively search the network graph of model Model_2, traverse the network layers in model Model_2 layer by layer, and perform network layer context pattern matching;
[0024] Among them, the GBR operator is the fusion of Matmul+Add+Relu operators;
[0025] After finding all GBR operators in model Model_2, replace them with the custom fused GBRFusionOp operator. GBRFusionOp is a custom operator obtained by reintegrating the GBR operator structure gemm+bias+relu in model Model_2.
[0026] After the replacement is completed, the new ONNX model is recorded as Model_3.
[0027] As a further description of the present invention, the GB operator in the optimization model Model_3 includes the following steps:
[0028] Iterate the network graph of model Model_3, traverse the network layers in model Model_3 layer by layer, and perform network layer context pattern matching;
[0029] Among them, the GB operator is the fusion of the two operators Matmul+Add;
[0030] After finding all GB operators in model Model_3, replace them with the custom fused GBFusionOp operator. GBFusionOp is a custom operator obtained by re-integrating the GB operator structure gemm+bias in model Model_3.
[0031] After the replacement is completed, the new ONNX model is recorded as Model_4.
[0032] As a further description of the present invention, the comparison of performance P_0, performance P_1, performance P_2, and performance P_3 includes the following steps:
[0033] Use Model_0, Model_1, Model_2, and Model_3 to perform model inference operations in the same device environment;
[0034] Count the total time consumed for the preset number of inferences and calculate the average time consumed;
[0035] The performance of each operator under optimization is evaluated based on the average time consumption.
[0036] A GPU inference performance optimization system based on a transformer model, which is used to execute the above-mentioned performance optimization method, including an original model acquisition inference module, an Attention operator optimization module, a LayerNorm operator optimization module, a GBR operator optimization module, a GB operator optimization module, and a performance comparison module;
[0037] The original model acquisition reasoning module is used to obtain the ONNX model Model_0, and perform reasoning through input data to obtain the initial performance P_0 of the model Model_0;
[0038] The Attention operator optimization module is used to traverse the model Model_0, find and replace the Attention operator to obtain Model_1, and perform reasoning through input data to obtain the performance P_1 of the model Model_1;
[0039] The LayerNorm operator optimization module is used to traverse the model Model_1, find and replace the LayerNorm operator to obtain Model_2, and perform reasoning through input data to obtain the performance P_2 of Model_2;
[0040] The GBR operator optimization module is used to traverse the model Model_2, find and replace the GBR operator to obtain Model_3, and perform reasoning through input data to obtain the performance P_3 of Model_3;
[0041] The GB operator optimization module is used to traverse the model Model_3, find and replace the GB operator to obtain Model_4, and perform reasoning through input data to obtain the performance P_4 of Model_4;
[0042] The performance comparison module compares the above performance P_1, performance P_2, performance P_3, and performance P_4 to find the model version with the best performance, and obtains the optimal model and the best performance optimization result.
[0043] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the memory is used to store a computer program;
[0044] The processor is used to execute the above-mentioned performance optimization method by running the computer program stored in the memory.
[0045] A computer-readable storage medium stores a computer program, wherein the computer program implements the above-mentioned performance optimization method when executed by a processor.
[0046] Compared with the prior art, the technical effects of the present invention are:
[0047] The present invention provides a GPU reasoning performance optimization method, system, device and storage medium based on a transformer model. The performance optimization method is a general transformer model reasoning performance tuning strategy, which is applicable to a variety of application scenarios and model types. It optimizes the Attention operator, LayerNorm operator, GBR operator, GB operator, etc. through custom operator replacement and fusion, significantly reduces the amount of calculation or memory usage, and improves model performance. The specific optimization steps include iterative search, pattern matching, operator replacement and fusion, which improves the training speed and reasoning speed of the deep learning model and reduces resource consumption and cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic diagram of the performance optimization method of the present invention;
[0049] Figure 2 This is a schematic diagram of the Attention layer of the present invention;
[0050] Figure 3 It is a schematic diagram of the LayerNorm layer of the present invention;
[0051] Figure 4 It is a schematic diagram of the GBR layer of the present invention;
[0052] Figure 5 It is a schematic diagram of the GB layer of the present invention;
[0053] Figure 6 It is a schematic diagram for comparing the performance optimization methods of the present invention. DETAILED DESCRIPTION
[0054] The present invention is described in detail below in conjunction with the accompanying drawings:
[0055] In one embodiment of the present invention, a GPU reasoning performance optimization method based on a transformer model is disclosed, referring to Figure 1-6 As shown in the figure, this method flexibly searches and replaces several key operators in the original model, such as the Attention operator and the LayerNorm operator, and uses custom operators to reintegrate the operator kernels, thereby significantly improving the performance of the transformer model on the GPU.
[0056] Specifically, in this embodiment, the above performance optimization method runs on the NVIDIA chip platform, and uses Python scripts combined with TensorRT and Cuda tools to optimize inference performance, specifically including the following steps:
[0057] Get the original ONNX model Model_0; use the input data to run inference through the ONNX model and record the initial performance indicator, recorded as P_0;
[0058] Optimize the Attention operator in the original ONNX model Model_0 to obtain the new model Model_1; evaluate the inference performance of Model_1, denoted as P_1;
[0059] Optimize the LayerNorm operator in the model Model_1 to obtain the new model Model_2; evaluate the inference performance of Model_2, denoted as P_2;
[0060] Optimize the GBR operator in model Model_2 to obtain a new model Model_3; evaluate the inference performance of Model_3, denoted as P_3;
[0061] Optimize the GB operator in model Model_3 to obtain a new model Model_4; evaluate the inference performance of Model_4, denoted as P_4;
[0062] Compare the performance of the original model P_0, the performance after optimizing the Attention operator P_1, the performance after optimizing the LayerNorm operator P_2, the performance after optimizing the GBR operator P_3, and the performance after optimizing the GB operator P_4 to find the model version with the best performance, recorded as Model_best, which is the optimal model and the best performance optimization result of the above method.
[0063] In this embodiment, the process of the above performance optimization method specifically includes the following contents:
[0064] 1. Description of application scenarios: In this embodiment, the application scenarios of GPU reasoning performance optimization include but are not limited to models such as image classification, target detection, and semantic segmentation, and are not limited to the algorithm implementation scenarios in the autonomous driving industry. Any other fields or applications involving transformer models in GPU reasoning can use the method of this embodiment to achieve performance tuning. The method of this embodiment is a general transformer model reasoning performance tuning strategy.
[0065] 2. Operating platform description: The GPU inference performance optimization method in this embodiment runs on NVIDIA chip platforms such as orin\xavier. The entire optimization process can be implemented using python scripts and TensorRT and Cuda tools.
[0066] 3. Obtain initial model performance: Use the original ONNX model Model_0 and obtain the model's initial inference performance index by running inference on the input data, denoted as P_0.
[0067] 4. Optimize the Attention operator: Optimize the Attention operator in the original ONNX model Model_0, including the following steps:
[0068] Iteratively search the network graph of the original ONNX model Model_0, traverse the network layers in the model Model_0 layer by layer, and perform context model matching on the network layers. The Attention operator consists of 2 Matmul layers + 1 softmax layer. In this embodiment, the search is performed starting from the softmax layer to check whether the previous and next layers are both Matmul types. If the softmax layer and its previous and next layers are all Matmul types, it is marked as an Attention operator, such as Figure 2 shown.
[0069] After finding all the Attention operators in the model Model_0, replace them with the custom fused AttentionFusionOp operator. AttentionFusionOp is a custom operator obtained after re-integrating the Attention operator structure in the model Model_0 to achieve better operation.
[0070] After the replacement is completed, the new ONNX model is recorded as Model_1, and the evaluation model inference performance is recorded as P_1.
[0071] It should be noted that the above AttentionFusionOp is a custom fusion operator implemented with reference to the flash attention algorithm. Specifically, the flashattention algorithm is used to divide the calculation of Matmul+softmax+Matmul into blocks, and the input is divided into blocks and the attention operation is performed on each block, thereby reducing the read and write operations on the high bandwidth memory (HBM).
[0072] 5. Optimize the LayerNorm operator: Optimize the LayerNorm operator in the model Model_1, including the following steps:
[0073] Iterate the network graph of model Model_1, traverse the network layers in model Model_1 layer by layer, and perform context model matching of the network layers. The LayerNorm operator includes 2 Add operators, 2 reduceMean operators, a sub operator, a pow operator, a Sqrt operator, a Div operator, and a mul operator. In this embodiment, the search starts from the sqrt layer, such as Figure 3 As shown, for example, the sqrt layer is followed by the Div operator and preceded by the Add operator, which is recorded as the LayerNorm operator. The calculation formula is as follows:
[0074]
[0075] Among them, X is the input data, that is, a tensor in the model, μ represents the sample mean, that is, the mean of the tensor X,
[0076] σ2 represents the variance of the tensor X, ∈ represents the smallest positive number that avoids the denominator being 0, γ represents the weight factor, and β represents the bias.
[0077] After all LayerNorm operators are found in model Model_1, they are replaced with custom fused LNFusionOp operators. LNFusionOp is a custom operator obtained by reintegrating the LayerNorm operator structure in model Model_1.
[0078] After the replacement is completed, the new ONNX model is recorded as Model_2, and the evaluated model reasoning performance is recorded as P_2.
[0079] It should be noted that the above LNFusionOp reimplements a complete kernel based on the official formula of layernorm, thus packaging it into a custom fusion operator. Reimplementation in the kernel achieves higher computational efficiency.
[0080] 6. Optimize the GBR operator: Optimize the GBR operator in model Model_2, including the following steps:
[0081] Iterate the network graph of model Model_2, traverse the network layers in model Model_2 layer by layer, and perform context model matching of the network layers. The GBR operator is a fusion of three operators: Matmul+Add+Relu. In this embodiment, the search starts from the Matmul layer, and the subsequent Add layer and Relu layer are satisfied, which are recorded as the GBR operator. Figure 4 shown.
[0082] After all GBR operators are found in model Model_2, they are replaced with the custom fused GBRFusionOp operator. GBRFusionOp is a custom operator obtained after the GBR operator structure in model Model_2 is reintegrated to achieve better operation.
[0083] After the replacement is completed, the new ONNX model is recorded as Model_3, and the evaluation model reasoning performance is recorded as P_3.
[0084] It should be noted that the above GBRFusionOp repackages the operation logic of Matmul, Add, and Relu into a complete kernel to recreate a custom fusion operator, saving context switching and memory copying between operators.
[0085] 7. Optimize GB operator: Optimize the GB operator in model Model_3, including the following steps:
[0086] Iterate the network graph of model Model_3, traverse the network layers in model Model_3 layer by layer, and perform context model matching of the network layers. The GB operator is the fusion of the two operators Matmul+Add. In this embodiment, the search starts from the Matmul layer, and the subsequent Add layer is satisfied, which is recorded as the GB operator, such as Figure 5 shown.
[0087] After all GB operators are found in model Model_3, they are replaced with the custom fused GBFusionOp operator. GBFusionOp is a custom operator obtained after the GB operator structure gemm+bias in model Model_3 is reintegrated to achieve better operation.
[0088] After the replacement is completed, the new ONNX model is recorded as Model_4, and the evaluation model reasoning performance is recorded as P_4.
[0089] It should be noted that the above GBFusionOp is the operation logic of Matmul and Add repackaged in a complete kernel to recreate the custom fusion operator.
[0090] 8. Obtain the final optimization result: Compare the performance P_0, performance P_1, performance P_2, and performance P_3 to find the model with the best performance, including the following steps:
[0091] Use Model_0, Model_1, Model_2, and Model_3 to perform model inference operations in the same device environment (for example, on the orin-X board);
[0092] Count the total time consumed for the preset number of inferences and calculate the average time consumed;
[0093] The performance of each operator under optimization is evaluated based on the average time consumption.
[0094] For example, the total time consumed for 1,000 inferences is counted, and the average time consumed is infer_time_0~3, in milliseconds. The performance of each operator under optimization is evaluated based on the time consumed. Usually, the model with the shortest inference time has the best performance result. Figure 6 As shown, the performance optimization method of the present invention is compared with the original optimization method, based on GPU reasoning of the transformer model, the test environment StreamPETRonorin, and FPS is the number of frames that can be processed per second.
[0095] The above embodiments disclose the performance optimization method of the present invention. Compared with the prior art, the method has the following advantages:
[0096] 1. The performance optimization method of the present invention is a general transformer model reasoning performance tuning strategy, which is applicable to a variety of application scenarios and model types;
[0097] 2. The performance optimization method of the present invention optimizes the Attention operator, LayerNorm operator, GBR operator, GB operator, etc. by replacing and merging custom operators, significantly reducing the amount of calculation or memory usage, and improving model performance;
[0098] 3. The performance optimization method of the present invention, the optimization steps include iterative search, pattern matching, operator replacement and fusion, which improves the training speed and reasoning speed of the deep learning model and reduces resource consumption and cost.
[0099] In another embodiment of the present invention, a GPU reasoning performance optimization system based on a transformer model is disclosed, which is used to implement the above-mentioned performance optimization method, including an original model acquisition reasoning module, an Attention operator optimization module, a LayerNorm operator optimization module, a GBR operator optimization module, a GB operator optimization module, and a performance comparison module;
[0100] The original model acquisition reasoning module is used to obtain the ONNX model Model_0, and perform reasoning through input data to obtain the initial performance P_0 of the model Model_0;
[0101] The Attention operator optimization module is used to traverse the model Model_0, find and replace the Attention operator to obtain Model_1, and perform reasoning through input data to obtain the performance P_1 of the model Model_1;
[0102] The LayerNorm operator optimization module is used to traverse the model Model_1, find and replace the LayerNorm operator to obtain Model_2, and perform reasoning through input data to obtain the performance P_2 of Model_2;
[0103] The GBR operator optimization module is used to traverse the model Model_2, find and replace the GBR operator to obtain Model_3, and perform reasoning through input data to obtain the performance P_3 of Model_3;
[0104] The GB operator optimization module is used to traverse the model Model_3, find and replace the GB operator to obtain Model_4, and perform reasoning through input data to obtain the performance P_4 of Model_4;
[0105] The performance comparison module compares the above performance P_1, performance P_2, performance P_3, and performance P_4 to find the model version with the best performance, and obtains the optimal model and the best performance optimization result.
[0106] In another embodiment of the present invention, an electronic device is also included. The electronic device may include a processor and a memory storing computer program instructions.
[0107] Specifically, in this embodiment, the processor may include a central processing unit (CPU), or a specific integrated circuit, or may be configured as one or more integrated circuits of this embodiment; the memory may include a large-capacity memory for data or instructions, including but not limited to a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these; where appropriate, the memory may include a removable or non-removable (or fixed) medium; in a specific embodiment, the memory is a non-volatile solid-state memory. In a specific embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.
[0108] The processor implements the performance optimization method disclosed above in the present invention by reading and executing computer program instructions stored in the memory.
[0109] It should also be noted that the electronic device of this embodiment may also include a communication interface and a communication bus. Among them, the processor, memory, and communication interface are connected through the communication bus and communicate with each other. The communication interface is mainly used to realize the communication between the various units, modules, devices or equipment in the embodiment of the present invention.
[0110] The communication bus mentioned above includes hardware, software or a combination of hardware and software, coupling the components of the online data flow device to each other. Where appropriate, the communication bus may include one or more buses.
[0111] In addition, in combination with the power grid electricity theft detection method in the above embodiment, the embodiment of the present invention can be implemented by providing a computer storage medium, on which computer program instructions are stored; the computer program instructions are executed by a processor to perform the above performance optimization method.
[0112] It should be clear that the present invention is not limited to the methods, systems, and devices disclosed above, but also includes various changes, modifications, and additions made by those skilled in the art based on the ideas of the present invention, or changes in the order of steps.
[0113] When the present invention is implemented in hardware, it can be an electronic circuit, a dedicated integrated circuit, appropriate firmware, plug-in, function card, etc.; when implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks, and the programs or code segments can be stored in a machine-readable medium, or uploaded on a transmission medium or communication link through a data signal carried in a carrier wave. "Machine-readable medium" may include any medium capable of storing or transmitting information, such as: electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, optical disks, hard disks, optical fiber media, radio frequency links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0114] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.
Claims
1. A GPU inference performance optimization method based on a transformer model, characterized in that: This method runs on the NVIDIA chip platform and uses Python scripts combined with TensorRT and Cuda tools to optimize inference performance. The specific steps include: Get the original ONNX model Model_0; use the input data to run inference through the ONNX model and record the initial performance indicator, recorded as P_0; Optimize the Attention operator in the original ONNX model Model_0 to obtain the new model Model_1; evaluate the inference performance of Model_1, denoted as P_1; Optimize the LayerNorm operator in the model Model_1 to obtain the new model Model_2; evaluate the inference performance of Model_2, denoted as P_2; Optimize the GBR operator in model Model_2 to obtain a new model Model_3; evaluate the inference performance of Model_3, denoted as P_3; Optimize the GB operator in model Model_3 to obtain a new model Model_4; evaluate the inference performance of Model_4, denoted as P_4; Compare the performance of the original model P_0, the performance after optimizing the Attention operator P_1, the performance after optimizing the LayerNorm operator P_2, the performance after optimizing the GBR operator P_3, and the performance after optimizing the GB operator P_4 to find the model version with the best performance, recorded as Model_best, which is the optimal model and the best performance optimization result of the above method.
2. The method for optimizing GPU inference performance based on a transformer model according to claim 1, characterized in that: Optimizing the Attention operator in the original ONNX model Model_0 includes the following steps: Iteratively search the network graph of the original ONNX model Model_0, traverse the network layers in the model Model_0 layer by layer, and perform network layer context pattern matching; Among them, the Attention operator consists of 2 Matmul layers + 1 softmax layer; After finding all the Attention operators in the model Model_0, replace them with the custom fused AttentionFusionOp operator. AttentionFusionOp is a custom operator obtained by re-integrating the Attention operator structure in the model Model_0. After the replacement is completed, the new ONNX model is recorded as Model_1.
3. The GPU reasoning performance optimization method based on the transformer model according to claim 1, characterized in that: Optimizing the LayerNorm operator in the model Model_1 includes the following steps: Iterate the network graph of model Model_1, traverse the network layers in model Model_1 layer by layer, and perform network layer context pattern matching; Among them, the LayerNorm operator includes 2 Add operators, 2 reduceMean operators, a sub operator, a pow operator, a Sqrt operator, a Div operator and a mul operator; After finding all LayerNorm operators in model Model_1, replace them with custom fused LNFusionOp operators. LNFusionOp is a custom operator obtained by reintegrating the LayerNorm operator structure in model Model_1. After the replacement is completed, the new ONNX model is recorded as Model_2.
4. The method for optimizing GPU inference performance based on a transformer model according to claim 1, characterized in that: Optimizing the GBR operator in model Model_2 includes the following steps: Iteratively search the network graph of model Model_2, traverse the network layers in model Model_2 layer by layer, and perform network layer context pattern matching; Among them, the GBR operator is the fusion of Matmul+Add+Relu operators; After finding all GBR operators in model Model_2, replace them with the custom fused GBRFusionOp operator. GBRFusionOp is a custom operator obtained by reintegrating the GBR operator structure gemm+bias+relu in model Model_2. After the replacement is completed, the new ONNX model is recorded as Model_3.
5. The method for optimizing GPU reasoning performance based on a transformer model according to claim 1, characterized in that: Optimizing the GB operator in the model Model_3 includes the following steps: Iterate the network graph of model Model_3, traverse the network layers in model Model_3 layer by layer, and perform network layer context pattern matching; Among them, the GB operator is the fusion of the two operators Matmul+Add; After finding all GB operators in model Model_3, replace them with the custom fused GBFusionOp operator. GBFusionOp is a custom operator obtained by re-integrating the GB operator structure gemm+bias in model Model_3. After the replacement is completed, the new ONNX model is recorded as Model_4.
6. The method for optimizing GPU reasoning performance based on a transformer model according to claim 1, characterized in that: The comparison of performance P_0, performance P_1, performance P_2, and performance P_3 includes the following steps: Use Model_0, Model_1, Model_2, and Model_3 to perform model inference operations in the same device environment; Count the total time taken for the preset number of inferences and calculate the average time taken; The performance of each operator under optimization is evaluated based on the average time consumption.
7. A GPU inference performance optimization system based on a transformer model according to the optimization method of any one of claims 1 to 6, characterized in that: The system includes an original model acquisition reasoning module, an Attention operator optimization module, a LayerNorm operator optimization module, a GBR operator optimization module, a GB operator optimization module, and a performance comparison module; The original model acquisition reasoning module is used to obtain the ONNX model Model_0, and perform reasoning through input data to obtain the initial performance P_0 of the model Model_0; The Attention operator optimization module is used to traverse the model Model_0, find and replace the Attention operator to obtain Model_1, and perform reasoning through input data to obtain the performance P_1 of the model Model_1; The LayerNorm operator optimization module is used to traverse the model Model_1, find and replace the LayerNorm operator to obtain Model_2, and perform reasoning through input data to obtain the performance P_2 of Model_2; The GBR operator optimization module is used to traverse the model Model_2, find and replace the GBR operator to obtain Model_3, and perform reasoning through input data to obtain the performance P_3 of Model_3; The GB operator optimization module is used to traverse the model Model_3, find and replace the GB operator to obtain Model_4, and perform reasoning through input data to obtain the performance P_4 of Model_4; The performance comparison module compares the above performance P_1, performance P_2, performance P_3, and performance P_4 to find the model version with the best performance, and obtains the optimal model and the best performance optimization result.
8. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein the memory is used to store a computer program; The processor is configured to execute the method according to any one of claims 1 to 6 by running the computer program stored in the memory.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program implements the method described in any one of claims 1 to 6 when executed by a processor.