Mixing precision calculation method and device

Abstract

The invention provides a mixing precision calculation method and device, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a mixing precision calculation graph, and performing compilation optimization according to the mixing precision calculation graph to obtain an optimized mixing precision calculation graph; and optimizing the mixing precision operator, and calculating to obtain a mixing precision calculation result according to the optimized mixing precision operator and the optimized mixing precision calculation graph. The device executes the method. According to the mixing precision calculation method and device provided by the embodiment of the invention, the mixing precision calculation efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mixed precision computing method and device. Background Art

[0002] Artificial intelligence has played a key role in many areas of production and life, such as chatbots, speech recognition, machine translation, and autonomous driving. Since artificial intelligence applications have extremely high demands on computing power, and the development of Moore's Law has shown a significant slowdown in recent years, the development of the artificial intelligence industry has been restricted to a certain extent. Compared with traditional scientific computing applications, artificial intelligence applications have more relaxed requirements on computing accuracy. Many hardware companies have launched chips with multiple precisions, including INT4, INT8, and FP8. Since mixed precision computing can maintain the computing accuracy of artificial intelligence applications, mixed precision computing has become an important development trend of artificial intelligence applications.

[0003] The existing technology has proposed various compilation optimizations (such as layer optimization, loop unrolling, and data layout optimization) to achieve scalable mixed-precision computing systems. In addition, in order to hide the potential complex system details from researchers and focus on model design, the study of manually designed automatic code generation methods is used to automatically generate different computing kernels to improve the computing efficiency of artificial intelligence systems.

[0004] However, existing compilation optimization technologies still have difficulty in implementing compilation optimization for mixed-precision computation graphs. Existing mixed-precision computation systems usually use specific data sets to support mixed-precision outlier search and mixed-precision computation. Although the high memory consumption of mixed-precision computation has been alleviated through preprocessing, the computational efficiency has been affected.

[0005] In addition, the existing technology is based on the design of single-precision computational graphs and does not consider how to fuse mixed-precision operators. If the existing technology is used for automatic code generation, the generated mixed-precision code performs calculations of different precisions at the global memory level and cannot simultaneously fuse operations of multiple precisions at the register level. The operating efficiency of registers is much higher than that of global memory, so the performance of the code generated using the above method is poor. Summary of the invention

[0006] In view of the problems in the prior art, an embodiment of the present invention provides a mixed precision computing method and device, which can at least partially solve the problems in the prior art.

[0007] In one aspect, the present invention provides a mixed precision computing method, comprising:

[0008] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0009] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0010] The step of compiling and optimizing the mixed precision computation graph to obtain an optimized mixed precision computation graph includes:

[0011] Obtaining an activation matrix, calculating a scaling factor according to a set of outliers of the activation matrix, and performing quantization calculation on the scaling factor;

[0012] If it is determined that at least one scaling factor after quantization calculation is greater than or equal to the threshold, outliers are searched, and the outlier set is updated according to the search result to obtain the optimized mixed precision calculation graph.

[0013] The mixed precision calculation method further includes:

[0014] If it is determined that the scaling factors after all quantization calculations are less than the threshold, the mixed precision calculation graph is used as the optimized mixed precision calculation graph.

[0015] The optimizing of the mixed precision operator includes:

[0016] Optimizes mixed-precision matrix multiplication.

[0017] The optimizing of mixed precision matrix multiplication includes:

[0018] Obtain in a shared memory a first activation matrix of INT8 type and a second activation matrix of FP16 type, a first weight matrix corresponding to the first activation matrix, and a second weight matrix corresponding to the second activation matrix;

[0019] Multiply the first activation matrix and the first weight matrix in a register to obtain a first matrix of INT32 type, and multiply the second activation matrix and the second weight matrix to obtain a second matrix of FP32 type;

[0020] Performing a dequantization calculation on the first matrix of the INT32 type to obtain a first matrix of the FP32 type, and performing an addition calculation on the first matrix of the FP32 type and the second matrix of the FP32 type to obtain a matrix of the FP32 type;

[0021] The FP32 type matrix is ​​converted in the global memory to obtain a FP16 type matrix.

[0022] The mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph, including:

[0023] Using optimized mixed-precision matrix multiplication to perform matrix multiplication calculation on the optimized mixed-precision computation graph;

[0024] Dequantization is performed on the matrix multiplication calculation result to obtain the mixed precision calculation result.

[0025] The step of performing matrix multiplication calculation on the optimized mixed precision computation graph by using the optimized mixed precision matrix multiplication includes:

[0026] The optimized mixed-precision matrix multiplication is used to perform matrix multiplication calculation of mixed-precision FP16 type data and INT8 / 4 type data on the optimized mixed-precision computation graph.

[0027] In one aspect, the present invention provides a mixed precision computing device, comprising:

[0028] An acquisition unit, used to acquire a mixed precision computation graph, and perform compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0029] The computing unit is used to optimize the mixed precision operator and calculate the mixed precision computing result according to the optimized mixed precision operator and the optimized mixed precision computing graph.

[0030] In another aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0031] The processor and the memory communicate with each other via the bus;

[0032] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the following method:

[0033] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0034] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0035] An embodiment of the present invention provides a non-transitory computer-readable storage medium, including:

[0036] The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the following method:

[0037] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0038] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0039] The mixed precision computing method and device provided in the embodiments of the present invention obtain a mixed precision computing graph, perform compilation optimization based on the mixed precision computing graph, and obtain an optimized mixed precision computing graph; optimize the mixed precision operator, and calculate the mixed precision computing result based on the optimized mixed precision operator and the optimized mixed precision computing graph, thereby improving the mixed precision computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0041] Figure 1 It is a flowchart of a mixed precision calculation method provided by an embodiment of the present invention.

[0042] Figure 2 It is a flowchart of an optimization method for mixed-precision computing provided by existing technologies.

[0043] Figure 3 It is a flowchart of an optimization method for mixed-precision computing provided by an embodiment of the present invention.

[0044] Figure 4 It is a flow chart of the mixed precision calculation method provided by the prior art.

[0045] Figure 5 It is a flow chart of a mixed precision computing method for compilation optimization provided by an embodiment of the present invention.

[0046] Figure 6 It is a flowchart of applying the mixed precision computing method provided by an embodiment of the present invention to artificial intelligence.

[0047] Figure 7 It is a flowchart of the kernel fusion method provided by the prior art.

[0048] Figure 8 It is a schematic flow chart of the optimized kernel fusion method provided by an embodiment of the present invention.

[0049] Figure 9(a) and Figure 9(b) are diagrams showing the performance improvement effect brought about by the compilation optimization of the mixed-precision computation graph provided by an embodiment of the present invention.

[0050] Figures 10(a), 10(b) and 10(c) are diagrams showing the performance improvement effect brought by the mixed-precision operator optimization provided by an embodiment of the present invention on an open source 8-bit framework.

[0051] Figures 11(a), 11(b) and 11(c) are diagrams showing the performance improvement effect brought by the mixed-precision operator optimization provided by an embodiment of the present invention on an open source 4-bit framework.

[0052] Figures 12(a), 12(b), 12(c) and 12(d) are diagrams showing the end-to-end acceleration effects achieved on the NVIDIA A100 GPU platform for different model structures relative to other frameworks, as provided by an embodiment of the present invention.

[0053] Fig.13 This is a diagram of the end-to-end acceleration effect that can be achieved on the H100 GPU platform for different model structures compared to other frameworks, as provided by an embodiment of the present invention.

[0054] FIG. 14( a ) and FIG. 14( b ) are diagrams showing the effect of reducing computing delay due to end-to-end optimization provided in an embodiment of the present invention.

[0055] Fig.15 It is a schematic diagram of the structure of a mixed-precision computing device provided by an embodiment of the present invention.

[0056] Fig.16 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily.

[0058] Explanation of related terms:

[0059] 1.Large Language Model (LLM): It is a natural language processing model based on deep learning, whose goal is to generate natural language text that conforms to grammatical and semantic rules.

[0060] 2. Mixed-precision computing: It is a technique to improve the speed of training and reasoning and reduce memory usage in deep learning. In PyTorch, it uses a combination of half-precision floating point numbers (16-bit floating point numbers, FP16) and low-precision integers (8-bit or 4-bit integers, INT8 / INT4).

[0061] 3. Outliers: refers to points in the sample space that are inconsistent with the general behavior or characteristics of other sample points, and are usually calculated with high precision.

[0062] 4. Non-Outliers: refers to points in the sample space that have the same general behavior or characteristics as other sample points, and are usually calculated using low precision.

[0063] 5.INT8: INT8 is a data type that represents an 8-bit signed integer. It can store integers ranging from -128 to 127. Compared with other data types, INT8 has a smaller storage space and only takes up one byte.

[0064] 6.INT4: INT4 is a data type that represents a 4-bit signed integer. It can store integers ranging from -8 to 7. Compared with other data types, INT8 has a smaller storage space and only takes up half a byte.

[0065] 7. Operator: A computing unit used to perform various mathematical operations and operations.

[0066] 8. Operator Fusion: It is to merge multiple computing units into one computing unit to complete the calculation, reduce the intermediate data reading and writing memory operations, and thus save computing time.

[0067] 9. Moore's Law: When the price remains unchanged, the number of components that can be accommodated on an integrated circuit will double approximately every 18-24 months, and the performance will also double.

[0068] Figure 1 is a flow chart of a mixed precision calculation method provided by an embodiment of the present invention. Figure 1 As shown, the mixed precision calculation method provided by the embodiment of the present invention includes:

[0069] Step S1: Obtain a mixed precision computation graph, and perform compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph.

[0070] Step S2: Optimize the mixed precision operator, and calculate the mixed precision calculation result according to the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0071] In the above step S1, the device obtains a mixed precision computation graph, and performs compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph. The device may be a computer device that executes the method.

[0072] like Figure 2 and Figure 3 As shown, the method of the present invention is embodied in the following aspects compared with the existing method:

[0073] 1. Added compilation optimization steps to avoid the "online detection of outliers" step.

[0074] 2. Added mixed precision operator optimization steps.

[0075] The compiling and optimizing the mixed precision computation graph to obtain an optimized mixed precision computation graph includes:

[0076] Obtaining an activation matrix, calculating a scaling factor according to a set of outliers of the activation matrix, and performing quantization calculation on the scaling factor;

[0077] If it is determined that at least one quantized scaling factor is greater than or equal to the threshold, outliers are searched, and the outlier set is updated according to the search result to obtain the optimized mixed precision calculation graph. The threshold can be set autonomously according to actual conditions.

[0078] The mixed precision calculation method further includes:

[0079] If it is determined that the scaling factors after all quantization calculations are less than the threshold, the mixed precision calculation graph is used as the optimized mixed precision calculation graph. Figure 4 and Figure 5 As shown, the method of the present invention is embodied in the following aspects compared with the existing method:

[0080] 1. Dynamically update the outlier set based on the search results of outliers.

[0081] 2. Added a judgment link for comparing the scaling factor with the threshold.

[0082] Finding outliers at runtime has high requirements for latency, so the method of the present invention proposes a lightweight detection algorithm to meet the time limit requirements. It can ensure that the current calculation process perceives and determines in advance whether there are outliers in the mixed precision calculation graph, and re-perceives and searches for outliers in the mixed precision calculation graph at the right time, so as to achieve mixed precision calculation faster.

[0083] The compilation optimization of the method of the present invention refers to optimizing the execution process of the mixed precision calculation graph, by changing the execution process of the original calculation graph, performing an equivalent transformation on the calculation graph, and obtaining the optimized calculation graph without changing the execution result. During the compilation optimization, the mixed precision calculation graph is first obtained, and then the mixed precision calculation graph is equivalently transformed, and finally the mixed precision calculation graph after the equivalent transformation is obtained. The above method is based on the problem of high overhead in calculating outliers. Before calculating the outliers, the scaling factor is calculated to determine whether there are any new outliers. If none of the values ​​in the scaling factor exceeds the threshold θ, it means that there are no new outliers in the activation matrix, and the calculation of the outliers is skipped. The core idea of ​​the algorithm is to use the locality of outliers to reduce the time overhead of calculating outliers, and ultimately obtain performance gains.

[0084] The method proposed in this paper considers the kernel computation overhead of multiple precisions when generating mixed precision code, which will reduce the time overhead of kernel computation of more different precisions. First, a computation kernel optimization for mixed precision computation is constructed. It contains more candidate mixed precision fusion methods than the single precision fusion scheme in previous work, providing greater performance opportunities, such as Figure 6 As shown in the figure, it is a mixed precision calculation process combined with artificial intelligence applications. Among them, the points of the activation matrix represent outliers, q_proj represents query projection, k_proj represents key projection, and v_proj represents value projection. gate_proj represents gate projection, up_proj represents up projection, down_proj represents down projection, and o_proj represents out projection.

[0085] In the above step S2, the device optimizes the mixed precision operator, and calculates the mixed precision calculation result according to the optimized mixed precision operator and the optimized mixed precision calculation graph. The optimization of the mixed precision operator includes:

[0086] Optimizing mixed precision matrix multiplication. Optimizing mixed precision matrix multiplication includes:

[0087] Obtain in a shared memory a first activation matrix of INT8 type and a second activation matrix of FP16 type, a first weight matrix corresponding to the first activation matrix, and a second weight matrix corresponding to the second activation matrix;

[0088] Multiply the first activation matrix and the first weight matrix in a register to obtain a first matrix of INT32 type, and multiply the second activation matrix and the second weight matrix to obtain a second matrix of FP32 type;

[0089] Performing a dequantization calculation on the first matrix of the INT32 type to obtain a first matrix of the FP32 type, and performing an addition calculation on the first matrix of the FP32 type and the second matrix of the FP32 type to obtain a matrix of the FP32 type;

[0090] The FP32 type matrix is ​​converted in the global memory to obtain the FP16 type matrix. Figure 7 and Figure 8 As shown, the calculation process steps executed in different storage media by the method of the present invention are different from those of the existing methods.

[0091] Mixed precision calculation mainly includes three operations: scale factor calculation, quantization and outlier calculation. The method of the present invention provides a fused quantization kernel for these three operations. On the calculation layer, the fusion operators between the last layer (such as LayerNorm) before matrix multiplication are combined and code generation is implemented. Wherein Xi8, Xf16 represent activation matrices of INT8 type and FP16 type activation matrices of data type, respectively. Wi8, Wf16 represent weight matrices of INT8 type and FP16 type weight matrices of data type, respectively. Ci32, Cf32 and Cf16 represent matrices of INT32 type, FP32 type and FP16 type data types, respectively. Compared with the improvements made in the existing methods, the method proposed in the present invention reduces the number of times Ci32 is read and written to the global memory, reduces the overhead of reading and writing, and improves the utilization of bandwidth.

[0092] Figure 9(a) and Figure 9(b) show the performance improvement brought by the mixed precision computation graph compilation optimization of the present invention; Bitsandbytes represents the reference performance calculated based on the Bitsandbytes standard library. LLaMA-7B represents the open source LLaMA-7 billion parameter language model, LLaMA-13B represents the open source LLaMA-13 ​​billion parameter language model, and LLaMA-70B represents the open source 70 billion parameter language model.

[0093] Figures 10(a), 10(b) and 10(c) show the performance improvement effect of the mixed precision operator optimization of the present invention on an open source 8-bit framework, with the ordinate representing one trillion floating-point operations per second. Figures 11(a), 11(b) and 11(c) show the performance improvement effect of the mixed precision operator optimization of the present invention on an open source 4-bit framework, with the ordinate representing one trillion floating-point operations per second.

[0094] Figures 12(a), 12(b), 12(c), and 12(d) show the end-to-end acceleration effect achieved for different model structures on the NVIDIA A100 GPU platform relative to other frameworks. The ordinate represents throughput, i.e., the number of characters processed per second, and the abscissa represents the problem size. Fig.13 It demonstrates that end-to-end acceleration can be achieved for different model structures on the H100 GPU platform compared to other frameworks.

[0095] Figures 14(a) and 14(b) show the effect of reducing computing delay brought about by the end-to-end optimization of the present invention. The vertical axis represents time in milliseconds, and the horizontal axis represents the problem scale. The lower the delay, the better the effect.

[0096] The present invention aims to achieve compilation optimization and efficient automatic code generation for mixed precision computing, combining the performance characteristics of mixed precision computing graph loads and unique computing kernels to achieve efficient automatic parallelization. This study intends to implement a two-stage solution: at compile time, the optimized program is obtained through mixed precision layer fusion and code generation; at runtime, the compilation optimization set is quickly and dynamically adapted to ensure that the current mixed precision strategy is suitable for the current workload.

[0097] The mixed precision computing method provided in the embodiment of the present invention obtains a mixed precision computing graph, performs compilation optimization based on the mixed precision computing graph, and obtains an optimized mixed precision computing graph; optimizes the mixed precision operator, and calculates the mixed precision computing result based on the optimized mixed precision operator and the optimized mixed precision computing graph, thereby improving the mixed precision computing efficiency.

[0098] Furthermore, compiling and optimizing the mixed precision computation graph to obtain an optimized mixed precision computation graph includes:

[0099] An activation matrix is ​​obtained, a scaling factor is calculated according to a set of outliers of the activation matrix, and the scaling factor is quantized. This can be described with reference to the above embodiment and will not be described in detail.

[0100] If it is determined that at least one quantized scaling factor is greater than or equal to the threshold, outliers are searched, and the outlier set is updated according to the search result to obtain the optimized mixed precision calculation graph. The above embodiment can be referred to for explanation and will not be repeated here.

[0101] Furthermore, the mixed precision calculation method further includes:

[0102] If it is determined that all the scaling factors after quantization calculation are less than the threshold, the mixed precision calculation graph is used as the optimized mixed precision calculation graph. The above embodiment can be referred to for explanation and will not be described in detail.

[0103] Furthermore, the optimizing the mixed precision operator includes:

[0104] The mixed precision matrix multiplication is optimized. Please refer to the above embodiment for description, which will not be repeated here.

[0105] Further, the optimizing the mixed precision matrix multiplication includes:

[0106] Obtain in the shared memory a first activation matrix of INT8 type and a second activation matrix of FP16 type, a first weight matrix corresponding to the first activation matrix, and a second weight matrix corresponding to the second activation matrix; the above-mentioned embodiment can be referred to for description and will not be repeated here.

[0107] The first activation matrix and the first weight matrix are multiplied in the register to obtain a first matrix of INT32 type, and the second activation matrix and the second weight matrix are multiplied to obtain a second matrix of FP32 type; the above embodiment can be referred to for description and will not be repeated here.

[0108] The first matrix of the INT32 type is dequantized to obtain a first matrix of the FP32 type, and the first matrix of the FP32 type and the second matrix of the FP32 type are added to obtain a matrix of the FP32 type; the above-mentioned embodiment can be referred to for description and will not be repeated here.

[0109] The FP32 type matrix is ​​converted in the global memory to obtain an FP16 type matrix. The above embodiment can be referred to for explanation and will not be described in detail.

[0110] Furthermore, the mixed precision calculation result is calculated according to the optimized mixed precision operator and the optimized mixed precision calculation graph, including:

[0111] The optimized mixed-precision matrix multiplication is used to perform matrix multiplication calculation on the optimized mixed-precision computation graph; the description may refer to the above embodiment and will not be repeated here.

[0112] The matrix multiplication calculation result is dequantized to obtain the mixed precision calculation result. The above description can be made with reference to the above embodiment and will not be repeated here.

[0113] Further, the adopting the optimized mixed precision matrix multiplication to perform matrix multiplication calculation on the optimized mixed precision computation graph includes:

[0114] The optimized mixed precision matrix multiplication is used to perform matrix multiplication calculation of mixed precision FP16 type data and INT8 / 4 type data on the optimized mixed precision calculation graph. The above embodiment can be referred to for description and will not be repeated here.

[0115] Fig.15 is a schematic diagram of the structure of a mixed precision computing device provided by an embodiment of the present invention. Fig.15 As shown, the mixed precision computing device provided by the embodiment of the present invention includes an acquisition unit 1501 and a computing unit 1502, wherein:

[0116] The acquisition unit 1501 is used to acquire a mixed precision calculation graph, and perform compilation optimization based on the mixed precision calculation graph to obtain an optimized mixed precision calculation graph; the calculation unit 1502 is used to optimize the mixed precision operator, and calculate the mixed precision calculation result based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0117] Specifically, the acquisition unit 1501 in the device is used to acquire a mixed precision calculation graph, compile and optimize the mixed precision calculation graph, and obtain an optimized mixed precision calculation graph; the calculation unit 1502 is used to optimize the mixed precision operator, and calculate the mixed precision calculation result based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0118] The mixed precision computing device provided in an embodiment of the present invention obtains a mixed precision computing graph, compiles and optimizes the mixed precision computing graph, and obtains an optimized mixed precision computing graph; optimizes the mixed precision operator, and calculates the mixed precision computing result based on the optimized mixed precision operator and the optimized mixed precision computing graph, thereby improving the mixed precision computing efficiency.

[0119] An embodiment of the mixed precision computing device provided by the embodiment of the present invention can be specifically used to execute the processing flow of the above-mentioned method embodiments. Its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0120] Fig.16 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig.16 As shown, the electronic device includes: a processor (processor) 1601, a memory (memory) 1602 and a bus 1603;

[0121] The processor 1601 and the memory 1602 communicate with each other via a bus 1603;

[0122] The processor 1601 is used to call the program instructions in the memory 1602 to execute the methods provided by the above method embodiments, for example, including:

[0123] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0124] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0125] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the methods provided by the above method embodiments, for example, including:

[0126] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0127] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0128] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables the computer to execute the methods provided by the above method embodiments, for example, including:

[0129] Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph;

[0130] The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

[0131] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0135] In the description of this specification, the description with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0136] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A mixed precision computing method, characterized in that: include: Obtaining a mixed precision computation graph, and performing compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph; The mixed precision operator is optimized, and a mixed precision calculation result is calculated based on the optimized mixed precision operator and the optimized mixed precision calculation graph.

2. The mixed precision computing method according to claim 1, characterized in that: The compiling and optimizing the mixed precision computation graph to obtain an optimized mixed precision computation graph includes: Obtaining an activation matrix, calculating a scaling factor according to a set of outliers of the activation matrix, and performing quantization calculation on the scaling factor; If it is determined that at least one scaling factor after quantization calculation is greater than or equal to the threshold, outliers are searched, and the outlier set is updated according to the search result to obtain the optimized mixed precision calculation graph.

3. The mixed precision computing method according to claim 2, characterized in that: The mixed precision calculation method further includes: If it is determined that the scaling factors after all quantization calculations are less than the threshold, the mixed precision calculation graph is used as the optimized mixed precision calculation graph.

4. The mixed precision computing method according to claim 1, characterized in that: The optimization of the mixed precision operator includes: Optimizes mixed-precision matrix multiplication.

5. The mixed precision computing method according to claim 4, characterized in that: The optimization of mixed precision matrix multiplication includes: Obtain in a shared memory a first activation matrix of INT8 type and a second activation matrix of FP16 type, a first weight matrix corresponding to the first activation matrix, and a second weight matrix corresponding to the second activation matrix; Perform multiplication calculation on the first activation matrix and the first weight matrix in a register to obtain a first matrix of INT32 type, and perform multiplication calculation on the second activation matrix and the second weight matrix to obtain a second matrix of FP32 type; Performing a dequantization calculation on the first matrix of the INT32 type to obtain a first matrix of the FP32 type, and performing an addition calculation on the first matrix of the FP32 type and the second matrix of the FP32 type to obtain a matrix of the FP32 type; The FP32 type matrix is ​​converted in the global memory to obtain a FP16 type matrix.

6. The mixed precision computing method according to claim 4, characterized in that: The mixed precision calculation result is calculated according to the optimized mixed precision operator and the optimized mixed precision calculation graph, including: Using optimized mixed-precision matrix multiplication to perform matrix multiplication calculation on the optimized mixed-precision computation graph; Dequantization is performed on the matrix multiplication calculation result to obtain the mixed precision calculation result.

7. The mixed precision computing method according to claim 6, characterized in that: The using the optimized mixed-precision matrix multiplication to perform matrix multiplication calculation on the optimized mixed-precision computation graph includes: The optimized mixed-precision matrix multiplication is used to perform matrix multiplication calculation of mixed-precision FP16 type data and INT8 / 4 type data on the optimized mixed-precision computation graph.

8. A mixed precision computing device, characterized in that: include: An acquisition unit, used to acquire a mixed precision computation graph, and perform compilation optimization according to the mixed precision computation graph to obtain an optimized mixed precision computation graph; The computing unit is used to optimize the mixed precision operator and calculate the mixed precision computing result according to the optimized mixed precision operator and the optimized mixed precision computing graph.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images