Data quantification method and system, computer equipment, medium and product

Abstract

The invention discloses a data quantification method and system, computer equipment, a medium and a product, and relates to the technical field of large models.The data quantification method comprises the steps that a to-be-quantized numerical value matrix is obtained, and the to-be-quantized numerical value matrix is divided into parallel processing data sets; normalizing each parallel processing data group based on each preset parallel processing unit to obtain each intermediate to-be-quantized value; and determining a quantization scaling factor based on the maximum value in the intermediate to-be-quantized values, and scaling the intermediate to-be-quantized values based on the quantization scaling factor to obtain quantized values, the maximum value being the maximum value of the absolute values in the intermediate to-be-quantized values. According to the technical scheme, the technical problem of how to improve the reasoning speed of the large language model is solved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a data quantization method, system, computer equipment, medium and product. Background Art

[0002] Low-bit quantization technology aims to reduce the size of the model, reduce computational complexity, and reduce resource consumption by converting high-precision floating-point weights and activation values ​​into low-bit integers. In the practice of large language models, a common strategy is to quantize only the weights without quantizing the activation values, because this approach often better maintains the model's reasoning performance. However, given that activation values ​​account for a large proportion of the reasoning process, the overall performance optimization that can be achieved by simply quantizing weights is relatively limited.

[0003] In order to further improve efficiency, trying to quantize weights and activation values ​​at the same time has become a research direction. Although this method can greatly save computing resources in the inference stage, it will inevitably sacrifice a certain degree of inference accuracy, resulting in a significant decrease in performance. In order to alleviate this contradiction, a strategy of retaining some key data bits without quantization during the quantization process is proposed to restore or approach the inference effect of the original model. However, this approach also introduces more complex quantization operations, which increases the computational burden of the inference process.

[0004] In order to solve the performance bottlenecks that may be caused by these complex quantization operations, existing large language models usually use independent, highly optimized operators to perform these operations when implementing low-bit integer quantization. Although these operators can efficiently process complex quantization logic, they often become performance bottlenecks in the overall operation process, resulting in a decrease in the overall operation speed.

[0005] Therefore, how to improve the speed of reasoning of large language models is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0006] The main purpose of this application is to provide a data quantization method, system, computer device, medium and product, aiming to solve the technical problem of how to improve the speed of reasoning of large language models.

[0007] To achieve the above objectives, the present application proposes a data quantification method, which includes:

[0008] Acquire a numerical matrix to be quantized, and divide the numerical matrix to be quantized into parallel processing data groups;

[0009] Normalizing each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized;

[0010] A quantization scaling factor is determined based on the maximum value among the intermediate values ​​to be quantized, and each intermediate value to be quantized is scaled based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

[0011] In one embodiment, before the step of determining the quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, the method further includes:

[0012] Determine the absolute value corresponding to each of the intermediate values ​​to be quantized, and compare each of the absolute values ​​with a preset threshold value;

[0013] Deleting a target value from each of the intermediate values ​​to be quantized, wherein an absolute value corresponding to the target value is greater than or equal to the threshold;

[0014] The step of determining the quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized comprises:

[0015] After deleting the target value, a quantization scaling factor is determined based on a maximum value among the remaining intermediate values ​​to be quantized.

[0016] In one embodiment, the step of dividing the to-be-quantized numerical matrix into parallel processing data groups comprises:

[0017] Converting the to-be-quantized numerical matrix into a one-dimensional array, wherein the to-be-quantized numerical matrix is ​​a three-dimensional or four-dimensional matrix;

[0018] Based on the preset number of parallel processing units, the values ​​to be quantized in the one-dimensional array are combined to obtain parallel processing data groups, wherein the number of the parallel processing data groups is the same as the number of the parallel processing units.

[0019] In one embodiment, each parallel processing data group includes each value to be quantized;

[0020] The step of normalizing each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized includes:

[0021] Based on each preset parallel processing unit, respectively calculating the square value of each of the to-be-quantized values;

[0022] The root mean square value corresponding to each of the square values ​​is calculated, and the ratio between each of the values ​​to be quantized and the root mean square value is used as the intermediate value to be quantized.

[0023] In one embodiment, the step of determining the quantization scaling factor based on the maximum value of each of the intermediate values ​​to be quantized comprises:

[0024] Acquire a quantization level value, and determine a quantization range value based on the maximum absolute value of each of the intermediate values ​​to be quantized;

[0025] A quantization interval is calculated based on the quantization range value and the quantization level value, and the inverse of the quantization interval is used as a quantization scaling factor.

[0026] In one embodiment, after the step of scaling each of the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, the method further includes:

[0027] Each of the quantized values ​​is input into a linear layer in a large language model.

[0028] In addition, to achieve the above purpose, the present application also proposes a data quantization system, the data quantization system comprising:

[0029] A grouping module, used for obtaining a numerical matrix to be quantized, and dividing the numerical matrix to be quantized into various parallel processing data groups;

[0030] A first processing module, configured to normalize each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized;

[0031] The second processing module determines a quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, and scales the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

[0032] In addition, to achieve the above objectives, the present application also proposes a computer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data quantization method described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data quantization method described above are implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the data quantization method described above are implemented.

[0035] In an embodiment of the present application, by obtaining a matrix of numerical values ​​to be quantized, and dividing the matrix of numerical values ​​to be quantized into various parallel processing data groups; then normalizing each parallel processing data group based on preset parallel processing units to obtain each intermediate numerical value to be quantized; finally, determining a quantization scaling factor based on the maximum value of each intermediate numerical value to be quantized, and scaling each intermediate numerical value to be quantized based on the quantization scaling factor to obtain each quantized completed numerical value, wherein the maximum value is the maximum value of the absolute value of each intermediate numerical value to be quantized, and the originally independent normalization step and quantization step can be integrated into an efficient composite operator. Then, based on the operator fusion strategy proposed in the present application, the number of unnecessary data transmissions between the computing unit (such as the GPU core) and the memory can be reduced, and the number of calls to the GPU computing core by the system can be reduced, thereby accelerating the reasoning process of the entire large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the data quantification method of the present application;

[0039] Figure 2 This is a schematic diagram of operator fusion of an embodiment of the data quantization method of the present application;

[0040] Figure 3 This is a schematic diagram of the processing flow of a specific embodiment of the data quantification method of the present application;

[0041] Figure 4 This is a schematic diagram of the module structure of the data quantization system according to an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the data quantization method in the embodiment of the present application.

[0043] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0045] It is understandable that nowadays, low-bit quantization technology aims to reduce the model size, computational complexity and resource consumption by converting high-precision floating-point weights and activation values ​​into low-bit integers. In the practice of large language models, a common strategy is to quantize only weights without quantizing activation values, because this approach often better maintains the model's reasoning performance. However, given that activation values ​​account for a large proportion of the reasoning process, the overall performance optimization that can be achieved by simply quantizing weights is relatively limited.

[0046] In order to further improve efficiency, trying to quantize weights and activation values ​​at the same time has become a research direction. Although this method can greatly save computing resources in the inference stage, it will inevitably sacrifice a certain degree of inference accuracy, resulting in a significant decrease in performance. In order to alleviate this contradiction, a strategy of retaining some key data bits without quantization during the quantization process is proposed to restore or approach the inference effect of the original model. However, this approach also introduces more complex quantization operations, which increases the computational burden of the inference process.

[0047] In order to solve the performance bottlenecks that may be caused by these complex quantization operations, existing large language models tend to use independent, highly optimized operators to perform these operations when implementing low-bit integer quantization. Although these operators can efficiently handle complex quantization logic, they often become performance bottlenecks in the overall operation process, resulting in a decrease in the overall operation speed.

[0048] Therefore, how to improve the speed of reasoning of large language models is a technical problem that needs to be solved by those skilled in the art.

[0049] This application can integrate the originally independent normalization steps and quantization steps into an efficient composite operator. Based on the operator fusion strategy, it can reduce the number of unnecessary data transmissions between the computing unit (such as the GPU core) and the memory, and reduce the number of calls to the GPU computing core by the system, thereby accelerating the reasoning process of the entire large language model.

[0050] It should be noted that the execution subject of the data quantification method of the present application may be a computer device with data processing, network communication and program running functions, such as a computer, a server, etc. The following takes a server as an example to illustrate various embodiments of the data quantification method of the present application.

[0051] Based on this, the present application embodiment provides a data quantification method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data quantization method of the present application.

[0052] In this embodiment, the data quantization method includes steps S10 to S30:

[0053] Step S10, obtaining a numerical matrix to be quantized, and dividing the numerical matrix to be quantized into parallel processing data groups;

[0054] It should be noted that the to-be-quantized numerical matrix includes multiple numerical values ​​to be quantized, and the numerical values ​​to be quantized are usually floating point numbers. For example, the to-be-quantized numerical matrix includes multiple high-precision floating point numbers. The parallel processing data group includes a group of to-be-quantized numerical values, and the to-be-quantized numerical values ​​in each parallel processing data group need to be processed in parallel.

[0055] In this embodiment, since there are a large number of values ​​to be quantized, it is necessary to reduce the data processing time by parallel processing. Therefore, for each value to be quantized, each value to be quantized can be evenly divided into a plurality of parallel processing data groups.

[0056] In a feasible implementation manner, the above step S10 further includes:

[0057] Step S101, converting the to-be-quantized numerical matrix into a one-dimensional array, wherein the to-be-quantized numerical matrix is ​​a three-dimensional or four-dimensional matrix;

[0058] In this embodiment, for a three-dimensional or four-dimensional matrix of numerical values ​​to be quantized, each numerical value to be quantized in the matrix of numerical values ​​to be quantized can be read out in a preset order, so that a one-dimensional array including all numerical values ​​to be quantized in the matrix of numerical values ​​to be quantized can be obtained.

[0059] Step S102, based on a preset number of parallel processing units, combining the values ​​to be quantized in the one-dimensional array to obtain parallel processing data groups, wherein the number of the parallel processing data groups is the same as the number of the parallel processing units.

[0060] It is understandable that the parallel processing unit is used to process the to-be-quantized values ​​in the parallel processing data group. Therefore, it is necessary to evenly divide the to-be-quantized values ​​in the one-dimensional array into multiple parallel processing data groups, so that the parallel processing unit and the parallel processing data groups can be processed one-to-one.

[0061] Exemplarily, when combining the various values ​​to be quantized in the one-dimensional array, the number of values ​​to be quantized in each parallel processing unit (hereinafter referred to as the "target number") can be determined by the total number of values ​​to be quantized and the number of parallel processing units. Then, the target number of values ​​to be quantized are respectively extracted from the one-dimensional array by random sampling, and the target number of values ​​to be quantized is used as a parallel processing data group. Alternatively, it is also possible to divide according to the sequence number in the one-dimensional value to obtain each parallel processing data group. The present application does not limit the specific method of group division.

[0062] Step S20, normalizing each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized;

[0063] It can be understood that normalization refers to mapping data to the interval of [-1, 1] or [0, 1]. The intermediate value to be quantized refers to the value obtained after normalization.

[0064] In this embodiment, after determining each parallel processing data group, each parallel processing data group can be allocated to each parallel processing unit, and then the data to be quantized in each parallel processing data group can be normalized based on each parallel processing unit to obtain each intermediate value to be quantized.

[0065] In a feasible implementation manner, each parallel processing data group includes various values ​​to be quantized, and the above step S20 includes:

[0066] Step S201, based on each preset parallel processing unit, respectively calculating the square value of each of the values ​​to be quantized;

[0067] Step S202, calculating the root mean square value corresponding to each of the square values, and taking the ratio between each of the values ​​to be quantized and the root mean square value as the intermediate value to be quantized.

[0068] It is understandable that each parallel processing unit can calculate the square value of the numerical value to be quantized in the corresponding parallel processing data group, and then the sum of the square values ​​calculated by all parallel processing units can be calculated through the reduction operation. Then, the ratio between the sum and the number of numerical values ​​to be quantized can be calculated, and then the square root of the ratio can be used as the root mean square value. Then, the ratio between the numerical value to be quantized and the root mean square value can be used as the normalized value. That is, the ratio between the numerical value to be quantized and the root mean square value is the intermediate numerical value to be quantized.

[0069] Step S30, determining a quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, and scaling the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

[0070] It should be noted that the quantization scaling factor is a coefficient for quantizing a numerical value.

[0071] It can be understood that in this embodiment, after normalization, the normalized data will be directly quantized, and the normalized results will not be stored, thereby avoiding the additional storage and reading overhead of the intermediate results (intermediate values ​​to be quantized), thereby significantly reducing the number of memory accesses and calculation delays, and achieving an acceleration effect on the model reasoning process.

[0072] In a feasible implementation manner, the above step S30 includes:

[0073] Step S301, obtaining a quantization level value, and determining a quantization range value based on the maximum absolute value of each of the intermediate values ​​to be quantized;

[0074] Step S302: Calculate a quantization interval based on the quantization range value and the quantization level value, and use the reciprocal of the quantization interval as a quantization scaling factor.

[0075] It should be noted that the quantization level value refers to the number of levels used when mapping continuously changing numerical values ​​to a finite number of discrete numerical values ​​during the quantization process; the quantization range value represents the numerical range that the quantization process needs to cover; and the quantization interval refers to the difference between two adjacent quantization levels.

[0076] For example, when the maximum absolute value of each intermediate value to be quantized is 1, the quantization range is [-1, 1], so the quantization range value is 2. If the quantization level value is 256, the quantization interval (the quantization interval refers to the range from the minimum value to the maximum value divided by the number of quantization levels minus 1) is: 2 / 255. Therefore, the quantization scaling factor is 255 / 2=127.5.

[0077] Then, assuming that a certain intermediate value to be quantized is 0.8, the integer corresponding to 0.8 after quantization is round(0.8*127.5)=102, where round is a rounding operation. Therefore, for a large number of values ​​to be quantized, the quantization process can be accelerated based on the fused operator, thereby improving the speed of model reasoning.

[0078] In a feasible implementation manner, after the above step S30, the method further includes:

[0079] Step S40: input each of the quantized values ​​into a linear layer in a large language model.

[0080] It is understandable that since the quantized low-bit large language model only has a quantization operation before the linear layer, the present application can only be applied to the case where a normalized layer is adjacent to a linear layer. Compared with the traditional method of using quantization as a separate operator, the fusion operator proposed in the present application will identify the specific neural network structure pattern in which a normalized layer and a linear layer are adjacent during the quantization process. Subsequently, the fusion operator proposed in the present application can be used to perform an integrated replacement process on the identified normalized layer and the quantized layer that follows it. This can reduce computational redundancy and optimize data flow, thereby significantly improving the execution speed of the model in the inference phase.

[0081] Please refer to Figure 2 , Figure 2 This is a schematic diagram of operator fusion of an embodiment of the data quantization method of this application. Figure 2 In the , the normalization calculation core and the quantization calculation core are merged, which reduces the total number of calculation core calls by one. At the same time, the activation data between normalization and quantization does not need to be stored in the memory and then read into the next calculation core, but can be directly reused in the calculation unit to improve the speed of model inference.

[0082] In an embodiment of the present application, by obtaining a matrix of values ​​to be quantized, and dividing the matrix of values ​​to be quantized into various parallel processing data groups; then normalizing each of the parallel processing data groups based on preset parallel processing units to obtain each intermediate value to be quantized; finally determining a quantization scaling factor based on the maximum value of each intermediate value to be quantized, and scaling each intermediate value to be quantized based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute value of each intermediate value to be quantized, and the originally independent normalization step and quantization step can be integrated into an efficient composite operator. Then, this operator fusion strategy can reduce the number of unnecessary data transmissions between the computing unit (such as the GPU core) and the memory, and reduce the number of calls to the GPU computing core by the system, thereby accelerating the reasoning process of the entire large language model.

[0083] Furthermore, based on the above-mentioned first embodiment of the data quantization method of the present application, a second embodiment of the data quantization method of the present application is proposed.

[0084] In this embodiment, before the above step S30, the method further includes:

[0085] Step S50, determining the absolute value corresponding to each of the intermediate values ​​to be quantized, and comparing each of the absolute values ​​with a preset threshold value;

[0086] Step S60, deleting a target value from each of the intermediate values ​​to be quantized, wherein the absolute value corresponding to the target value is greater than or equal to the threshold;

[0087] It is understandable that after normalization, discrete values ​​will affect the accuracy of model reasoning, so it is necessary to delete the discrete values ​​in each intermediate value to be quantized.

[0088] Based on this, the above step S30 further includes:

[0089] Step S303: after deleting the target value, determining a quantization scaling factor based on a maximum value among the remaining intermediate values ​​to be quantized.

[0090] In this embodiment, after the discrete values ​​are deleted, intermediate values ​​to be quantized with higher quality can be obtained, and thus, a more accurate quantization scaling factor can be further obtained.

[0091] Furthermore, based on the above-mentioned various embodiments of the data quantization method of the present application, a specific embodiment of the data quantization method of the present application is proposed.

[0092] Please refer to Figure 3 , Figure 3 This is a processing flow diagram of a specific embodiment of the data quantification method of this application. Figure 3 In , the operator that combines normalization and quantization first performs a preprocessing step: it flattens the input floating-point data sequence into a one-dimensional array and efficiently distributes this data evenly to each parallel processing unit (such as the thread bundle in the GPU). Then, the threads in each thread bundle calculate the square of their assigned values ​​and aggregate these square values ​​through a specification operation (such as parallel summation) to calculate the sum. Based on this sum, the RMS value is further calculated and normalized to ensure that the data is within a suitable numerical range. After this stage, the intermediate activation values ​​are still retained inside each thread computing unit.

[0093] Then, in order to improve data quality and reduce the impact of anomalies, the operator implements an outlier removal mechanism in each thread. For the calculation of outliers, the method of setting a threshold is often adopted. Data with an absolute value greater than or equal to the threshold is marked as an outlier and does not participate in the quantization operation, while data with an absolute value less than the threshold participates in the quantization operation. Existing quantization schemes allow outliers to be dynamically calculated before quantization or calculated outside the reasoning process. Unlike this, the scheme in this application requires that the calculation of outliers must be completed outside the reasoning process. After the outlier removal is completed, the remaining values ​​are again reduced to find the maximum value of all non-outlier data.

[0094] Then, a quantization scaling factor is calculated based on the maximum value, which is used to map the processed floating-point data to the integer space while maintaining the dynamic range and precision of the data as much as possible. Finally, the calculated quantization scaling factor is applied to the remaining data, and the data type is converted by the quantization unit to generate the final integer output, so that the present application can efficiently complete the quantization process.

[0095] In addition, Figure 2 In this application, the data allocation strategy of the normalization process and the quantization process and the segmentation dimension of the parallel processing are aligned and optimized to ensure that the same thread only needs to focus on a single, continuous data stream during the processing process, thereby reducing the communication requirements and overhead between threads.

[0096] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data quantification method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0097] This application also provides a data quantization system, please refer to Figure 4 , the data quantization system comprises:

[0098] A grouping module 10 is used to obtain a numerical matrix to be quantized and divide the numerical matrix to be quantized into parallel processing data groups;

[0099] A first processing module 20 is used to normalize each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized;

[0100] The second processing module 30 determines a quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, and scales the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

[0101] In one embodiment, the data quantization system further includes:

[0102] A numerical comparison module, used to determine the absolute value corresponding to each of the intermediate values ​​to be quantized, and compare each of the absolute values ​​with a preset threshold value;

[0103] A data elimination module, used to delete a target value from each of the intermediate values ​​to be quantized, wherein the absolute value corresponding to the target value is greater than or equal to the threshold value;

[0104] Based on this, the second processing module 30 is further used for:

[0105] After deleting the target value, a quantization scaling factor is determined based on a maximum value among the remaining intermediate values ​​to be quantized.

[0106] In one embodiment, the grouping module 10 is further configured to:

[0107] Converting the to-be-quantized numerical matrix into a one-dimensional array, wherein the to-be-quantized numerical matrix is ​​a three-dimensional or four-dimensional matrix;

[0108] Based on the preset number of parallel processing units, the values ​​to be quantized in the one-dimensional array are combined to obtain parallel processing data groups, wherein the number of the parallel processing data groups is the same as the number of the parallel processing units.

[0109] In one embodiment, each parallel processing data group includes various values ​​to be quantized, and the first processing module 20 is further used to:

[0110] Based on each preset parallel processing unit, respectively calculating the square value of each of the to-be-quantized values;

[0111] The root mean square value corresponding to each of the square values ​​is calculated, and the ratio between each of the values ​​to be quantized and the root mean square value is used as the intermediate value to be quantized.

[0112] In one embodiment, the second processing module 30 is further used for:

[0113] Acquire a quantization level value, and determine a quantization range value based on the maximum absolute value of each of the intermediate values ​​to be quantized;

[0114] A quantization interval is calculated based on the quantization range value and the quantization level value, and the inverse of the quantization interval is used as a quantization scaling factor.

[0115] In one embodiment, the data quantization system further includes:

[0116] The switching module is used to input the quantized values ​​into the linear layer in the large language model.

[0117] The data quantization system provided by the present application adopts the data quantization method in the above embodiment, which can solve the technical problem of how to improve the speed of reasoning of a large language model. Compared with the prior art, the beneficial effects of the data quantization system provided by the present application are the same as the beneficial effects of the data quantization method provided by the above embodiment, and other technical features in the data quantization system are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0118] The present application provides a computer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data quantization method in the above-mentioned embodiment 1.

[0119] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer device suitable for implementing an embodiment of the present application. Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 5 As shown, the computer device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the computer device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the computer device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a computer device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0121] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0122] The computer device provided by the present application adopts the data quantization method in the above embodiment to solve the technical problem of how to improve the speed of reasoning of a large language model. Compared with the prior art, the beneficial effects of the computer device provided by the present application are the same as the beneficial effects of the data quantization method provided by the above embodiment, and the other technical features in the computer device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0123] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0124] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0125] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the data quantization method in the above-mentioned embodiment.

[0126] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0127] The computer-readable storage medium may be included in the computer device; or may exist independently without being installed in the computer device.

[0128] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a computer device, the computer device: obtains a numerical matrix to be quantized, and divides the numerical matrix to be quantized into various parallel processing data groups; normalizes each of the parallel processing data groups based on preset parallel processing units to obtain various intermediate numerical values ​​to be quantized; determines a quantization scaling factor based on the maximum value among each of the intermediate numerical values ​​to be quantized, and scales each of the intermediate numerical values ​​to be quantized based on the quantization scaling factor to obtain each quantized completed numerical value, wherein the maximum value is the maximum value of the absolute values ​​among each of the intermediate numerical values ​​to be quantized.

[0129] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0132] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data quantization method, and can solve the technical problem of how to improve the speed of large language models during reasoning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data quantization method provided in the above-mentioned embodiment, and will not be repeated here.

[0133] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data quantization method when executed by a processor.

[0134] The computer program product provided by the present application can solve the technical problem of how to improve the speed of reasoning of a large language model. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data quantization method provided by the above embodiment, which will not be repeated here.

[0135] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data quantization method, characterized in that: The data quantification method comprises: Acquire a numerical matrix to be quantized, and divide the numerical matrix to be quantized into parallel processing data groups; Normalizing each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized; A quantization scaling factor is determined based on the maximum value among the intermediate values ​​to be quantized, and each intermediate value to be quantized is scaled based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

2. The data quantization method according to claim 1, characterized in that: Before the step of determining the quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, the method further includes: Determine the absolute value corresponding to each of the intermediate values ​​to be quantized, and compare each of the absolute values ​​with a preset threshold value; Deleting a target value from each of the intermediate values ​​to be quantized, wherein an absolute value corresponding to the target value is greater than or equal to the threshold; The step of determining the quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized comprises: After deleting the target value, a quantization scaling factor is determined based on a maximum value among the remaining intermediate values ​​to be quantized.

3. The data quantization method according to claim 1, characterized in that: The step of dividing the to-be-quantized numerical matrix into respective parallel processing data groups comprises: Converting the to-be-quantized numerical matrix into a one-dimensional array, wherein the to-be-quantized numerical matrix is ​​a three-dimensional or four-dimensional matrix; Based on the preset number of parallel processing units, the values ​​to be quantized in the one-dimensional array are combined to obtain parallel processing data groups, wherein the number of the parallel processing data groups is the same as the number of the parallel processing units.

4. The data quantization method according to claim 3, characterized in that: Each parallel processing data group includes various values ​​to be quantified; The step of normalizing each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized includes: Based on each preset parallel processing unit, respectively calculating the square value of each of the to-be-quantized values; The root mean square value corresponding to each of the square values ​​is calculated, and the ratio between each of the values ​​to be quantized and the root mean square value is used as the intermediate value to be quantized.

5. The data quantization method according to claim 1, characterized in that: The step of determining the quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized comprises: Acquire a quantization level value, and determine a quantization range value based on the maximum absolute value of each of the intermediate values ​​to be quantized; A quantization interval is calculated based on the quantization range value and the quantization level value, and the inverse of the quantization interval is used as a quantization scaling factor.

6. The data quantization method according to claim 1, characterized in that: After the step of scaling each of the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, the method further includes: Each of the quantized values ​​is input into a linear layer in a large language model.

7. A data quantization system, characterized in that: The data quantification system comprises: A grouping module, used for obtaining a numerical matrix to be quantized, and dividing the numerical matrix to be quantized into various parallel processing data groups; A first processing module, configured to normalize each of the parallel processing data groups based on each preset parallel processing unit to obtain each intermediate value to be quantized; The second processing module determines a quantization scaling factor based on the maximum value among the intermediate values ​​to be quantized, and scales the intermediate values ​​to be quantized based on the quantization scaling factor to obtain each quantized value, wherein the maximum value is the maximum value of the absolute values ​​among the intermediate values ​​to be quantized.

8. A computer device, characterized in that: The computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data quantization method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data quantization method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data quantization method according to any one of claims 1 to 6 are implemented.

Images