A data processing method, apparatus and device

Abstract

The application provides a data processing method, device and equipment. The data processing method provided by the application comprises: for each weight coefficient in a weight matrix, taking the weight coefficient as to-be-processed data, constructing FP16 data corresponding to the weight coefficient based on a pre-set FP16 data construction rule; for a zero point deviation of each weight coefficient, taking the zero point deviation as to-be-processed data, constructing FP16 data corresponding to the zero point deviation based on the FP16 data construction rule; and using a pre-determined target matrix multiplication operation formula corresponding to the FP16 data construction rule, calculating a multiplication operation result of an input matrix and the weight matrix according to FP16 data corresponding to each weight coefficient of the input matrix and the weight matrix, FP16 data corresponding to the zero point deviation of the weight coefficient, and a scaling factor of the weight coefficient. The data processing method, device and equipment provided by the application can improve the running speed of a large model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, and device. Background Technology

[0002] In recent years, with the popularization of the internet and the development of information technology, large-scale text data has become more readily available, providing a sufficient data foundation for training large models. Large models have achieved significant breakthroughs in the field of natural language processing, excelling not only in tasks such as text generation, question answering systems, and language translation, but also showing broad potential in other application areas, such as medical diagnosis, financial forecasting, and virtual assistants. Due to their powerful capabilities, the importance of large models is increasingly prominent.

[0003] However, despite the significant performance improvements offered by large models, several challenges remain in practical inference. One of the most significant bottlenecks is the limitation of memory bandwidth. Large models typically require a large number of parameters and computational resources, leading to frequent data reads and writes during inference, thus placing enormous demands on memory bandwidth.

[0004] To address the memory bandwidth bottleneck, low-bit quantization has become a highly effective method. Low-bit quantization reduces the storage and transmission overhead of parameters by decreasing their bit width. Currently, the most common approach is 4-bit quantization, which converts the original 32-bit floating-point parameters into 4-bit fixed-point parameters. This quantization can compress memory bandwidth by at least 8 times, significantly reducing data transmission overhead during inference and improving inference speed.

[0005] While 4-bit quantization performs well in terms of performance and bandwidth compression, many hardware platforms do not support 4-bit, making it impossible to deploy large models on these platforms. Summary of the Invention

[0006] In view of this, this application provides a data processing method, apparatus, and device to enable the deployment of large models with 4-bit quantization on relevant hardware platforms and improve the running speed of large models.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] The first aspect of this application provides a data processing method, the method comprising:

[0009] For each weight coefficient in the weight matrix, the weight coefficient is used as the data to be processed, and FP16 data corresponding to the weight coefficient is constructed based on the pre-defined FP16 data construction rules; wherein, the FP16 data construction rules indicate that FP16 data corresponding to the data to be processed is constructed based on the data to be processed with data type int4.

[0010] For each weight coefficient's zero-point deviation, the zero-point deviation is used as the data to be processed, and the FP16 data corresponding to the zero-point deviation is constructed based on the FP16 data construction rules.

[0011] Using a predetermined target matrix multiplication formula corresponding to the FP16 data construction rules, the multiplication result between the input matrix and the weight matrix is ​​calculated based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient. The target matrix multiplication formula is derived from the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the predetermined mapping relationship corresponding to the FP16 data construction rules. This mapping relationship represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed.

[0012] A second aspect of this application provides a data processing apparatus, the apparatus comprising: a construction module and a computing module; wherein...

[0013] The construction module is used to construct FP16 data corresponding to each weight coefficient in the weight matrix, taking the weight coefficient as the data to be processed, based on a pre-defined FP16 data construction rule; wherein, the FP16 data construction rule indicates that the FP16 data corresponding to the data to be processed is constructed based on the data to be processed with the data type int4.

[0014] The construction module is also used to take the zero-point deviation of each weight coefficient as the data to be processed, and construct the FP16 data corresponding to the zero-point deviation based on the FP16 data construction rules.

[0015] The calculation module is used to calculate the multiplication result of the input matrix and the weight matrix using a predetermined target matrix multiplication formula corresponding to the FP16 data construction rules, based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient; wherein, the target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the mapping relationship corresponding to the predetermined FP16 data construction rules; the mapping relationship represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed.

[0016] A third aspect of this application provides a data processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0017] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0018] The data processing method, apparatus, and device provided in this application pre-define FP16 data construction rules and the corresponding mapping relationships. Based on the mapping relationships, the quantization formulas, and the initial matrix multiplication formulas, the target matrix multiplication formulas corresponding to the FP16 data construction rules are derived in advance. Thus, during the inference stage, FP16 data corresponding to the weight coefficients and FP16 data corresponding to the zero-point deviations of the weight coefficients can be directly constructed based on the pre-defined FP16 data construction rules. Then, the multiplication result of the input matrix and the weight matrix can be directly calculated using the pre-determined target matrix multiplication formula, according to the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviations of the weight coefficients, and the scaling factor of the weight coefficients. In this way, firstly, based on the FP16 construction rules, the FP16 data corresponding to the weight coefficients and the FP16 data corresponding to the zero-point deviation of the weight coefficients can be directly obtained. This allows hardware platforms supporting 8-bit, 16-bit, or 32-bit to process FP16 data, enabling the deployment of large models with 4-bit quantization on hardware platforms that do not support 4-bit. Secondly, during data type conversion, inverse quantization is not required; data type conversion is performed directly based on the pre-defined FP16 data construction rules, resulting in faster data type conversion speed, exceeding bandwidth speed and improving computational performance. Thirdly, after data type conversion based on the pre-defined FP16 data construction rules, the multiplication result can be directly calculated using the pre-determined target matrix multiplication formula, improving the multiplication speed and minimizing the memory bandwidth requirements of large models. Attached Figure Description

[0019] Figure 1 A flowchart of an embodiment of the data processing method provided in this application;

[0020] Figure 2 A flowchart illustrating a second embodiment of the data processing method of this application, as shown in an exemplary embodiment;

[0021] Figure 3 A flowchart illustrating a third embodiment of the data processing method of this application, as shown in an exemplary embodiment;

[0022] Figure 4 A flowchart of Embodiment 4 of the data processing method provided in this application;

[0023] Figure 5 A hardware structure diagram of the data processing device in which the data processing device provided in this application is located;

[0024] Figure 6 This is a schematic diagram of the structure of a data processing device according to an embodiment of this application. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0026] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0028] This application provides a data processing method, apparatus, and device to enable the deployment of large models with 4-bit quantization on relevant hardware platforms and improve the running speed of large models.

[0029] The data processing method, apparatus, and device provided in this application pre-define FP16 data construction rules and the corresponding mapping relationships. Based on the mapping relationships, the quantization formulas, and the initial matrix multiplication formulas, the target matrix multiplication formulas corresponding to the FP16 data construction rules are derived in advance. Thus, during the inference stage, FP16 data corresponding to the weight coefficients and FP16 data corresponding to the zero-point deviations of the weight coefficients can be directly constructed based on the pre-defined FP16 data construction rules. Then, the multiplication result of the input matrix and the weight matrix can be directly calculated using the pre-determined target matrix multiplication formula, according to the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviations of the weight coefficients, and the scaling factor of the weight coefficients. In this way, firstly, based on the FP16 construction rules, the FP16 data corresponding to the weight coefficients and the FP16 data corresponding to the zero-point deviation of the weight coefficients can be directly obtained. This allows hardware platforms supporting 8-bit, 16-bit, or 32-bit to process FP16 data, enabling the deployment of large models with 4-bit quantization on hardware platforms that do not support 4-bit. Secondly, during data type conversion, inverse quantization is not required; data type conversion is performed directly based on the pre-defined FP16 data construction rules, resulting in faster data type conversion speed, exceeding bandwidth speed and improving computational performance. Thirdly, after data type conversion based on the pre-defined FP16 data construction rules, the multiplication result can be directly calculated using the pre-determined target matrix multiplication formula, improving the multiplication speed and minimizing the memory bandwidth requirements of large models.

[0030] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0031] Figure 1 A flowchart illustrating an embodiment of the data processing method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0032] S101. For each weight coefficient in the weight matrix, the weight coefficient is taken as the data to be processed, and the FP16 data corresponding to the weight coefficient is constructed based on the pre-set FP16 data construction rules; wherein, the FP16 data construction rules indicate that the FP16 data corresponding to the data to be processed is constructed based on the data to be processed with data type int4.

[0033] Specifically, the weight matrix refers to the weight coefficient matrix in a neural network model, used to transform the data to be processed into output data. Each weight coefficient is an element in the weight matrix, representing the degree of influence of each data point on the output data. In essence, the weight coefficients determine how the neural network processes the data.

[0034] It should be noted that the weight matrix refers to the quantized weight matrix, and the data type of each weight coefficient in the weight matrix is ​​int4. Understandably, during the training phase, because floating-point numbers offer higher precision and flexibility, allowing the neural network model to better adapt to the training data, the data type of each weight coefficient in the weight matrix is ​​floating-point during neural network training. Furthermore, during the inference phase, to reduce computational costs and the storage space of the neural network model, the data type of the weight coefficients is quantized, for example, by converting floating-point values ​​to fixed-point values, thereby reducing computational and storage requirements. In this application, 4-bit quantization is performed, quantizing the original floating-point parameters into fixed-point parameters with only 4 bits, and the data type of each weight coefficient in the weight matrix is ​​int4.

[0035] Furthermore, it is understandable that since most hardware platforms do not support quantization of data of type int4, but mainly support quantization of data of type int8 or FP16, in order to deploy a large model with 4-bit quantization on a hardware platform that does not support 4-bit, it is necessary to convert data of type int4 to data of type FP16. However, if the data type conversion speed is lower than the bandwidth speed, the data type conversion will become a performance bottleneck, defeating the purpose of 4-bit quantization to compress bandwidth. In this application, data type conversion is performed based on pre-defined FP16 data construction rules. Since it only involves the data filling process, data type conversion can be performed efficiently, minimizing the memory bandwidth requirements of large models.

[0036] Specifically, the FP16 data construction rule instructs the construction of FP16 data corresponding to the data to be processed based on the data to be processed with data type int4.

[0037] Data to be processed with data type int4 refers to data represented using 4-bit integers, while FP16 data refers to data represented using 16-bit floating-point numbers.

[0038] Specifically, it can be understood that the first bit of FP16 data represents the sign bit, used to indicate whether the number is positive or negative; the second to sixth bits represent the exponent bits, used to indicate the number's exponent; and the lower 10 bits represent the mantissa bits, used to indicate the fractional part of the number.

[0039] The FP16 data construction rule instructs that data to be processed, which is of data type int4, be converted into FP16 data of data type FP16. Specifically, the FP16 data construction rule is set according to actual needs, and this embodiment does not impose any limitations on it.

[0040] Optionally, in one possible implementation, the FP16 data construction rule is used to instruct the data to be processed to be filled into four consecutive empty spaces in a pre-defined 16-bit array to construct data of type FP16, and the constructed data is identified as the FP16 data corresponding to the data to be processed. The values ​​of the bits other than the four consecutive empty spaces in the 16-bit array are preset values.

[0041] Referring to the preceding description, the data to be processed is represented using a 4-bit integer. For example, for an unsigned int4, the range is 0 to 15, where the binary representation of 0 is 0000, the binary representation of 1 is 0001, ..., and the binary representation of 15 is 1111. That is, the data to be processed contains a total of 4 bits. Therefore, when constructing FP16 data based on the data to be processed, the data to be processed can be directly filled into four consecutive empty bits in a pre-defined 16-bit array to obtain data of data type FP16.

[0042] Specifically, the pre-defined 16-bit number contains 16 bits, including four consecutive empty bits, and the values ​​of the bits other than the four consecutive empty bits are preset values ​​(0 or 1). In this way, by filling the data to be processed into the four consecutive empty bits in the pre-defined 16-bit number, a data of data type FP16 can be obtained.

[0043] It should be noted that the specified four consecutive empty spaces are set according to actual needs, and are not limited in this embodiment. Optionally, in one possible implementation, the specified four consecutive empty spaces are four consecutive bits in the lower 10 bits.

[0044] Furthermore, the values ​​of the 16-bit number other than the four consecutive empty bits are preset values. The specific values ​​of the other bits (whether a particular bit is 0 or 1) are set according to actual needs, and are not limited in this embodiment.

[0045] The following example illustrates the concept of "specifying four consecutive empty positions as the 7th to 10th positions, and pre-setting the 16-bit number as 001111****00000". In this case, for example, when the data to be processed is 0001, the obtained FP16 data is 0011110001000000.

[0046] It should be noted that when processing data based on int4 data to obtain the corresponding FP16 data, if the data conversion speed is lower than the bandwidth speed, the data conversion will become a performance bottleneck. That is, the data conversion speed will seriously affect the calculation speed, thus negating the significance of quantization compression bandwidth.

[0047] The method provided in this application, based on the FP16 data construction rules and data filling, can efficiently and quickly convert data to be processed with data type int4 into FP16 data with data type FP16. This can shorten the data conversion time, improve the calculation speed, and maintain high calculation accuracy.

[0048] S102. For the zero-point deviation of each weight coefficient, take the zero-point deviation as the data to be processed, and construct the FP16 data corresponding to the zero-point deviation based on the FP16 data construction rules.

[0049] Specifically, as described above, during weight quantization, floating-point values ​​need to be converted to fixed-point values. Zero-point offset represents the actual value corresponding to zero in the fixed-point number. In weight quantization, zero-point offset is typically used to map the weight range to the range represented by the fixed-point number.

[0050] In practice, the zero-point deviation of each weight coefficient can be obtained. Furthermore, as mentioned earlier, the zero-point deviation of each weight coefficient can be used as the data to be processed. According to the above FP16 data construction rules, the data to be processed is filled into four consecutive empty spaces in a pre-set 16-digit number to obtain the FP16 data corresponding to the zero-point deviation.

[0051] S103. Using a predetermined target matrix multiplication formula corresponding to the FP16 data construction rule, calculate the multiplication result between the input matrix and the weight matrix based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient; wherein, the target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the mapping relationship corresponding to the predetermined FP16 data construction rule; the mapping relationship represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed.

[0052] It should be noted that the input matrix is ​​the data to be processed in a large model. The processing result of the input matrix is ​​obtained by calculating the multiplication result of the input matrix and the weight matrix using the target matrix multiplication formula. The target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the mapping relationship corresponding to the pre-determined FP16 data construction rules.

[0053] Specifically, the initial matrix multiplication formula corresponding to the weight matrix represents the mapping relationship between the input matrix and the output matrix. It is used to process the input matrix based on the weight matrix to obtain the output matrix, as shown below:

[0054]

[0055] in, Represents the input matrix; Represents the weight matrix; Let m and k represent the dimensions of the input matrix, k and n represent the dimensions of the weight matrix, and k and n represent the dimensions of the output matrix.

[0056] Furthermore, the quantization formula corresponding to the weight matrix refers to the formula used when quantizing the weight matrix. Referring to the preceding description, the quantization of the weight matrix typically involves converting floating-point values ​​to fixed-point values. The quantization formula characterizes the mapping relationship between floating-point and fixed-point values. Specifically, the quantization formula is as follows:

[0057] W FP16 =(W int4 —Z int4 )×scale FP16

[0058] Among them, W in4 Z represents the weight coefficients in the quantized weight matrix, and its data type is Int4; int4 This represents the zero-point deviation of the weight coefficients in the quantized weight matrix; scale FP16 W represents the scaling factor of the weight coefficients in the quantized weight matrix. FP16 This refers to the weight coefficients in the weight matrix before quantization, and the data type is FP16. It should be noted that, as described above, the weight matrix referred to in this article is the weight matrix after quantization.

[0059] Furthermore, the mapping relationship corresponding to the FP16 data construction rule represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed. Referring to the previous description, the data to be processed is data represented using a 4-bit integer, for example, 0001, which represents the actual value 1. Further, FP16 data refers to data represented using a 16-bit floating-point number. The first bit represents the sign bit, used to indicate the positive or negative sign of the number; the second to sixth bits represent the exponent bits, used to represent the exponent of the number; and the lower 10 bits represent the mantissa bits, used to represent the fractional part of the number. For example, when the data to be processed is 0001, its corresponding FP16 data is 010000 0001000000, which represents the actual value 1.00 + 1 / 2^5.

[0060] It should be noted that the mapping relationship corresponding to the FP16 data construction rule is specific to the FP16 data construction rule itself. Different FP16 data construction rules result in different mapping relationships. For example, referring to the example above, when the FP16 data construction rule is "fill the four consecutive empty spaces in 001111****00000 to obtain the corresponding FP16 data," the mapping relationship corresponding to this FP16 data construction rule is:

[0061]

[0062] Where, x int4 This represents the actual numerical value of the data to be processed.

[0063] y FP16 This represents the actual numerical value of the FP16 data corresponding to the data to be processed.

[0064] Specifically, based on the preceding introduction, the initial matrix multiplication formula corresponding to the weight matrix can be transformed according to the quantization formula corresponding to the weight matrix and the mapping relationship corresponding to the pre-determined FP16 data construction rules, so as to derive the target matrix multiplication formula.

[0065] Optional, Figure 2 This is a flowchart illustrating a second embodiment of the data processing method, as shown in an exemplary embodiment of this application. Please refer to... Figure 2 The method provided in this embodiment, based on the above embodiments, includes the following steps for determining the target matrix multiplication formula:

[0066] S201. Based on the mapping relationship, obtain the first mapping relationship between the weight coefficient and the FP16 data corresponding to the weight coefficient, and the second mapping relationship between the zero-point deviation of the weight coefficient and the FP16 data corresponding to the zero-point deviation of the weight coefficient.

[0067] Specifically, as described above, the FP16 data corresponding to the weight coefficients is obtained by constructing rules based on the aforementioned FP16 data, using the weight coefficients as the data to be processed. Therefore, by substituting the weight coefficients and their corresponding FP16 data into the mapping relationship described above, we can obtain the first mapping relationship between the weight coefficients and their corresponding FP16 data, as follows:

[0068] W1 FP16 =W int4 / 64+1.0

[0069] Among them, W int4 W1 represents the weighting coefficient, with data type int4; FP16 This represents the FP16 data corresponding to the weighting coefficients.

[0070] Similarly, the FP16 data corresponding to the zero-point deviation of the weighting coefficients is obtained by using the zero-point deviation as the data to be processed and constructing rules based on the aforementioned FP16 data. Therefore, by substituting the zero-point deviation and the corresponding FP16 data into the above mapping relationship, a second mapping relationship between the zero-point deviation and the corresponding FP16 data can be obtained. The second mapping relationship is as follows:

[0071] Z1 FP16 =Z int4 / 64+1.0

[0072] Among them, Z int4 Z1 represents the zero-point deviation of the weighting coefficients, and its data type is int4; FP16 This represents the FP16 data corresponding to the zero-point deviation of the weighting coefficients.

[0073] S202. Substitute the first mapping relationship and the second mapping relationship into the quantization formula corresponding to the weight matrix to obtain the intermediate formula.

[0074] Specifically, as described above, the quantification formula is as follows:

[0075] W FP16 =(W int4 -Z int4 )×scale FP16

[0076] In this step, the first and second mapping relationships are substituted into the quantization formula. The specific substitution process is as follows:

[0077] W FP16 =((W1) FP16 -1.0)×64-(Z1 FP16 -1.0)×64)×scale FP16

[0078] =(W1) FP16 -Z1 FP16 )×64×scale FP16

[0079] The final intermediate formula is as follows:

[0080] W FP16 =(W1) FP16 -Z1 FP16 )×64×scale FP16

[0081] Among them, W1 FP16 Z1 represents the FP16 data corresponding to the weighting coefficients. FP16 This represents the FP16 data corresponding to the zero-point deviation of the weighting coefficients; scale FP16 W represents the scaling factor of the weighting coefficients. FP16 This represents the weight coefficients in the weight matrix before quantization.

[0082] S203. Substitute the intermediate formula into the initial matrix multiplication formula corresponding to the weight matrix to obtain the target matrix multiplication formula.

[0083] Referring to the preceding description, the initial matrix multiplication formula is as follows:

[0084]

[0085] Understandably, in one possible implementation, quantization is performed by scaling based on columns, with each column's weight coefficient having the same scaling factor and zero-point deviation. In this step, the intermediate formula is substituted into the initial matrix multiplication formula to obtain the target matrix multiplication formula. The specific substitution process is as follows:

[0086]

[0087] The final formula for multiplying the target matrix is ​​as follows:

[0088]

[0089] As can be seen from the above target matrix multiplication formula, when calculating the result of the multiplication of the input matrix and the weight matrix, it is only necessary to obtain the FP16 data corresponding to each weight coefficient in the weight matrix and the FP16 data corresponding to the zero-point deviation of each weight coefficient through the above method. Then, based on the above target matrix multiplication formula, the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient are substituted into the target matrix multiplication formula to obtain the result of the multiplication of the input matrix and the weight matrix.

[0090] In practice, the cube hardware unit can be used for execution. The process is executed using the vector processing unit (VU). The process. Here, "cube" refers to the hardware unit used to perform matrix operations, such as a tensor processing unit (TPU) or a graphics processing unit (GPU).

[0091] Optionally, in one possible implementation, 64×scale can be used. F16 Noted as Scale1 F16 The above formula can be converted to

[0092]

[0093] In practical implementation, before performing matrix operations, it can be based on scale1. Fp16 Scale1 was calculated in advance F16 Save it so that you can directly substitute it into matrix operations later. To further improve computational efficiency, calculations are performed.

[0094] The data processing method provided in this embodiment pre-sets FP16 data construction rules and the corresponding mapping relationships. Based on the mapping relationships, the quantization formulas, and the initial matrix multiplication formulas, the method pre-derives the target matrix multiplication formula corresponding to the FP16 data construction rules. Thus, during the inference stage, it can directly construct the FP16 data corresponding to the weight coefficients and the FP16 data corresponding to the zero-point deviations of the weight coefficients based on the pre-set FP16 data construction rules. Then, it can directly use the pre-determined target matrix multiplication formula to calculate the multiplication result of the input matrix and the weight matrix based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviations of the weight coefficients, and the scaling factor of the weight coefficients. In this way, firstly, based on the FP16 construction rules, the FP16 data corresponding to the weight coefficients and the FP16 data corresponding to the zero-point deviation of the weight coefficients can be directly obtained. This allows hardware platforms supporting 8-bit, 16-bit, or 32-bit to process FP16 data, enabling the deployment of large models with 4-bit quantization on hardware platforms that do not support 4-bit. Secondly, during data type conversion, inverse quantization is not required; data type conversion is performed directly based on the pre-defined FP16 data construction rules, resulting in faster data type conversion speed, exceeding bandwidth speed and improving computational performance. Thirdly, after data type conversion based on the pre-defined FP16 data construction rules, the multiplication result can be directly calculated using the pre-determined target matrix multiplication formula, improving the multiplication speed and minimizing the memory bandwidth requirements of large models.

[0095] Figure 3 This is a flowchart illustrating a third embodiment of the data processing method of this application, as shown in an exemplary embodiment. Please refer to... Figure 3 The method provided in this embodiment, based on the above embodiments, includes the following steps for determining the mapping relationship:

[0096] S301. For each data point in the data range represented by the int4 data type, based on the FP16 data construction rule, construct an FP16 data of data type FP16 according to the binary representation of the data point, and obtain the FP16 data corresponding to the data point.

[0097] Specifically, when using the int4 data type to represent unsigned integers, the range of data it represents is from 0 to 15. Referring to the previous introduction, based on the FP16 data construction rules, each data point within the range represented by the int4 data type can be converted to FP16 data.

[0098] In practical implementation, for example, combining the above example, when the FP16 data construction rule is "fill the four consecutive empty spaces in 001111****00000 to obtain the FP16 data corresponding to the data to be processed", in this step, for each data point within 0000 to 1111, based on the FP16 data construction rule, the FP16 data corresponding to each data point is constructed, i.e., 001111. 0000 000000 to 001111 1111 000000.

[0099] S302. For each data point, take the actual value represented by the data point and the actual value represented by the FP16 data corresponding to the data point as a data pair to obtain the data pair corresponding to the data point.

[0100] Specifically, the actual value represented by each data point and the actual value represented by the corresponding FP16 data for each data point are used as data pairs.

[0101] For example, combining the above example, in this step, we can obtain a data pair consisting of the actual value represented by each data point and the actual value represented by the corresponding FP16 data: (0000, 001111) 0000 (000000) to (1111, 001111) 1111 000000).

[0102] S303. Determine the mapping relationship based on the data pairs corresponding to all data points in the data range.

[0103] Specifically, based on the data points in all data pairs, the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the corresponding FP16 data can be determined.

[0104] For example, in conjunction with the above example, Table 1 is a mapping table corresponding to the FP16 data construction rules illustrated in an exemplary embodiment of this application:

[0105] Table 1. Mapping Relationships for FP16 Data Construction Rules

[0106]

[0107]

[0108] The method provided in this embodiment determines the mapping relationship corresponding to the FP16 data construction rules, and then the target matrix multiplication operation formula can be derived based on the mapping relationship. Then, calculation can be performed directly based on the target matrix multiplication operation formula, which can improve the calculation speed and efficiency.

[0109] Figure 4 A flowchart of Embodiment 4 of the data processing method provided in this application. Please refer to... Figure 4 The method provided in this embodiment may include:

[0110] S401. Obtain the weight matrix from the memory; wherein two adjacent weight coefficients in the weight matrix are combined into an 8-bit unsigned integer and stored in the memory.

[0111] Specifically, memory refers to a component used to store data, typically for storing programs, application data, operating systems, and other types of information. In this embodiment, the weight matrix is ​​stored in memory. It should be noted that the weight matrix is ​​encoded before being stored in memory. Optionally, in one possible implementation, when the hardware platform is an 8-bit hardware platform, during the encoding of the weight matrix, adjacent weight coefficients are combined into an 8-bit unsigned integer, which is then stored in memory. This achieves data compression, reduces storage requirements, and improves data transmission efficiency.

[0112] Specifically, when combining two adjacent weight coefficients, one weight coefficient can be used as the high 4 bits and the other weight coefficient as the low 4 bits.

[0113] Specifically, once the weight matrix is ​​stored in memory, during the inference phase, when performing calculations, each 8-bit unsigned integer can be read directly from memory in sequence.

[0114] S402. For each unsigned integer in the weight matrix, split it into two weight coefficients.

[0115] Specifically, when obtaining each unsigned integer, each 8-bit unsigned integer can be split into two 4-bit unsigned integers, which represent two adjacent weight coefficients respectively.

[0116] For example, in one embodiment, an 8-bit unsigned integer read from memory is u8, which can be split into u8_h and u8_l, where u8_h represents the high 4 bits and u8_l represents the low 4 bits.

[0117] S403. For each weight coefficient in the weight matrix, take the weight coefficient as the data to be processed, and construct the FP16 data corresponding to the weight coefficient based on the pre-set FP16 data construction rules; wherein, the FP16 data construction rules indicate that the FP16 data corresponding to the data to be processed is constructed based on the data to be processed with data type int4.

[0118] S404. For the zero-point deviation of each weight coefficient, take the zero-point deviation as the data to be processed, and construct the FP16 data corresponding to the zero-point deviation based on the FP16 data construction rules.

[0119] S405. Using a predetermined target matrix multiplication formula corresponding to the FP16 data construction rule, calculate the multiplication result between the input matrix and the weight matrix based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient; wherein, the target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the mapping relationship corresponding to the predetermined FP16 data construction rule; the mapping relationship represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed.

[0120] Specifically, the implementation process and principle of steps S403-S405 can be found in the description in the above embodiments, and will not be repeated here.

[0121] The method provided in this application, firstly, reduces storage requirements and improves data transmission efficiency by combining two adjacent weight coefficients into an 8-bit unsigned integer. Secondly, by splitting each 8-bit unsigned integer into two weight coefficients, allowing each 8-bit unsigned integer to carry two weight values, the amount of data managed and transmitted is effectively reduced. Thirdly, through the above processing, not only can 4-bit quantized models be deployed on 8-bit hardware platforms, but computational speed and efficiency can also be improved.

[0122] Corresponding to the aforementioned embodiment of a data processing method, this application also provides an embodiment of a data processing apparatus.

[0123] The embodiments of the data processing apparatus provided in this application can be applied to data processing devices. The apparatus embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logically defined device, it is formed by a data processing processor that reads corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of a data processing device containing the data processing apparatus provided in this application. Except for... Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, the data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0124] Figure 6 This is a schematic diagram of the structure of a data processing apparatus according to Embodiment 1 of this application. Please refer to... Figure 6 The apparatus provided in this embodiment includes: a construction module 610 and a computing module 620; wherein,

[0125] The construction module 610 is used to construct FP16 data corresponding to each weight coefficient in the weight matrix, taking the weight coefficient as the data to be processed, based on a pre-set FP16 data construction rule; wherein, the FP16 data construction rule indicates that the FP16 data corresponding to the data to be processed is constructed based on the data to be processed with the data type int4.

[0126] The construction module 610 is also used to take the zero-point deviation of each weight coefficient as the data to be processed, and construct the FP16 data corresponding to the zero-point deviation based on the FP16 data construction rules.

[0127] The calculation module 620 is used to calculate the multiplication result of the input matrix and the weight matrix using a predetermined target matrix multiplication formula corresponding to the FP16 data construction rule, based on the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient; wherein, the target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and the mapping relationship corresponding to the predetermined FP16 data construction rule; the mapping relationship represents the mapping relationship between the actual value represented by the data to be processed and the actual value represented by the FP16 data corresponding to the data to be processed.

[0128] The apparatus of this embodiment can be used to perform... Figure 1The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0129] Optionally, the FP16 data construction rule is used to instruct the data to be processed to be filled into four consecutive empty spaces in a pre-defined 16-bit number to construct a data of type FP16, and to determine the constructed data as the FP16 data corresponding to the data to be processed; wherein, the values ​​of the other bits of the 16-bit number other than the four consecutive empty spaces are preset values.

[0130] Optionally, the process of determining the mapping relationship includes:

[0131] For each data point in the data range represented by the int4 data type, based on the FP16 data construction rules, an FP16 data of data type FP16 is constructed according to the binary representation of the data point, thus obtaining the FP16 data corresponding to the data point;

[0132] For each data point, the actual value represented by the data point and the actual value represented by the corresponding FP16 data are taken as a data pair to obtain the data pair corresponding to the data point;

[0133] The mapping relationship is determined based on the data pairs corresponding to all data points within the data range.

[0134] Optionally, the process of determining the target matrix multiplication formula includes:

[0135] Based on the mapping relationship, the first mapping relationship between the weight coefficient and the corresponding FP16 data is obtained, as well as the second mapping relationship between the zero-point deviation of the weight coefficient and the corresponding FP16 data.

[0136] Substituting the first mapping relationship and the second mapping relationship into the quantization formula corresponding to the weight matrix, an intermediate formula is obtained;

[0137] Substituting the intermediate formula into the initial matrix multiplication formula corresponding to the weight matrix, the target matrix multiplication formula is obtained.

[0138] Optionally, the four consecutive empty bits are four consecutive bits in the lower 10 bits.

[0139] Optionally, before using the weighting coefficient as data to be processed, the method further includes:

[0140] The weight matrix is ​​retrieved from the memory; wherein two adjacent weight coefficients in the weight matrix are combined into an 8-bit unsigned integer and stored in the memory.

[0141] For each unsigned integer in the weight matrix, it is split into two weight coefficients.

[0142] Please continue to refer to Figure 5 This application also provides a data processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0143] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in this application.

[0144] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0145] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0146] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized by, The method comprises: For each weight coefficient in the weight matrix, the weight coefficient is taken as to-be-processed data, and FP16 data corresponding to the weight coefficient is constructed based on a preset FP16 data construction rule; wherein the FP16 data construction rule indicates that, based on to-be-processed data of a data type of int4, FP16 data corresponding to the to-be-processed data is constructed; For each zero-point deviation of a weight coefficient, the zero-point deviation is taken as to-be-processed data, and FP16 data corresponding to the zero-point deviation is constructed based on the FP16 data construction rule; An input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and a scaling factor of the weight coefficient are used to calculate a multiplication result of the input matrix and the weight matrix by using a preset target matrix multiplication operation formula corresponding to the FP16 data construction rule; wherein the target matrix multiplication operation formula is derived based on an initial matrix multiplication operation formula corresponding to the weight matrix, a quantization formula corresponding to the weight matrix, and a preset mapping relationship corresponding to the FP16 data construction rule; the mapping relationship represents a mapping relationship between an actual value represented by to-be-processed data and an actual value represented by FP16 data corresponding to the to-be-processed data; The FP16 data construction rule is used to indicate that the to-be-processed data is filled into specified consecutive four empty positions in a preset 16-bit number to construct a data of a data type of FP16, and the constructed data is determined as the FP16 data corresponding to the to-be-processed data; wherein a value of a bit of the 16-bit number other than the consecutive four empty positions is a preset value.

2. The method of claim 1, wherein, The determination process of the mapping relationship comprises: For each data point in a data range represented by int4 data type, based on the FP16 data construction rule, an FP16 data of a data type of FP16 is constructed according to a binary representation of the data point, to obtain FP16 data corresponding to the data point; For each data point, an actual value represented by the data point and an actual value represented by the FP16 data corresponding to the data point are taken as a data pair, to obtain the data pair corresponding to the data point; The mapping relationship is determined according to the data pairs corresponding to all data points in the data range.

3. The method according to claim 1 or 2, characterized in that, The determination process of the target matrix multiplication operation formula comprises: According to the mapping relationship, a first mapping relationship between a weight coefficient and FP16 data corresponding to the weight coefficient, and a second mapping relationship between a zero-point deviation of the weight coefficient and FP16 data corresponding to the zero-point deviation are obtained; The first mapping relationship and the second mapping relationship are substituted into a quantization formula corresponding to the weight matrix to obtain an intermediate formula; The intermediate formula is substituted into an initial matrix multiplication operation formula corresponding to the weight matrix to obtain a target matrix multiplication operation formula.

4. The method of claim 1, wherein, The consecutive four empty positions are consecutive four bits in low 10 bits.

5. The method of claim 1, wherein, Before the weight coefficient is taken as to-be-processed data, the method further comprises: obtaining the weight matrix from the memory; wherein two adjacent weight coefficients in the weight matrix are combined into one 8-bit unsigned integer and stored in the memory; splitting each unsigned integer in the weight matrix into two weight coefficients.

6. A data processing apparatus, characterized by The device comprises a construction module and a calculation module; wherein, The construction module is configured to, for each weight coefficient in the weight matrix, take the weight coefficient as to-be-processed data, and construct the FP16 data corresponding to the weight coefficient based on a pre-set FP16 data construction rule; wherein the FP16 data construction rule indicates that the FP16 data corresponding to the to-be-processed data is constructed based on the to-be-processed data of the int4 type; The construction module is further configured to, for each zero-point deviation of a weight coefficient, take the zero-point deviation as to-be-processed data, and construct the FP16 data corresponding to the zero-point deviation based on the FP16 data construction rule; The calculation module is configured to calculate the multiplication result of the input matrix and the weight matrix by using a pre-determined target matrix multiplication formula corresponding to the FP16 data construction rule, according to the input matrix, the FP16 data corresponding to each weight coefficient of the weight matrix, the FP16 data corresponding to the zero-point deviation of the weight coefficient, and the scaling factor of the weight coefficient; wherein the target matrix multiplication formula is derived based on the initial matrix multiplication formula corresponding to the weight matrix, the quantization formula corresponding to the weight matrix, and a pre-determined mapping relationship corresponding to the FP16 data construction rule; the mapping relationship represents the mapping relationship between the actual value represented by the to-be-processed data and the actual value represented by the FP16 data corresponding to the to-be-processed data. The FP16 data construction rule is configured to indicate that the to-be-processed data is filled into the specified four consecutive empty positions in the pre-set 16-bit number to construct a data of the FP16 type, and the constructed data is determined as the FP16 data corresponding to the to-be-processed data; wherein the values of the other positions of the 16-bit number except the four consecutive empty positions are pre-set values.

7. A data processing device, characterized by A computer program product comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1-5 when executing the program.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method according to any one of claims 1-5.

Images