A data dequantization method, device, apparatus and computer readable storage medium

Abstract

The application relates to the technical field of deep learning, and discloses a data dequantization method, device and equipment and a computer readable storage medium. According to position information corresponding to int4 type bias of a current group and position information corresponding to fp16 type quantization coefficients of the current group, int4 type bias and fp16 type quantization coefficients are read out from a cache. According to position information corresponding to int4 type weights, int4 type weights are read out from the cache. When all int4 type weights in the current group are read out, according to a dequantization processing rule, all int4 type weights, int4 type bias and fp16 type quantization coefficients in the current group are processed to obtain a dequantization result. The above operation is repeatedly performed until dequantization calculation of all data in the cache is completed. 4-bit dequantization is realized on a processor that does not support 4-bit data types.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a data dequantization method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Model quantization is a common technique in deep learning used to reduce model size and computational resource requirements, thereby improving model efficiency. Model quantization refers to the process of converting parameters in a neural network model from high-precision floating-point numbers to low-precision integers. Common high-precision floating-point numbers include fp32 and fp16; low-precision integers include int8 and int4. Dequantization is a crucial step in the model inference process. When the deployed model performs inference, dequantization is used to convert the quantized low-precision integer parameters back to high-precision floating-point parameters.

[0003] `int8` is a standard data type on most hardware platforms, hence 8-bit quantization is widely used. To further reduce the storage requirements and computational complexity of the model, 4-bit quantization was developed. The implementation difficulty of 8-bit and 4-bit quantization is no different. However, the hardware platforms on which the quantized model is deployed vary widely, and not all hardware platforms support 4-bit computation.

[0004] It is evident that how to implement 4-bit dequantization on processors that do not support 4-bit data types, thereby fully leveraging the advantages of 4-bit quantization in terms of small storage space and high computational efficiency, is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a data inverse quantization method, apparatus, device, and computer-readable storage medium that can solve the application problem of 4-bit quantization on hardware platforms that do not support 4-bit data types.

[0006] To address the aforementioned technical problems, embodiments of the present invention provide a data inverse quantization method, comprising:

[0007] If the dequantization calculation is not completed, the int4 type bias is read from the cache according to the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group.

[0008] If not all int4 type weights in the current group are read, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights;

[0009] After reading all the int4 type weights in the current group, the int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed according to the dequantization processing rules to obtain the dequantization result corresponding to the current group.

[0010] If the current group is not the last group, the position information corresponding to each type of data in the next group is used as the position information of each type of data in the latest current group. The steps of reading the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group, and reading the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, are repeated until the dequantization calculation of all data in the cache is completed. Among them, each type of data includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight.

[0011] On one hand, based on the location information corresponding to the int4 type bias of the current group, the int4 type bias is read from the cache; based on the location information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache, including:

[0012] Based on the starting address and starting offset corresponding to the int4 type bias of the current group, read the int4 type bias of the current group from the cache and store the int4 type bias of the current group;

[0013] Based on the accumulated address of the int4 type bias of the current group, determine the address and offset corresponding to the int4 type bias of the next group;

[0014] Based on the starting address and starting offset corresponding to the quantization coefficients of the current group's fp16 type, read out the quantization coefficients of the current group's fp16 type and store them.

[0015] Based on the accumulated address of the quantization coefficients of type fp16 in the current group, determine the address and offset corresponding to the quantization coefficients of type fp16 in the next group.

[0016] On the one hand, if not all int4 type weights within the current group are read, the int4 type weights are read from the cache based on their corresponding position information, including:

[0017] Based on the starting address and starting offset corresponding to the int4 type weights, read the int4 type weights contained in the current group from the cache and store the int4 type weights of the current group;

[0018] Based on the accumulated address of the int4 type weights of the current group, calculate the address and offset corresponding to the int4 type weights of the next group.

[0019] On the one hand, according to the dequantization processing rules, all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group are processed to obtain the dequantization result corresponding to the current group, including:

[0020] Convert the int4 type weight to the int8 type weight; where the int8 type weight includes the high-order int8 type weight and the low-order int8 type weight;

[0021] Convert the int4 type bias to an int8 type bias; where the int8 type bias includes the high-order int8 type bias and the low-order int8 type bias;

[0022] According to the vector subtraction instruction, calculate the first difference data of the low-order int8 type weight and the low-order int8 type bias int8 type;

[0023] Call the data type conversion instruction to convert the lower 256 bits of the first difference data into the first low-order data of type fp16, and store the first low-order data of type fp16;

[0024] According to the shift instruction, the first difference data is shifted right by 256 bits, and the shifted first difference data is converted into the first high-order data of type fp16 and stored.

[0025] According to the vector subtraction instruction, calculate the second difference data of the high-order int8 type weight and the high-order int8 type bias int8 type;

[0026] Call the data type conversion instruction to convert the lower 256 bits of the second difference data into the second least significant bit data of type fp16, and store the second least significant bit data of type fp16;

[0027] According to the shift instruction, the second difference data is shifted right by 256 bits, and the shifted second difference data is converted into the second high-order data of type fp16 and stored.

[0028] Calculate the product of the quantization coefficient of type fp16 and its corresponding type fp16 data, use each product value as the inverse quantization result for each group, and store the inverse quantization result for each group.

[0029] On the one hand, converting int4 type weights to int8 type weights includes:

[0030] Parse the 512-bit data in the int4 type weight into 64 int8 type weights;

[0031] According to the vector right shift instruction, each int8 type weight is shifted to the right and the high-order int8 type weight obtained after the right shift is stored;

[0032] The weights of each int8 type are shifted to the left according to the vector left shift instruction;

[0033] According to the vector right shift instruction, the weights of each int8 type after the left shift are right-shifted, and the low-order int8 type weights obtained after the right shift are stored.

[0034] On the one hand, converting an int4 type bias to an int8 type bias includes:

[0035] Parse the 512-bit data in the int4 type bias into 64 int8 type biases;

[0036] According to the vector right shift instruction, the biases of each int8 type are shifted to the right and the high-order int8 type biases obtained after the right shift are stored;

[0037] The biases of each int8 type are shifted to the left according to the vector left shift instruction;

[0038] According to the vector right shift instruction, the offsets of each int8 type after the left shift are right-shifted, and the low-order int8 type offsets obtained after the right shift are stored.

[0039] On one hand, the product of the quantization coefficients of type fp16 and their corresponding type fp16 data is calculated, and each product value is used as the inverse quantization result for each group. The inverse quantization results for each group are stored, including:

[0040] Calculate the product of the quantization coefficient of type fp16 and the first least significant bit of type fp16, use the product value as the first set of corresponding inverse quantization results, and store the first set of corresponding inverse quantization results;

[0041] Calculate the product of the quantization coefficient of type fp16 and the first high-order data of type fp16, use the product value as the corresponding inverse quantization result of the second group, and store the corresponding inverse quantization result of the second group;

[0042] Calculate the product of the quantization coefficient of type fp16 and the second least significant bit of type fp16, use the product value as the inverse quantization result of the third group, and store the inverse quantization result of the third group.

[0043] Calculate the product of the quantization coefficient of type fp16 and the second highest bit of type fp16, use the product value as the inverse quantization result of the fourth group, and store the inverse quantization result of the fourth group.

[0044] This invention also provides a data dequantization device, including a first reading unit, a second reading unit, a dequantization unit, and a data processing unit;

[0045] The first reading unit is used to read the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the dequantization calculation is not completed; and to read the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group.

[0046] The second reading unit is used to read the int4 type weights from the cache according to the position information corresponding to the int4 type weights when not all int4 type weights in the current group have been read.

[0047] The dequantization unit is used to process all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group according to the dequantization processing rules after reading all int4 type weights in the current group, so as to obtain the dequantization result corresponding to the current group.

[0048] The unit is used to, when the current group is not the last group, take the position information corresponding to each type of data in the next group as the position information of each type of data in the latest current group, and return the steps of reading the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group; and reading the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed; wherein, each type of data includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight.

[0049] On the one hand, the first reading unit is used to read the int4 type bias of the current group from the cache according to the starting address and starting offset corresponding to the int4 type bias of the current group, and store the int4 type bias of the current group;

[0050] Based on the accumulated address of the int4 type bias of the current group, determine the address and offset corresponding to the int4 type bias of the next group;

[0051] Based on the starting address and starting offset corresponding to the quantization coefficients of the current group's fp16 type, read out the quantization coefficients of the current group's fp16 type and store them.

[0052] Based on the accumulated address of the quantization coefficients of type fp16 in the current group, determine the address and offset corresponding to the quantization coefficients of type fp16 in the next group.

[0053] On the one hand, the second reading unit is used to read the int4 type weights contained in the current group from the cache according to the starting address and starting offset corresponding to the int4 type weights, and store the int4 type weights of the current group;

[0054] Based on the accumulated address of the int4 type weights of the current group, calculate the address and offset corresponding to the int4 type weights of the next group.

[0055] On one hand, the inverse quantization unit includes a first conversion subunit, a second conversion subunit, a first subtraction subunit, a third conversion subunit, a first storage subunit, a first shift subunit, a fourth conversion subunit, a second storage subunit, a second subtraction subunit, a fifth conversion subunit, a third storage subunit, a second shift subunit, a sixth conversion subunit, a fourth storage subunit, and a multiplication subunit;

[0056] The first conversion subunit is used to convert the int4 type weights into int8 type weights; wherein, the int8 type weights include the high-order int8 type weights and the low-order int8 type weights;

[0057] The second conversion subunit is used to convert the int4 type bias to the int8 type bias; wherein the int8 type bias includes the high-order int8 type bias and the low-order int8 type bias;

[0058] The first subtraction subunit is used to calculate the first difference data of the low-order int8 type weight and the low-order int8 type bias int8 type according to the vector subtraction instruction;

[0059] The third conversion subunit is used to call a data type conversion instruction to convert the lower 256 bits of the first difference data into the first low-order data of type fp16.

[0060] The first storage subunit is used to store the first low-order data of type fp16;

[0061] The first shift subunit is used to shift the first difference data to the right by 256 bits according to the shift instruction;

[0062] The fourth conversion subunit is used to convert the right-shifted first difference data into the first high-order data of type fp16;

[0063] The second storage subunit is used to store the first high-order data of type fp16;

[0064] The second subtraction subunit is used to calculate the second difference data of the high-order int8 type weight and the high-order int8 type bias int8 type according to the vector subtraction instruction;

[0065] The fifth conversion subunit is used to call a data type conversion instruction to convert the lower 256 bits of the second difference data into the second low-order data of type fp16.

[0066] The third storage subunit is used to store the second low-order data of type fp16;

[0067] The second shift subunit is used to shift the second difference data to the right by 256 bits according to the shift instruction;

[0068] The sixth conversion subunit is used to convert the right-shifted second difference data into the second high-order data of type fp16;

[0069] The fourth storage subunit is used to store the second high-order data of type fp16;

[0070] The multiplication subunit is used to calculate the product of the quantization coefficient of type fp16 and its corresponding data of type fp16, use each product value as the inverse quantization result of each group, and store the inverse quantization result of each group.

[0071] On one hand, the first conversion subunit is used to parse the 512-bit data in the int4 type weights into 64 int8 type weights;

[0072] According to the vector right shift instruction, each int8 type weight is shifted to the right and the high-order int8 type weight obtained after the right shift is stored;

[0073] The weights of each int8 type are shifted to the left according to the vector left shift instruction;

[0074] According to the vector right shift instruction, the weights of each int8 type after the left shift are right-shifted, and the low-order int8 type weights obtained after the right shift are stored.

[0075] On the one hand, the second conversion subunit is used to parse 512 bits of data in the int4 type bias into 64 int8 type biases;

[0076] According to the vector right shift instruction, the biases of each int8 type are shifted to the right and the high-order int8 type biases obtained after the right shift are stored;

[0077] The biases of each int8 type are shifted to the left according to the vector left shift instruction;

[0078] According to the vector right shift instruction, the offsets of each int8 type after the left shift are right-shifted, and the low-order int8 type offsets obtained after the right shift are stored.

[0079] On the one hand, the multiplication sub-unit is used to calculate the product of the quantization coefficient of type fp16 and the first low-order data of type p16, and use the product value as the first group of corresponding inverse quantization results, and store the first group of corresponding inverse quantization results;

[0080] Calculate the product of the quantization coefficient of type fp16 and the first high-order data of type fp16, use the product value as the corresponding inverse quantization result of the second group, and store the corresponding inverse quantization result of the second group;

[0081] Calculate the product of the quantization coefficient of type fp16 and the second least significant bit of type fp16, use the product value as the inverse quantization result of the third group, and store the inverse quantization result of the third group.

[0082] Calculate the product of the quantization coefficient of type fp16 and the second highest bit of type fp16, use the product value as the inverse quantization result of the fourth group, and store the inverse quantization result of the fourth group.

[0083] This invention also provides a data inverse quantization device, comprising:

[0084] Memory, used to store computer programs;

[0085] A processor for executing the computer program to implement the steps of the data dequantization method described above.

[0086] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data dequantization method described above.

[0087] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described data dequantization method.

[0088] As can be seen from the above technical solution, when the dequantization calculation is not complete, the int4 type bias is read from the cache based on the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group. If not all int4 type weights in the current group have been read, the int4 type weights are read from the cache based on the position information corresponding to the int4 type weights. Once all int4 type weights in the current group have been read, it indicates that the data for the current group has been read completely. At this point, the dequantization processing rules can be applied to all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group to obtain the dequantization result for the current group. If the current group is not the last group, the position information corresponding to each type of data in the next group is used as the position information of each type of data in the latest current group. The process then returns the steps of reading the int4 type bias from the cache based on the position information corresponding to the int4 type bias of the current group, and reading the fp16 type quantization coefficient from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed. Each type of data includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight. The beneficial effect of this invention is that it can sequentially obtain the data required for performing the dequantization operation through a loop, including the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight within the same group. By setting the dequantization processing rules according to the int4 type data storage method, 4-bit dequantization can be implemented on processors that do not support 4-bit data types, thereby fully leveraging the advantages of 4-bit quantization, such as small storage space occupation and high computational efficiency. Attached Figure Description

[0089] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 A flowchart of a data inverse quantization method provided in an embodiment of the present invention;

[0091] Figure 2 A schematic diagram of a three-level cache provided in an embodiment of the present invention;

[0092] Figure 3A flowchart of a method for dequantizing int4 type weights, int4 type biases, and fp16 type quantization coefficients provided in an embodiment of the present invention;

[0093] Figure 4 This is a schematic diagram illustrating the storage and parsing of an int type according to an embodiment of the present invention;

[0094] Figure 5 This is a schematic diagram of the structure of a data dequantization device provided in an embodiment of the present invention;

[0095] Figure 6 This is a structural diagram of a data dequantization device provided in an embodiment of the present invention. Detailed Implementation

[0096] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0097] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0098] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0099] Model quantization can significantly reduce the size of model files, which is especially important for devices with limited storage. Using the lower-precision data type obtained through quantization can speed up computation, particularly on hardware that supports low-precision calculations. Quantization can also reduce the power consumption required during computation, which is especially important for battery-powered devices.

[0100] The purpose of dequantization is to restore the model's accuracy as much as possible while maintaining its inference speed. Dequantization is typically done by multiplying by the scaling factor used during quantization and adding the zero point offset. For linear quantization, the dequantization operation can be expressed as: floating-point number = (integer - offset) × scaling factor.

[0101] With 4-bit quantization, each model parameter is represented using 4 bits, requiring less storage space than 8 bits. Due to the use of a smaller data type, 4-bit quantization can further reduce the amount of data transferred during computation, thereby improving computational efficiency. On hardware that supports low-precision computation, 4-bit quantization can result in faster inference speeds. However, the hardware platforms on which the quantized model is deployed vary widely, including Graphics Processing Units (GPUs) and Application Specific Integrated Circuits (ASICs), and not all hardware platforms support 4-bit computation. On hardware that does not support 4-bit computation, dequantization during inference may require conversion to a higher-precision representation, which may offset some of the efficiency advantages.

[0102] Therefore, embodiments of the present invention provide a data dequantization method, apparatus, device, and computer-readable storage medium, which implements a two-dimensional grouped quantization int4 data processing flow based on a dual-channel vector processor. This allows for 4-bit dequantization to be implemented on processors that do not support 4-bit data types, fully leveraging the advantages of 4-bit quantization: small storage space usage and high computational efficiency. It can be applied to processors with different vector register widths, different numbers of vector calculation channels, or those supporting SIMD.

[0103] In this embodiment of the invention, the 4-bit weight is of type int4, obtained group by group from the original fp16 type weight through a linear quantization method. After quantization, in addition to the int4 type weight (weight_int4), there is also an int4 type bias (zero_int4) and an fp16 type quantization coefficient (scale_fp16). weight_int4 represents the int4 type weight, zero_int4 represents the int4 type bias, and scale_fp16 represents the fp16 type quantization coefficient.

[0104] Assuming the original fp16 type weights (weight_fp16) have a dimension of [K, N] and the group size is group_size, then the quantized weight_int4 has a dimension of [K, N]; zero_int4 has a dimension of [K / group_size, N]; and scale_fp16 has a dimension of [K / group_size, N]. The dequantization calculation formula is weight_fp16 = (weight_int4 - zero_int4) * scale_fp16. weight_fp16 represents the fp16 type weights.

[0105] Different models have different values ​​for K and N. For example, in the weights of llama3-70b Q, K=N=8192. The group_size can also be set as needed, and is usually set to 128.

[0106] Next, a data inverse quantization method provided by an embodiment of the present invention will be described in detail. Figure 1 A flowchart of a data inverse quantization method provided in an embodiment of the present invention, the method comprising:

[0107] S101: If the dequantization calculation is not completed, read the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group; read the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group.

[0108] In this embodiment of the invention, the processor used can be a multi-core processor, which has multiple computing cores inside, and the cache space that the processor can access is divided into three levels. Figure 2 This diagram illustrates a three-level cache according to an embodiment of the present invention. The three-level cache includes L3 cache, L2 cache, and L1 cache. The third-level L3 cache is High Bandwidth Memory (HBM), the second-level L2 cache is a shared storage space among the processor's on-chip cores, and the first-level L1 cache is a storage space dedicated to each core. From L3 to L1, the storage space decreases, but the storage bandwidth increases, resulting in faster data transfer.

[0109] Initially, the inputs weight_int4, zero_int4, and scale_fp16 required for dequantization are stored in the L3 cache. Before the kernel begins computation, these input data need to be moved to the L1 cache. Since weight_int4 has a large data volume and the bandwidth of the L3->L1 direct data storage (DMA) channel is small, L3->L1 means moving data directly from the L3 cache to the L1 cache. Frequently moving weight_int4 from the L3 cache to the L1 cache will reduce overall computational efficiency. To enable parallel computation among the kernels, weight_int4 can first be moved from the L3 cache to the L2 cache, and then each kernel can move the portion of weight_int4 data required for computation from the L2 cache to the L1 cache. zero_int4 and scale_fp16 have smaller data volumes, so each computation kernel can directly move the portion of data required for computation from the L3 cache to the L1 cache.

[0110] Assuming there are m kernels, and each kernel processes a weight_int4 with a dimension of [k_part, n_part], then n_part = N / m. Next, k_part is calculated based on the available L1 cache size within the kernel, while ensuring that k_part is an integer multiple of group_size. The corresponding dimensions of scale_fp16 and zero_int4 are [scale_k_part, n_part], where scale_k_part = k_part / group_size.

[0111] Each kernel corresponds to one L1 cache. For example, if the allocatable L1 cache is 256KB, then it is necessary to ensure that k_part*n_part / (2*(data size of weight_int4))<=256k. The numerical combination of k_part and n_part can be selected according to this constraint.

[0112] In this embodiment of the invention, the processor supports dual-channel vector operations (simultaneously executing two sets of vector operations under the control of the same instruction), and the vector register has a width of 512 bits. It supports data types with widths of 8 bits, 16 bits, and 32 bits, and the 512 bits of data in the vector register can be interpreted as several elements in units of 8 bits, 16 bits, or 32 bits. It supports performing the same operation on all vector elements with a single instruction; it supports setting the starting address and offset for each vector channel, where the starting address is the address at the time of the first vector access, and address jumps are achieved by accumulating the offset after each access.

[0113] In practical applications, the position information corresponding to the int4 type bias and the fp16 type quantization coefficient can be pre-stored. The position information can include address and offset. The address in the initial state can be considered the starting address, and the offset in the initial state can be considered the starting offset.

[0114] To read the bias of type int4, the starting address and starting offset of the current group's int4 type bias can be retrieved from the cache. For easier subsequent calls, the current group's int4 type bias can be stored in the bias vector register (vr_zero_int4). vr_zero_int4 represents the bias vector register, used to store int4 type biases. After each access, the address and offset of the next group's int4 type bias can be determined based on the accumulated address of the current group's int4 type bias.

[0115] To read quantized coefficients of type fp16, the quantized coefficients of the current group can be read based on the starting address and starting offset. For easier subsequent calls, the quantized coefficients of the current group can be stored in the coefficient vector register (vr_scale_fp16). vr_scale_fp16 represents the coefficient vector register, used to store quantized coefficients of type fp16. After each access, the address and offset of the next group's quantized coefficients can be determined based on the accumulated address of the current group's quantized coefficients.

[0116] S102: If not all int4 type weights in the current group are read, read the int4 type weights from the cache according to the position information corresponding to the int4 type weights.

[0117] In this embodiment of the invention, a three-loop method is used to sequentially read the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight.

[0118] The outermost loop is an `n_part` loop used to process different columns. Since the vector register is 512 bits wide and can store 128 `int4` data, the outermost loop processes 128 elements at a time. The next iteration jumps to the next group of 128 elements in the same row. The loop count is `n_part>>7`. `n_part>>7` means right shift by seven bits, equivalent to `n_part / 128`.

[0119] The intermediate loop, `scale_k_part`, is used to process different rows. Each loop processes one group, and the next loop jumps to the next adjacent group. The number of loops is `scale_k_part`. The intermediate loop can read the int4 type bias and fp16 type quantization coefficients corresponding to each group.

[0120] The innermost loop is a loop of size `group_size`, used to process different rows. Leveraging the processor core's support for dual-channel vector operations, each channel performs calculations on adjacent rows separately. Therefore, each loop can process the weight_int4 of two adjacent rows within the same group. The next loop jumps to the next two adjacent rows, with the loop iteration count being `group_size>>1`. `group_size>>1` represents a right shift by one bit, equivalent to `group_size / 2`. The outermost loop processes the weight_int4 of `group_size` rows within the same group, processing two rows simultaneously; the zero_int4 and scale_fp16 values ​​for a group's weight_int4 are only on one row.

[0121] In this embodiment of the invention, the starting address and starting offset corresponding to the int4 type weights can be pre-stored. Based on the starting address and starting offset corresponding to the int4 type weights, the int4 type weights contained in the current group can be read from the cache, and the int4 type weights of the current group can be stored; based on the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the int4 type weights of the next group can be calculated. All int4 type weights in the current group can be read through the innermost loop.

[0122] S103: After reading all int4 type weights in the current group, process all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group according to the dequantization processing rules to obtain the dequantization result corresponding to the current group.

[0123] The dequantization processing rules can be set according to the data storage method of the int4 type.

[0124] By performing dequantization on all int4 type weights, int4 type biases, and fp16 type quantization coefficients within the same group, the dequantization result can be obtained, which is the fp16 type weight.

[0125] The formula for calculating dequantization is weight_fp16 = (weight_int4 - zero_int4) * scale_fp16.

[0126] Since weight_int4 and zero_int4 are of type int4, during dequantization, the data in the weight vector register (vr_weight_int4) and the data in vr_zero_int4 must first be parsed and converted to type int8 using the same method. vr_weight_int4 represents the weight vector register, which stores weights of type int4.

[0127] To perform multiplication with scale_fp16, the int8 data type needs to be converted to fp16 data type before calculating the difference between weight_int4 and zero_int4. The calculated difference is then multiplied by scale_fp16 to obtain the final weight_fp16. For the detailed dequantization process, please refer to [link to documentation / documentation]. Figure 3 Introduction, about Figure 3 The content is described in detail later.

[0128] S104: If the current group is not the last group, take the position information of each type of data in the next group as the position information of each type of data in the current group, and return the steps of reading the int4 type bias from the cache according to the position information of the int4 type bias of the current group; and reading the fp16 type quantization coefficient from the cache according to the position information of the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed.

[0129] If the current group is not the last group, it means there are still unprocessed groups. Since the addresses and offsets of various data types for the next round are determined based on the accumulated addresses of the current loop, if the current group is not the last group, the addresses and offsets of each data type in the next group can be used as the addresses and offsets of each data type in the current group. The process then returns: retrieving the int4 type bias from the cache based on the address and offset corresponding to the int4 type bias of the current group; retrieving the fp16 type quantization coefficient from the cache based on the address and offset corresponding to the fp16 type quantization coefficient of the current group; and so on, until the dequantization calculation of all data in the cache is completed. Each data type includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight.

[0130] The above describes the dequantization calculation implemented using vector operations, consisting of three nested loops. The outermost loop processes 128 elements each time. The middle loop processes one group each time. The innermost loop processes the int4 weights within the same group. Each loop processes two rows of int4 weights simultaneously. The following section will describe the data reading process for various types of data.

[0131] The reading of weight_int4 is implemented in the innermost loop. Before the loop starts, the starting addresses of the two vector channels for reading weight_int4 are set to the starting address of row 0, weight_addr, and the starting address of row 1, weight_addr + (n_part >> 7), respectively. The innermost loop processes 128 adjacent elements in the same two rows in two consecutive iterations. Since weight_int4 is stored contiguously row by row, the offsets for reading weight_int4 from both channels are set to (n_part >> 7) * 2 in the innermost loop. After the innermost loop finishes, the address of `weight_int4` is read and accumulated to the starting addresses of the two lines of the next group's `weight_int4` (`weight_addr + group_size * (n_part >> 7)` and `weight_addr + (group_size + 1) * (n_part >> 7)`). Reading can continue in the same pattern until the intermediate loops finish. At this point, all lines corresponding to the current `n_part` loop have had their `weight_int4` values ​​processed, and the addresses of `weight_int4` read from the two vector channels are accumulated to `weight_addr + k_part * (n_part >> 7)` and `weight_addr + (k_part + 1) * (n_part >> 7)` respectively. Then, the loop jumps to the next `n_part` loop.

[0132] To ensure the correct address is read from weight_int4, the offsets for reading weight_int4 from both vector channels need to be set to -(k_part*(n_part>>7)-1) before the next n_part loop begins. This way, the addresses for the next weight_int4 read will be weight_addr+1 and weight_addr+(n_part>>7)+1, allowing the reading of the next group of 128 weight_int4 values. The read data is stored in the weight vector register.

[0133] The reading of `zero_int4` is implemented in an intermediate loop. Before the loop begins, the starting address for reading `zero_int4` in both vector channels is set to the starting address of line 0, `zero_addr`. Since the next intermediate loop processes the `weight_int4` of the next group, corresponding to the `zero_int4` of the next line, the offset for reading `zero_int4` is set to `n_part >> 7` inside the intermediate loop. At the end of the intermediate loop, the address for reading `zero_int4` has been accumulated to `zero_addr + (k_part / group_size) * (n_part >> 7)`. Then, it jumps to the next `n_part` loop. To ensure the correct address for reading `zero_int4`, the offset for reading `zero_int4` needs to be set to `-((k_part / group_size) * (n_part >> 7) - 1)` before the next `n_part` loop begins. The address for reading `zero_int4` becomes `zero_addr + 1`. The read data is stored in the bias vector register.

[0134] The reading of scale_fp16 is implemented in the intermediate loop. Before the loop starts, the starting address of the two vector channels for reading scale_fp16 is set to the starting address of line 0, scale_addr. Since the innermost loop can read 128 int4 type weights per read of weight_int4, while the intermediate loop can only read 32 fp16 type scales per read, reading scale_fp16 requires four consecutive reads. The address offset for the first three reads is set to 1. After the third read, the address is accumulated to scale_addr+3, and the fourth read starts from this address. However, the next intermediate loop needs to read the scale_fp16 corresponding to the next group, whose address is scale_addr+(n_part>>5). Therefore, the offset address for the fourth read should be set to (n_part>>5)-3. After the fourth read, the address needs to jump to scale_addr+(n_part)>>5, and the address for the fourth read is scale_addr+3, so the offset needs to be set to the above value.

[0135] When the intermediate loop ends, the address of `scale_fp16` has been accumulated to `scale_addr + (k_part / group_size) * (n_part >> 5)`, after which it jumps to the next `n_part` loop. To ensure the correct address for reading `scale_fp16`, the offset for reading `scale_fp16` needs to be set to -((k_part / group_size) * (n_part >> 5) - 4) before the next `n_part` loop begins. The address for the next read of `scale_fp16` then becomes `scale_addr + 4`. Since four reads are involved, four coefficient vector registers can be set to store the corresponding data respectively. In practical applications, the read data can be stored sequentially in the first coefficient vector register (`vr_scale_fp16_0`), the second coefficient vector register (`vr_scale_fp16_1`), the third coefficient vector register (`vr_scale_fp16_2`), and the fourth coefficient vector register (`vr_scale_fp16_3`). vr_scale_fp16_0 represents the first coefficient vector register, vr_scale_fp16_1 represents the second coefficient vector register, vr_scale_fp16_2 represents the third coefficient vector register, and vr_scale_fp16_3 represents the fourth coefficient vector register.

[0136] The storage of the dequantized results is implemented in the innermost loop. This loop processes two rows of `weight_int4` simultaneously, resulting in two rows of `weight_fp16`. Before the loop begins, the starting addresses for storing `weight_fp16` in the two vector channels are set to the starting address `out_addr` of row 0 and the starting address `out_addr + (n_part >> 5)` of row 1, respectively. Each iteration of the innermost loop processes 128 `weight_int4` values, resulting in 128 `weight_fp16` values. This requires storage in four vector registers. When storing the `weight_fp16` values ​​in the first three vector registers, the address offset is set to 1. After storage, the address is incremented to `out_addr + 3`, and the `weight_fp16` value in the fourth vector register is stored at this address. Since the next innermost loop will process the next two lines of weight, the calculation result should also be saved to the next two lines. Therefore, when saving weight_fp16 in the fourth vector register, the offset address is set to 2*(n_part>>5)-3, and the address jumps to out_addr+(n_part>>5).

[0137] In this embodiment of the invention, the processor's three-level cache structure is utilized to improve data transfer efficiency. A specific int4 data storage method is designed based on the width of the vector register, improving the efficiency of int4 data parsing. Following the loop method described above, dequantization calculations can be completed quickly and accurately.

[0138] After completing the dequantization calculation, the int4 type bias, fp16 type quantization coefficient, int4 type weight, and their corresponding dequantization results can be stored in a corresponding list. When performing dequantization on data of the same type subsequently, the dequantization result matching the current data can be directly retrieved from the corresponding list.

[0139] In this embodiment of the invention, by recording the inverse quantization results corresponding to common data types, the inverse quantization operation is avoided every time the same data is processed, effectively improving the data processing efficiency.

[0140] As can be seen from the above technical solution, when the dequantization calculation is not complete, the int4 type bias is read from the cache based on the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group. If not all int4 type weights in the current group have been read, the int4 type weights are read from the cache based on the position information corresponding to the int4 type weights. Once all int4 type weights in the current group have been read, it indicates that the data for the current group has been read completely. At this point, the dequantization processing rules can be applied to all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group to obtain the dequantization result for the current group. If the current group is not the last group, the position information corresponding to each type of data in the next group is used as the position information of each type of data in the latest current group. The process then returns the steps of reading the int4 type bias from the cache based on the position information corresponding to the int4 type bias of the current group, and reading the fp16 type quantization coefficient from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed. Each type of data includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight. The beneficial effect of this invention is that it can sequentially obtain the data required for performing the dequantization operation through a loop, including the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight within the same group. By setting the dequantization processing rules according to the int4 type data storage method, 4-bit dequantization can be implemented on processors that do not support 4-bit data types, thereby fully leveraging the advantages of 4-bit quantization, such as small storage space occupation and high computational efficiency.

[0141] Figure 3 A flowchart of a method for dequantizing int4 type weights, int4 type biases, and fp16 type quantization coefficients provided in this embodiment of the invention is provided. The method includes:

[0142] S301: Convert the weights of type int4 to type int8.

[0143] The weights of type int8 can include both high-order int8 weights and low-order int8 weights.

[0144] Weights of type int4 are stored in vr_weight_int4. During type conversion, the 512-bit data in vr_weight_int4 is first parsed into 64 weights of type int8. Then, following the vector right shift instruction, the high-order int8 weights obtained after right shifting are stored in the weight high-order vector register (vr_weight_int8_hi). Following the vector left shift instruction, the weights of type int8 are left-shifted; following the vector right shift instruction, the left-shifted weights of type int8 are right-shifted, and the low-order int8 weights obtained after right shifting are stored in the weight low-order vector register (vr_weight_int8_lo). vr_weight_int8_hi represents the weight high-order vector register, and vr_weight_int8_lo represents the weight low-order vector register.

[0145] S302: Convert the bias of type int4 to the bias of type int8.

[0146] The bias of type int8 can include the bias of type int8 in the high-order bits and the bias of type int8 in the low-order bits.

[0147] The data in `vr_zero_int4` is parsed and converted to `int8` type using the same processing method as `vr_weight_int4`. The parsed data is stored in the bias high-order vector register (`vr_zero_int8_hi`) and the bias low-order vector register (`vr_zero_int8_lo`). `vr_zero_int8_hi` represents the bias high-order vector register, used to store biases 64 to 127. `vr_zero_int8_lo` represents the bias low-order vector register, used to store biases 0 to 63.

[0148] When performing type conversion, the 512-bit data in the bias vector register can first be parsed into 64 int8 type biases. Following the vector right shift instruction, the high-order int8 type biases obtained after right shifting each int8 type bias are stored in vr_zero_int8_hi. Following the vector left shift instruction, each int8 type bias is left-shifted; following the vector right shift instruction, the left-shifted int8 type biases are right-shifted, and the low-order int8 type biases obtained after right shifting are stored in vr_zero_int8_lo.

[0149] Figure 4This diagram illustrates the storage and parsing of an int type data according to an embodiment of the present invention. After reading zero_int4 through an intermediate layer, the 128 int4 data stored in vr_zero_int4 need to be parsed first. Because the processor used in this invention does not support the int4 data type, the int4 data needs to be converted to int8 data type for calculation.

[0150] First, using vector shift instructions, the 512-bit data in `vr_zero_int4` is parsed into 64 `int8` data elements. Each `int8` element is then right-shifted by 4 bits, and the resulting data is stored in `vr_zero_int8_hi`, thus obtaining 64 zeros stored in bits 4-7, 12-15, ..., 508-511. Next, using vector shift instructions, `vr_zero_int4` is left-shifted by 4 bits in units of `int8`, with the result still stored in `vr_zero_int4`. This retains the 64 zeros stored in bits 0-3, 4-7, 8-11, ..., 504-507, and 508-511. However, since these 64 zeros are stored in the high 4 bits of their respective `int8` elements, vector shift instructions are used again to right-shift each `int8` element by 4 bits, shifting the valid data back to the low 4 bits. The result is stored in `vr_zero_int8_lo`.

[0151] To avoid rearranging the data order after parsing and achieve optimal efficiency, the data in `vr_zero_int8_hi` and `vr_zero_int8_lo` must be stored sequentially. Specifically, `vr_zero_int8_lo` stores zeros 0-63, and `vr_zero_int8_hi` stores zeros 64-127. Considering the vector register width is 512 bits, when storing `zero_int4`, it needs to be stored in groups of 128, starting from the least significant bit: `zero_0`, `zero_64`, `zero_1`, `zero_65`...`zero_63`, `zero_127`.

[0152] S303: Calculate the first difference of the low-order int8 type weight and the low-order int8 type bias in the vector subtraction instruction.

[0153] To calculate the difference between weight and zero, we can use the vector subtraction instruction to calculate the difference between vr_weight_int8_lo and vr_zero_int8_lo, and the result is still stored in vr_weight_int8_lo. Similarly, we can use the vector subtraction instruction to calculate the difference between vr_weight_int8_hi and vr_zero_int8_hi, and the result is still stored in vr_weight_int8_hi.

[0154] For ease of distinction, the difference between vr_weight_int8_lo and vr_zero_int8_lo can be called the first difference data, and the difference between vr_weight_int8_hi and vr_zero_int8_hi can be called the second difference data.

[0155] S304: Call the data type conversion instruction to convert the lower 256 bits of the first difference data into the first low-order bit data of type fp16, and store the first low-order bit data of type fp16.

[0156] To perform multiplication with the fp16 type scale, the int8 data in vr_weight_int8_lo needs to be converted to fp16 data first. This can be done using data type conversion instructions to convert the lower 256 bits of the 32 int8 data stored in vr_weight_int8_lo to fp16 data, i.e., the first least significant bit data, and then store this first least significant bit data in the first fp16 weight register (vr_weight_fp16_lo_0). vr_weight_fp16_lo_0 represents the first fp16 weight register, used to store weights 0 to 31.

[0157] S305: According to the shift instruction, shift the first difference data to the right by 256 bits, convert the shifted first difference data into the first high-order data of type fp16, and store the first high-order data of type fp16.

[0158] The entire `vr_weight_int8_lo` is shifted right by 256 bits using a shift instruction, thus removing the lower 256 bits and retaining the higher 256 bits (stored starting from the least significant bit). Then, a data type conversion instruction converts the 32 `int8` data bytes stored in the higher 256 bits into the first high-order bit data. This first high-order bit data is of type `fp16` and is stored in the second `fp16` weight register (`vr_weight_fp16_lo_1`). `vr_weight_fp16_lo_1` represents the second `fp16` weight register, used to store weights 32 to 63.

[0159] S306: Calculate the second difference of the high-order int8 type weight and the high-order int8 type bias in accordance with the vector subtraction instruction.

[0160] In this embodiment of the invention, the difference between vr_weight_int8_hi and vr_zero_int8_hi can be calculated using the same method as the weight and bias of the low-order int8 type described above.

[0161] S307: Call the data type conversion instruction to convert the lower 256 bits of the second difference data into the second least significant bit data of type fp16, and store the second least significant bit data of type fp16.

[0162] After converting the lower 256 bits of the second difference data into the second least significant bit of type fp16, it is stored in the third fp16 weight register (vr_weight_fp16_hi_0). vr_weight_fp16_hi_0 represents the third fp16 weight register, which is used to store the 64th to 95th weights.

[0163] S308: According to the shift instruction, the second difference data is shifted right by 256 bits, and the shifted second difference data is converted into the second high-order data of type fp16 and stored.

[0164] The second most significant bit is stored in the fourth fp16 weight register (vr_weight_fp16_hi_1). vr_weight_fp16_hi_1 represents the fourth fp16 weight register, which is used to store the 96th to 127th weights.

[0165] S309: Calculate the product of the quantization coefficient of type fp16 and its corresponding type fp16 data, use each product value as the inverse quantization result of each group, and store the inverse quantization result of each group.

[0166] As described in the S104 description, there are four coefficient vector registers: the first coefficient vector register, the second coefficient vector register, the third coefficient vector register, and the fourth coefficient vector register. There are also four fp16 weight registers corresponding to the quantization coefficients: the first fp16 weight register, the second fp16 weight register, the third fp16 weight register, and the fourth fp16 weight register.

[0167] When multiplying the fp16 type data recorded in the fp16 weight register with the fp16 type quantization coefficients recorded in each coefficient vector register, the product of the fp16 type quantization coefficient recorded in the first coefficient vector register and the first low-order fp16 type data recorded in the first fp16 weight register can be calculated. The product value is used as the first set of corresponding dequantization results, and the first set of corresponding dequantization results is stored in the first dequantization register (vr_fp16_lo_0). vr_fp16_lo_0 represents the first dequantization register.

[0168] Calculate the product of the quantization coefficients of type fp16 recorded in the second coefficient vector register and the first high-order data of type fp16 recorded in the second fp16 weight register. Use the product value as the second set of corresponding dequantization results and store the second set of corresponding dequantization results in the second dequantization register (vr_fp16_lo_1). vr_fp16_lo_1 represents the second dequantization register.

[0169] Calculate the product of the quantization coefficients of type fp16 recorded in the third coefficient vector register and the second least significant bit of type fp16 recorded in the third fp16 weight register. Use the product value as the corresponding dequantization result for the third group and store the corresponding dequantization result for the third group in the third dequantization register (vr_fp16_hi_0). vr_fp16_hi_0 represents the third dequantization register.

[0170] Calculate the product of the quantization coefficients of type fp16 recorded in the fourth coefficient vector register and the second high-order data of type fp16 recorded in the fourth fp16 weight register. Use the product value as the dequantization result of the fourth group and store the dequantization result of the fourth group in the fourth dequantization register (vr_fp16_hi_1). vr_fp16_hi_1 represents the fourth dequantization register.

[0171] For example, to calculate the product of weight and scale, use the vector multiplication instruction to calculate the product of vr_weight_fp16_lo_0 and vr_scale_fp16_0, store the result in vr_fp16_lo_0, and then store vr_weight_fp16_lo_0 in the L1 cache; process vr_weight_fp16_lo_1 and vr_scale_fp16_1, vr_weight_fp16_hi_0 and vr_scale_fp16_2, vr_weight_fp16_hi_1 and vr_scale_fp16_3 in the same way.

[0172] This invention implements parsing and dequantization of int4 data based on a dual-channel vector processor. It enables 4-bit dequantization on processors that do not support 4-bit data types, fully leveraging the advantages of 4-bit quantization: small storage space and high computational efficiency. The processor's three-level cache structure improves data transfer efficiency; a specific int4 data storage method is designed based on the width of the vector register, improving the efficiency of int4 data parsing and avoiding the need to rearrange the data order after parsing. A complete processing flow based on dual-channel vector operations is presented for grouped quantized two-dimensional int4 data.

[0173] Figure 5 A schematic diagram of a data dequantization device provided in an embodiment of the present invention includes a first reading unit 51, a second reading unit 52, a dequantization unit 53, and a data processing unit 54;

[0174] The first reading unit 51 is used to read the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the dequantization calculation is not completed; and to read the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group.

[0175] The second reading unit 52 is used to read the int4 type weights from the cache according to the position information corresponding to the int4 type weights when not all int4 type weights in the current group have been read.

[0176] The dequantization unit 53 is used to process all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group according to the dequantization processing rules after reading all int4 type weights in the current group, so as to obtain the dequantization result corresponding to the current group; wherein, the dequantization processing rules are set according to the data storage method of int4 type.

[0177] Unit 54 is used to take the position information of each type of data in the next group as the position information of each type of data in the latest current group when the current group is not the last group, and return the steps of reading the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group; and reading the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed; wherein, each type of data includes at least the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight.

[0178] In some embodiments, the first reading unit is used to read the int4 type bias of the current group from the cache according to the starting address and starting offset corresponding to the int4 type bias of the current group, and store the int4 type bias of the current group;

[0179] Based on the accumulated address of the int4 type bias of the current group, determine the address and offset corresponding to the int4 type bias of the next group;

[0180] Based on the starting address and starting offset corresponding to the quantization coefficients of the current group's fp16 type, read out the quantization coefficients of the current group's fp16 type and store them.

[0181] Based on the accumulated address of the quantization coefficients of type fp16 in the current group, determine the address and offset corresponding to the quantization coefficients of type fp16 in the next group.

[0182] In some embodiments, the second reading unit is used to read the int4 type weights contained in the current group from the cache according to the starting address and starting offset corresponding to the int4 type weights, and store the int4 type weights of the current group;

[0183] Based on the accumulated address of the int4 type weights of the current group, calculate the address and offset corresponding to the int4 type weights of the next group.

[0184] In some embodiments, the dequantization unit includes a first conversion subunit, a second conversion subunit, a first subtraction subunit, a third conversion subunit, a first storage subunit, a first shift subunit, a fourth conversion subunit, a second storage subunit, a second subtraction subunit, a fifth conversion subunit, a third storage subunit, a second shift subunit, a sixth conversion subunit, a fourth storage subunit, and a multiplication subunit;

[0185] The first conversion subunit is used to convert the int4 type weights to int8 type weights; wherein, the int8 type weights include the high-order int8 type weights and the low-order int8 type weights;

[0186] The second conversion subunit is used to convert the int4 type bias to the int8 type bias; wherein the int8 type bias includes the high-order int8 type bias and the low-order int8 type bias;

[0187] The first subtraction subunit is used to calculate the first difference data of low-order int8 type weight and low-order int8 type bias in int8 type according to the vector subtraction instruction;

[0188] The third conversion subunit is used to call the data type conversion instruction to convert the lower 256 bits of the first difference data into the first low-order data of type fp16.

[0189] The first storage subunit is used to store the first low-order data of type fp16;

[0190] The first shift subunit is used to shift the first difference data to the right by 256 bits according to the shift instruction;

[0191] The fourth conversion subunit is used to convert the right-shifted first difference data into the first high-order data of type fp16;

[0192] The second storage subunit is used to store the first high-order data of type fp16;

[0193] The second subtraction subunit is used to calculate the second difference data of the high-order int8 type weight and the high-order int8 type bias according to the vector subtraction instruction;

[0194] The fifth conversion subunit is used to call the data type conversion instruction to convert the lower 256 bits of the second difference data into the second least significant bit of the fp16 type.

[0195] The third storage subunit is used to store the second least significant bit of data of type fp16;

[0196] The second shift subunit is used to shift the second difference data to the right by 256 bits according to the shift instruction;

[0197] The sixth conversion subunit is used to convert the right-shifted second difference data into the second high-order bit data of type fp16;

[0198] The fourth storage subunit is used to store the second most significant bit of data of type fp16;

[0199] The multiplication sub-unit is used to calculate the product of the quantization coefficient of type fp16 and its corresponding type fp16 data, and to store the inverse quantization result of each group.

[0200] In some embodiments, the first conversion subunit is used to parse 512 bits of data in the int4 type weights into 64 int8 type weights;

[0201] According to the vector right shift instruction, each int8 type weight is shifted to the right and the high-order int8 type weight obtained after the right shift is stored;

[0202] The weights of each int8 type are shifted to the left according to the vector left shift instruction;

[0203] According to the vector right shift instruction, the weights of each int8 type after the left shift are right-shifted, and the low-order int8 type weights obtained after the right shift are stored.

[0204] In some embodiments, the second conversion subunit is used to parse 512 bits of data in the int4 type bias into 64 int8 type biases;

[0205] According to the vector right shift instruction, the biases of each int8 type are shifted to the right and the high-order int8 type biases obtained after the right shift are stored;

[0206] The biases of each int8 type are shifted to the left according to the vector left shift instruction;

[0207] According to the vector right shift instruction, the offsets of each int8 type after the left shift are right-shifted, and the low-order int8 type offsets obtained after the right shift are stored.

[0208] In some embodiments, the multiplication subunit is used to calculate the product of the quantization coefficient of type fp16 and the first low-order data of type fp16, use the product value as the first set of corresponding inverse quantization results, and store the first set of corresponding inverse quantization results.

[0209] Calculate the product of the quantization coefficient of type fp16 and the first high-order data of type fp16, use the product value as the corresponding inverse quantization result of the second group, and store the corresponding inverse quantization result of the second group;

[0210] Calculate the product of the quantization coefficient of type fp16 and the second least significant bit of type fp16, use the product value as the inverse quantization result of the third group, and store the inverse quantization result of the third group.

[0211] Calculate the product of the quantization coefficient of type fp16 and the second highest bit of type fp16, use the product value as the inverse quantization result of the fourth group, and store the inverse quantization result of the fourth group.

[0212] As can be seen from the above technical solution, when the dequantization calculation is not complete, the int4 type bias is read from the cache based on the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group. If not all int4 type weights in the current group have been read, the int4 type weights are read from the cache based on the position information corresponding to the int4 type weights. Once all int4 type weights in the current group have been read, it indicates that the data for the current group has been read completely. At this point, the dequantization processing rules can be applied to all int4 type weights, int4 type biases, and fp16 type quantization coefficients in the current group to obtain the dequantization result for the current group. If the current group is not the last group, the position information corresponding to each type of data in the next group is used as the position information of each type of data in the latest current group. The process then returns the steps of reading the int4 type bias from the cache based on the position information corresponding to the int4 type bias of the current group, and reading the fp16 type quantization coefficient from the cache based on the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed. Each type of data includes at least an int4 type bias, an fp16 type quantization coefficient, and an int4 type weight. The beneficial effect of this invention is that it can sequentially obtain the data required for performing the dequantization operation through a loop, including the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight within the same group. By setting the dequantization processing rules according to the int4 type data storage method, 4-bit dequantization can be implemented on processors that do not support 4-bit data types, thereby fully leveraging the advantages of 4-bit quantization, such as small storage space occupation and high computational efficiency.

[0213] Figure 5 For a description of the features in the corresponding embodiments, please refer to Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0214] Figure 6 A structural diagram of a data inverse quantization device provided in an embodiment of the present invention is shown below. Figure 6 As shown, the data dequantization device includes: a memory 60 for storing computer programs;

[0215] The processor 61 is used to implement the steps of the data dequantization method as described in the above embodiments when executing a computer program.

[0216] The data dequantization device provided in this embodiment may include, but is not limited to, smartphones, tablets, laptops, or desktop computers.

[0217] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0218] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the data dequantization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include, but is not limited to, the address and offset corresponding to the int4 type bias of the current group, the address and offset corresponding to the fp16 type quantization coefficient of the current group, the int4 type bias, the fp16 type quantization coefficient, the address and offset corresponding to the int4 type weight, and the int4 type weight, etc.

[0219] In some embodiments, the data dequantization device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0220] Those skilled in the art will understand that Figure 6The structure shown does not constitute a limitation on the data dequantization device and may include more or fewer components than illustrated.

[0221] It is understood that if the data dequantization method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0222] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data dequantization method described above.

[0223] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-described data dequantization method.

[0224] The foregoing has provided a detailed description of a data inverse quantization method, apparatus, device, computer-readable storage medium, and computer program product provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0225] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0226] The foregoing has provided a detailed description of the data inverse quantization method, apparatus, device, computer-readable storage medium, and computer program product provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method of data dequantization, the method comprising: The method comprises the following steps: In the case that the dequantization calculation is not completed, reading out the int4 type bias corresponding to the position information of the current group from the cache; In the case that all int4 type weights in the current group are not read out, reading out the int4 type weight from the cache according to the position information corresponding to the int4 type weight; In the case that all int4 type weights in the current group are read out, processing all int4 type weights, int4 type bias and fp16 type quantization coefficient in the current group according to the dequantization processing rule to obtain the dequantization result corresponding to the current group; In the case that the current group is not the last group, taking the position information of each type of data of the next group as the position information of each type of data of the latest current group, and returning to the step of reading out the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group; reading out the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, until the dequantization calculation of all data in the cache is completed; wherein the data of each type at least includes int4 type bias, fp16 type quantization coefficient and int4 type weight; Processing all int4 type weights, int4 type bias and fp16 type quantization coefficient in the current group according to the dequantization processing rule to obtain the dequantization result corresponding to the current group comprises: Converting the int4 type weight into int8 type weight; wherein the int8 type weight includes high int8 type weight and low int8 type weight; Converting the int4 type bias into int8 type bias; wherein the int8 type bias includes high int8 type bias and low int8 type bias; According to the vector subtraction instruction, calculating the int8 type first difference data of the low int8 type weight and the low int8 type bias; Calling the data type conversion instruction to convert the low 256 bit data in the first difference data into fp16 type first low bit data, and storing the fp16 type first low bit data; According to the shift instruction, shifting the first difference data right by 256 bits, and converting the first difference data after right shifting into fp16 type first high bit data, and storing the fp16 type first high bit data; According to the vector subtraction instruction, calculating the int8 type second difference data of the high int8 type weight and the high int8 type bias; According to the vector subtraction instruction, calculating the int8 type second difference data of the high int8 type weight and the high int8 type bias; The data type conversion instruction is called to convert low 256-bit data in the second difference data into second low-bit data of the fp16 type, and the second low-bit data of the fp16 type is stored. The second difference data is right shifted by 256 bits according to a shift instruction, and the second difference data after the right shift is converted into second high-bit data of the fp16 type, and the second high-bit data of the fp16 type is stored. The product of the quantization coefficient of the fp16 type and the corresponding data of the fp16 type is calculated, each product value is taken as the corresponding dequantization result of each group, and the corresponding dequantization result of each group is stored.

2. The data dequantization method of claim 1, wherein, The int4 type bias corresponding to the position information of the int4 type bias of the current group is read from the cache. The fp16 type quantization coefficient corresponding to the position information of the fp16 type quantization coefficient of the current group is read from the cache, including: The int4 type bias of the current group is read from the cache according to the starting address and the starting offset corresponding to the int4 type bias of the current group, and the int4 type bias of the current group is stored. The address and the offset corresponding to the int4 type bias of the next group are determined according to the accumulated address of the int4 type bias of the current group. The fp16 type quantization coefficient of the current group is read and stored according to the starting address and the starting offset corresponding to the fp16 type quantization coefficient of the current group. The address and the offset corresponding to the fp16 type quantization coefficient of the next group are determined according to the accumulated address of the fp16 type quantization coefficient of the current group.

3. The method of claim 1, wherein, In the case where all the int4 type weights in the current group are not read, the int4 type weight is read from the cache according to the position information corresponding to the int4 type weight, including: The int4 type weight contained in the current group is read from the cache according to the starting address and the starting offset corresponding to the int4 type weight, and the int4 type weight of the current group is stored. The address and the offset corresponding to the int4 type weight of the next group are calculated according to the accumulated address of the int4 type weight of the current group.

4. The method of claim 1, wherein, The int4 type weight is converted into the int8 type weight, including: The 512-bit data in the int4 type weight is parsed into 64 int8 type weights; Each int8 type weight is right shifted according to a vector right shift instruction, and the high-bit int8 type weight obtained after the right shift is stored; Each int8 type weight is left shifted according to a vector left shift instruction; Each int8 type weight after the left shift is right shifted according to a vector right shift instruction, and the low-bit int8 type weight obtained after the right shift is stored.

5. The method of claim 1, wherein, The int4 type bias is converted into the int8 type bias, including: The 512-bit data in the int4 type bias is parsed into 64 int8 type biases; Each int8 type bias is right shifted according to a vector right shift instruction, and the high-bit int8 type bias obtained after the right shift is stored; Each int8 type bias is left shifted according to a vector left shift instruction; According to the vector right shift instruction, right shift is performed on each int8 type bias after left shift, and the low bit int8 type bias obtained after right shift is stored.

6. The method of claim 1, wherein, The product of the fp16 type quantization coefficient and the corresponding fp16 type data is calculated, each product value is taken as the corresponding dequantization result of each group, and the corresponding dequantization result of each group is stored, including: The product of the fp16 type quantization coefficient and the first low bit data of the fp16 type is calculated, the product value is taken as the dequantization result corresponding to the first group, and the dequantization result corresponding to the first group is stored. The product of the fp16 type quantization coefficient and the first high bit data of the fp16 type is calculated, the product value is taken as the dequantization result corresponding to the second group, and the dequantization result corresponding to the second group is stored. The product of the fp16 type quantization coefficient and the second low bit data of the fp16 type is calculated, the product value is taken as the dequantization result corresponding to the third group, and the dequantization result corresponding to the third group is stored. The product of the fp16 type quantization coefficient and the second high bit data of the fp16 type is calculated, the product value is taken as the dequantization result corresponding to the fourth group, and the dequantization result corresponding to the fourth group is stored.

7. A data dequantization apparatus characterized by comprising: The first reading unit, the second reading unit, the dequantization unit and the as unit are included. The first reading unit is configured to read out the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the dequantization calculation is not completed, read out the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, and move the int4 type bias from the L3 cache to the L2 cache, move the int4 type weight data required for calculation from the L2 cache to the L1 cache by each kernel, and move the int4 type bias and the fp16 type quantization coefficient from the L3 cache to the L1 cache directly by each calculation kernel. The second reading unit is configured to read the int4 type weight from the cache according to the position information corresponding to the int4 type weight when all the int4 type weights in the current group are not read out. The dequantization unit is configured to, when all the int4 type weights in the current group are read out, process all the int4 type weights, the int4 type bias and the fp16 type quantization coefficient in the current group according to the dequantization processing rule to obtain the dequantization result corresponding to the current group. The unit is configured to, in the case that the current group is not the last group, read out the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group, and read out the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group, until the inverse quantization calculation of all data in the cache is completed; wherein the data includes at least the int4 type bias, the fp16 type quantization coefficient and the int4 type weight; The inverse quantization unit comprises a first conversion subunit, a second conversion subunit, a first subtraction subunit, a third conversion subunit, a first storage subunit, a first shift subunit, a fourth conversion subunit, a second storage subunit, a second subtraction subunit, a fifth conversion subunit, a third storage subunit, a second shift subunit, a sixth conversion subunit, a fourth storage subunit and a multiplication subunit; The first conversion subunit is configured to convert the int4 type weight into an int8 type weight; wherein the int8 type weight comprises a high-bit int8 type weight and a low-bit int8 type weight; The second conversion subunit is configured to convert the int4 type bias into an int8 type bias; wherein the int8 type bias comprises a high-bit int8 type bias and a low-bit int8 type bias; The first subtraction subunit is configured to calculate, according to a vector subtraction instruction, an int8 type first difference data of the low-bit int8 type weight and the low-bit int8 type bias; The third conversion subunit is configured to convert, by calling a data type conversion instruction, low 256-bit data in the first difference data into an fp16 type first low-bit data; The first storage subunit is configured to store the fp16 type first low-bit data; The first shift subunit is configured to right shift, according to a shift instruction, the first difference data by 256 bits; The fourth conversion subunit is configured to convert the right-shifted first difference data into an fp16 type first high-bit data; The second storage subunit is configured to store the fp16 type first high-bit data; The second subtraction subunit is configured to calculate, according to a vector subtraction instruction, an int8 type second difference data of the high-bit int8 type weight and the high-bit int8 type bias; The fifth conversion subunit is configured to convert, by calling a data type conversion instruction, low 256-bit data in the second difference data into an fp16 type second low-bit data; The third storage subunit is configured to store the fp16 type second low-bit data; The second shift subunit is configured to right shift, according to a shift instruction, the second difference data by 256 bits; The sixth conversion subunit is configured to convert the right-shifted second difference data into an fp16 type second high-bit data; The fourth storage subunit is configured to store the fp16 type second high-bit data; A multiplication unit is configured to calculate products of the quantized coefficients of the fp16 type and the data of the fp16 type corresponding to the quantized coefficients, take each product value as a corresponding dequantization result of each group, and store the dequantization results of each group.

8. A data dequantization apparatus characterized by comprising: The application further provides a computer readable storage medium having stored thereon a computer program. The computer readable storage medium has stored thereon a computer program, and the computer program is executed by the processor to implement the steps of the data dequantization method according to any one of claims 1 to 6. The computer readable storage medium has stored thereon a computer program, and the computer program is executed by the processor to implement the steps of the data dequantization method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, ​

Images