Data inverse quantization method, device and equipment and computer readable storage medium

Abstract

The invention relates to the technical field of deep learning, and discloses a data inverse quantization method, device and equipment and a computer readable storage medium, and the method comprises the steps: reading the bias of the int4 type and the quantization coefficient of the fp16 type from a cache according to the position information corresponding to the bias of the int4 type of the current group and the position information corresponding to the quantization coefficient of the fp16 type of the current group; and reading the weight of the int4 type from the cache according to the position information corresponding to the weight of the int4 type. And when the weights of all the int4 types in the current group are read, processing the weights of all the int4 types in the current group, the bias of the int4 types and the quantization coefficient of the fp16 type according to an inverse quantization processing rule so as to obtain an inverse quantization result. And repeatedly executing the operation until the inverse quantization calculation of all the data in the cache is completed. And 4-bit inverse quantization is realized on a processor which does not support a 4-bit data type.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a data dequantization method, device, equipment and computer-readable storage medium. Background Art

[0002] Model quantization is a common technology in the field of deep learning, which is used to reduce the size of the model and the demand for computing resources, thereby improving the operating efficiency of the model. Model quantization refers to the process of converting the parameters in the neural network model from high-precision floating-point numbers to low-precision integers. Common high-precision floating-point numbers include fp32 and fp16; low-precision integers include int8 and int4. Dequantization is an important step in the model reasoning process. When the deployed model performs reasoning, it is necessary to convert the quantized low-precision integer parameters back to high-precision floating-point parameters through dequantization.

[0003] Int8 is a standard data type on most hardware platforms, so 8-bit quantization is widely used. In order to further reduce the storage requirements and computational complexity of the model, 4-bit quantization was further developed. There is no difference in the difficulty of implementing 8-bit quantization and 4-bit quantization. However, the hardware platforms on which the quantized model is deployed are diverse, and not all hardware platforms support 4-bit calculations.

[0004] It can be seen that how to implement 4-bit inverse quantization on a processor that does not support 4-bit data types, so as to give full play to the advantages of 4-bit quantization of small storage space and high computational efficiency, is a problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data dequantization method, device, equipment and computer-readable storage medium, which can solve the application problem of 4-bit quantization on a hardware platform that does not support 4-bit data type.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a data dequantization method, comprising:

[0007] When the inverse quantization calculation is not completed, the int4 type bias is read from the cache according to the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group;

[0008] In the case that all the int4 type weights in the current group have not been read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights;

[0009] When all int4 type weights in the current group are read out, all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed according to the dequantization processing rules to obtain the dequantization result corresponding to the current group;

[0010] When the current group is not the last group, the position information corresponding to each type of data of the next group is used as the latest position information of each type of data of the current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache until the inverse quantization calculation of all data in the cache is completed; wherein each type of data includes at least int4 type bias, fp16 type quantization coefficient and int4 type weight.

[0011] On the one hand, according to the position information corresponding to the int4 type bias of the current group, the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache including:

[0012] According to the starting address and starting offset corresponding to the int4 type offset of the current group, the int4 type offset of the current group is read from the cache, and the int4 type offset of the current group is stored;

[0013] According to the accumulated address of the int4 type offset of the current group, the address and offset corresponding to the int4 type offset of the next group are determined;

[0014] According to the starting address and starting offset corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient of the current group is read out, and the fp16 type quantization coefficient of the current group is stored;

[0015] According to the accumulated addresses of the fp16 type quantization coefficients of the current group, the addresses and offsets corresponding to the fp16 type quantization coefficients of the next group are determined.

[0016] On the one hand, in the case where all the int4 type weights in the current group are not read out, according to the position information corresponding to the int4 type weights, reading the int4 type weights from the cache includes:

[0017] According to the starting address and starting offset corresponding to the int4 type weight, read the int4 type weight contained in the current group from the cache, and store the int4 type weight of the current group;

[0018] According to the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the next group of int4 type weights are calculated.

[0019] On the one hand, according to the dequantization processing rules, all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed to obtain the dequantization results corresponding to the current group, including:

[0020] Convert the int4 type weight to the int8 type weight; the int8 type weight includes the high-order int8 type weight and the low-order int8 type weight;

[0021] Convert the int4 type bias to the int8 type bias; wherein the int8 type bias includes the high-order int8 type bias and the low-order int8 type bias;

[0022] According to the vector subtraction instruction, calculate the first difference data of the int8 type of the weight of the low-order int8 type and the int8 type of the bias of the low-order int8 type;

[0023] Calling a data type conversion instruction to convert the lower 256 bits of the first difference data into the first lower bits of the fp16 type, and storing the first lower bits of the fp16 type;

[0024] According to the shift instruction, the first difference data is right-shifted by 256 bits, and the first difference data after the right shift is converted into the first high-order data of the fp16 type, and the first high-order data of the fp16 type is stored;

[0025] According to the vector subtraction instruction, calculate the second difference data of the int8 type of the weight of the high-order int8 type and the bias of the high-order int8 type;

[0026] Calling a data type conversion instruction to convert the lower 256-bit data in the second difference data into the second lower data of the fp16 type, and storing the second lower data of the fp16 type;

[0027] According to the shift instruction, the second difference data is right-shifted by 256 bits, and the second difference data after the right-shift is converted into the second high-order data of the fp16 type, and the second high-order data of the fp16 type is stored;

[0028] The product of the fp16 type quantization coefficient and the corresponding fp16 type data is calculated, each product value is used as the dequantization result corresponding to each group, and the dequantization result corresponding to each group is stored.

[0029] On the one hand, converting the weights of type int4 to type int8 includes:

[0030] Parse the 512-bit data in the int4 type weight into 64 int8 type weights;

[0031] According to the vector right shift instruction, each int8 type weight is shifted right and the high-order int8 type weight obtained after the right shift is stored;

[0032] According to the vector left shift instruction, the weights of each int8 type are shifted left;

[0033] According to the vector right shift instruction, each int8 type weight after the left shift is right shifted, and the low-order int8 type weight obtained after the right shift is stored.

[0034] On the one hand, converting the bias of type int4 to the bias of type int8 includes:

[0035] Parse the 512-bit data in the int4 type offset into 64 int8 type offsets;

[0036] According to the vector right shift instruction, each int8 type offset is shifted right and the high-order int8 type offset obtained after the right shift is stored;

[0037] According to the vector left shift instruction, each int8 type offset is shifted left;

[0038] According to the vector right shift instruction, each int8 type offset after the left shift is right shifted, and the low-order int8 type offset obtained after the right shift is stored.

[0039] On the one hand, calculating the product of the fp16 type quantization coefficient and the corresponding fp16 type data, using each product value as the dequantization result corresponding to each group, and storing the dequantization result corresponding to each group includes:

[0040] Calculate the product of the fp16 type quantization coefficient and the first low-order data of the fp16 type, use the product value as the first group of corresponding inverse quantization results, and store the first group of corresponding inverse quantization results;

[0041] Calculate the product of the quantization coefficient of the fp16 type and the first high-order data of the fp16 type, use the product value as the dequantization result corresponding to the second group, and store the dequantization result corresponding to the second group;

[0042] Calculate the product of the fp16 type quantization coefficient and the second lowest bit data of the fp16 type, use the product value as the dequantization result corresponding to the third group, and store the dequantization result corresponding to the third group;

[0043] The product of the fp16 type quantization coefficient and the second high-order data of the fp16 type is calculated, the product value is used as the dequantization result corresponding to the fourth group, and the dequantization result corresponding to the fourth group is stored.

[0044] The embodiment of the present invention further provides a data dequantization device, comprising a first reading unit, a second reading unit, a dequantization unit and a serving unit;

[0045] The first reading unit is used to read the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the inverse quantization calculation is not completed; and read the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group;

[0046] The second reading unit is used to read the int4 type weights from the cache according to the position information corresponding to the int4 type weights when all the int4 type weights in the current group are not read out;

[0047] The dequantization unit is used to process all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group according to the dequantization processing rules after reading out all int4 type weights in the current group, so as to obtain the dequantization result corresponding to the current group;

[0048] The as unit is used to, when the current group is not the last group, use the position information corresponding to each type of data of the next group as the latest position information of each type of data of the current group, and return the position information corresponding to the int4 type bias of the current group, and read the int4 type bias from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, read the fp16 type quantization coefficient from the cache until the inverse quantization calculation of all data in the cache is completed; wherein each type of data includes at least int4 type bias, fp16 type quantization coefficient and int4 type weight.

[0049] On the one hand, the first reading unit is used to read the int4 type offset of the current group from the cache according to the starting address and starting offset corresponding to the int4 type offset of the current group, and store the int4 type offset of the current group;

[0050] According to the accumulated address of the int4 type offset of the current group, the address and offset corresponding to the int4 type offset of the next group are determined;

[0051] According to the starting address and starting offset corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient of the current group is read out, and the fp16 type quantization coefficient of the current group is stored;

[0052] According to the accumulated addresses of the fp16 type quantization coefficients of the current group, the addresses and offsets corresponding to the fp16 type quantization coefficients of the next group are determined.

[0053] On the one hand, the second reading unit is used to read the int4 type weights contained in the current group from the cache according to the starting address and starting offset corresponding to the int4 type weights, and store the int4 type weights of the current group;

[0054] According to the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the next group of int4 type weights are calculated.

[0055] On the one hand, the dequantization unit includes a first conversion subunit, a second conversion subunit, a first subtraction subunit, a third conversion subunit, a first preservation subunit, a first shift subunit, a fourth conversion subunit, a second preservation subunit, a second subtraction subunit, a fifth conversion subunit, a third preservation subunit, a second shift subunit, a sixth conversion subunit, a fourth preservation subunit and a multiplication subunit;

[0056] The first conversion subunit is used to convert the weight of the int4 type into the weight of the int8 type; wherein the weight of the int8 type includes the high-order int8 type weight and the low-order int8 type weight;

[0057] The second conversion subunit is used to convert the int4 type bias into an int8 type bias; wherein the int8 type bias includes a high-order int8 type bias and a low-order int8 type bias;

[0058] The first subtraction subunit is used to calculate the first difference data of the int8 type of the weight of the low-order int8 type and the int8 type of the bias of the low-order int8 type according to the vector subtraction instruction;

[0059] The third conversion sub-unit is used to call a data type conversion instruction to convert the lower 256-bit data in the first difference data into the first lower-bit data of the fp16 type;

[0060] The first storage subunit is used to store the first low-order data of the fp16 type;

[0061] The first shift subunit is used to right shift the first difference data by 256 bits according to a shift instruction;

[0062] The fourth conversion subunit is used to convert the first difference data after right shift into the first high-order data of fp16 type;

[0063] The second storage subunit is used to store the first high-order data of the fp16 type;

[0064] The second subtraction subunit is used to calculate the second difference data of the int8 type of the weight of the high-order int8 type and the bias of the high-order int8 type according to the vector subtraction instruction;

[0065] The fifth conversion subunit is used to call a data type conversion instruction to convert the lower 256-bit data in the second difference data into the second lower data of the fp16 type;

[0066] The third storage subunit is used to store the second low-order data of the fp16 type;

[0067] The second shift subunit is used to right shift the second difference data by 256 bits according to the shift instruction;

[0068] The sixth conversion subunit is used to convert the second difference data after right shift into the second high-order data of fp16 type;

[0069] The fourth storage subunit is used to store the second high-order data of the fp16 type;

[0070] The multiplication subunit is used to calculate the product of the fp16 type quantization coefficient and its corresponding fp16 type data, use each product value as the dequantization result corresponding to each group, and store the dequantization result corresponding to each group.

[0071] On the one hand, the first conversion subunit is used to parse 512-bit data in the int4 type weight into 64 int8 type weights;

[0072] According to the vector right shift instruction, each int8 type weight is shifted right and the high-order int8 type weight obtained after the right shift is stored;

[0073] According to the vector left shift instruction, the weights of each int8 type are shifted left;

[0074] According to the vector right shift instruction, each int8 type weight after the left shift is right shifted, and the low-order int8 type weight obtained after the right shift is stored.

[0075] On the one hand, the second conversion subunit is used to parse 512-bit data in the int4 type offset into 64 int8 type offsets;

[0076] According to the vector right shift instruction, each int8 type offset is shifted right and the high-order int8 type offset obtained after the right shift is stored;

[0077] According to the vector left shift instruction, each int8 type offset is shifted left;

[0078] According to the vector right shift instruction, each int8 type offset after the left shift is right shifted, and the low-order int8 type offset obtained after the right shift is stored.

[0079] On the one hand, the multiplication subunit is used to calculate the product of the fp16 type quantization coefficient and the first low-order data of the p16 type, use the product value as the first group of corresponding dequantization results, and store the first group of corresponding dequantization results;

[0080] Calculate the product of the quantization coefficient of the fp16 type and the first high-order data of the fp16 type, use the product value as the dequantization result corresponding to the second group, and store the dequantization result corresponding to the second group;

[0081] Calculate the product of the fp16 type quantization coefficient and the second lowest bit data of the fp16 type, use the product value as the dequantization result corresponding to the third group, and store the dequantization result corresponding to the third group;

[0082] The product of the fp16 type quantization coefficient and the second high-order data of the fp16 type is calculated, the product value is used as the dequantization result corresponding to the fourth group, and the dequantization result corresponding to the fourth group is stored.

[0083] The embodiment of the present invention further provides a data dequantization device, including:

[0084] Memory for storing computer programs;

[0085] A processor is used to execute the computer program to implement the steps of the above-mentioned data dequantization method.

[0086] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data dequantization method are implemented.

[0087] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data dequantization method when executed by a processor.

[0088] It can be seen from the above technical solution that when the inverse quantization calculation is not completed, the int4 type bias is read from the cache according to the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group. When all the int4 type weights in the current group are not read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights. When all the int4 type weights in the current group are read out, it means that the data corresponding to the current group has been read out. At this time, all the int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group can be processed according to the inverse quantization processing rules to obtain the inverse quantization result corresponding to the current group. In the case where the current group is not the last group, the position information corresponding to each type of data of the next group is used as the position information of each type of data of the latest current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache until the dequantization calculation of all data in the cache is completed; wherein, each type of data includes at least the int4 type bias, the fp16 type quantization coefficient and the int4 type weight. The beneficial effect of the present invention is that the data required for performing the dequantization operation can be obtained in sequence in a cyclic manner, including the int4 type bias, the fp16 type quantization coefficient and the int4 type weight in the same group. According to the int4 type data storage method, the dequantization processing rules are set, and 4-bit dequantization can be implemented on a processor that does not support 4-bit data types, thereby giving full play to the advantages of 4-bit quantization occupying small storage space and high computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0090] Figure 1 A flow chart of a data dequantization method provided by an embodiment of the present invention;

[0091] Figure 2 A schematic diagram of a three-level cache provided by an embodiment of the present invention;

[0092] Figure 3A flowchart of a method for performing dequantization processing on an int4 type weight, an int4 type bias, and an fp16 type quantization coefficient provided by an embodiment of the present invention;

[0093] Figure 4 A schematic diagram of int type data storage and parsing provided by an embodiment of the present invention;

[0094] Figure 5 A schematic diagram of the structure of a data dequantization device provided by an embodiment of the present invention;

[0095] Figure 6 A structural diagram of a data dequantization device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0098] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0099] Model quantization can significantly reduce the size of model files, which is especially important for devices with limited storage. Using quantized lower-precision data types can speed up calculations, especially on hardware that supports low-precision calculations. Quantization can reduce the power required during calculations, which is especially important for battery-powered devices.

[0100] The purpose of dequantization is to restore the accuracy of the model as much as possible while maintaining the model inference speed. Dequantization is usually done by multiplying the scale factor (Scale) used during quantization and adding the offset (Zero Point). For linear quantization, the dequantization operation can be expressed as: floating point number = (integer − offset) × scale factor.

[0101] For 4-bit quantization, each model parameter is represented by 4 bits, which occupies less storage space than 8 bits. Due to the use of smaller data types, 4-bit quantization can further reduce the amount of data transmission during the calculation process, thereby improving calculation efficiency. On hardware that supports low-precision calculations, 4-bit quantization can bring faster inference speeds. However, the hardware platforms for quantized model deployment vary, including graphics processors (Graphic Process Unit, GPU), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), etc., and not all hardware platforms support 4-bit calculations. On hardware that does not support 4-bit calculations, in order to achieve dequantization during the reasoning process, it may be necessary to convert to a higher-precision representation, which may offset some of the efficiency advantages.

[0102] Therefore, the embodiments of the present invention provide a data dequantization method, device, equipment and computer-readable storage medium, which implements the processing flow of two-dimensional grouped quantized int4 data based on a dual-channel vector processor, can implement 4-bit dequantization on a processor that does not support 4-bit data types, and give full play to the advantages of 4-bit quantization in that it occupies a small storage space and has high computational efficiency. It can be applied to processors with different vector register widths, different numbers of vector calculation channels, or those that support SIMD.

[0103] In the embodiment of the present invention, the 4-bit weight is of int4 type, which is obtained group by group from the original fp16 type weight through a linear quantization method. After quantization, in addition to the int4 type weight (weight_int4), there are also int4 type bias (zero_int4) and fp16 type quantization coefficient (scale_fp16). Weight_int4 represents the int4 type weight, zero_int4 represents the int4 type bias, and scale_fp16 represents the fp16 type quantization coefficient.

[0104] Assume that the dimension of the original fp16 type weight (weight_fp16) is [K, N], and the group size is group_size, then the dimension of the quantized weight_int4 is [K, N]; the dimension of zero_int4 is [K / group_size, N]; the dimension of scale_fp16 is [K / group_size, N]. The calculation formula for inverse quantization is weight_fp16=(weight_int4-zero_int4)*scale_fp16. weight_fp16 represents the weight of fp16 type.

[0105] Different models have different values ​​of K and N. For example, in the weight of llama3-70b Q, K=N=8192. group_size can also be set as needed, usually 128.

[0106] Next, a data dequantization method provided by an embodiment of the present invention is introduced in detail. Figure 1 A flowchart of a data dequantization method provided by an embodiment of the present invention, the method comprising:

[0107] S101: When the inverse quantization calculation is not completed, the int4 type offset is read from the cache according to the position information corresponding to the int4 type offset of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group.

[0108] In the embodiment of the present invention, the processor used may be a multi-core processor having multiple computing cores therein, and the cache space accessible by the processor is divided into three levels. Figure 2 A schematic diagram of a three-level cache provided in an embodiment of the present invention, wherein the three-level cache includes an L3 cache, an L2 cache, and an L1 cache. The third-level L3 cache is a high bandwidth memory (HBM), the second-level L2 cache is a storage space shared between the cores in the processor chip, and the first-level L1 cache is a storage space exclusive to each core. From L3 to L1, the storage space is getting smaller and smaller, but the storage bandwidth is getting larger and larger, and data transmission is getting faster and faster.

[0109] Initially, the inputs weight_int4, zero_int4, and scale_fp16 required for the dequantization operation are stored in the L3 cache. These input data need to be moved to the L1 cache before the kernel starts the operation. Since the data size of weight_int4 is large and the bandwidth of the direct data storage (Data Memory Access, DMA) channel of L3->L1 is small, L3->L1 means moving the data directly from the L3 cache to the L1 cache. Frequently moving weight_int4 from the L3 cache to the L1 cache will reduce the overall computing efficiency; at the same time, in order to enable each kernel to calculate in parallel, weight_int4 can first be moved from the L3 cache to the L2 cache, and then each kernel will move part of the weight_int4 data required for the calculation from the L2 cache to the L1 cache. The data size of zero_int4 and scale_fp16 is small, and each computing kernel directly moves part of the data required for the calculation from the L3 cache to the L1 cache.

[0110] Assume that the number of cores is m, and the dimension size of weight_int4 processed by each core is [k_part, n_part], then n_part=N / m, and then k_part is calculated based on the size of L1 cache that can be allocated in the core, and k_part must be an integer multiple of group_size. The corresponding dimensions of scale_fp16 and zero_int4 are [scale_k_part, n_part], where scale_k_part=k_part / group_size.

[0111] Each core corresponds to an L1 cache. For example, if the allocatable L1 cache is 256KB, it is necessary to ensure that k_part*n_part / (2*(the amount of data in weight_int4))<=256k. The value combination of k_part and n_part can be selected based on this restriction.

[0112] In the embodiment of the present invention, the processor used supports dual-channel vector operations (two sets of vector operations are performed simultaneously under the control of the same instruction), the width of the vector register is 512 bits; it supports data types with widths of 8 bits, 16 bits and 32 bits, and the 512-bit data in the vector register can be interpreted as several elements in units of 8 bits, 16 bits or 32 bits. It supports one instruction to perform the same operation on all vector elements; it supports setting the starting address and offset of access data for each vector channel, the starting address is the address of the first vector access, and the address is accumulated and the offset is realized after each access is completed to realize the address jump.

[0113] In practical applications, the position information corresponding to the int4 type offset and the position information corresponding to the fp16 type quantization coefficient can be pre-stored. The position information may include an address and an offset. The address in the initial state can be regarded as the starting address, and the offset in the initial state can be regarded as the starting offset.

[0114] For reading the int4 type offset, the int4 type offset of the current group can be read from the cache according to the starting address and starting offset corresponding to the int4 type offset of the current group. In order to facilitate subsequent calls, the int4 type offset of the current group can be stored in the offset vector register (vr_zero_int4). vr_zero_int4 represents the offset vector register, which is used to store the int4 type offset. After each access is completed, the address and offset corresponding to the int4 type offset of the next group can be determined according to the accumulated address of the int4 type offset of the current group;

[0115] For reading fp16 type quantization coefficients, the fp16 type quantization coefficients of the current group can be read out according to the starting address and starting offset corresponding to the fp16 type quantization coefficients of the current group. In order to facilitate subsequent calls, the fp16 type quantization coefficients of the current group can be stored in the coefficient vector register (vr_ scale_fp16). vr_ scale_fp16 represents a coefficient vector register, which is used to store fp16 type quantization coefficients. After each access is completed, the address and offset corresponding to the fp16 type quantization coefficients of the next group can be determined according to the accumulated address of the fp16 type quantization coefficients of the current group.

[0116] S102: when all the int4 type weights in the current group have not been read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights.

[0117] In the embodiment of the present invention, a three-layer loop is used to sequentially read the int4 type bias, the fp16 type quantization coefficient, and the int4 type weight.

[0118] The outermost loop is a loop of n_part, which is used to process different columns. Since the width of the vector register is 512 bits, it can store 128 int4 data. Therefore, the outermost loop processes 128 elements each time. The next loop jumps to the next group of 128 elements in the same row. The number of loops is n_part>>7. n_part>>7 means right shift by seven bits, which is equivalent to n_part / 128.

[0119] The middle layer loop is scale_k_part, which is used to process different rows. Each loop processes a group, and the next loop jumps to the next adjacent group. The number of loops is scale_k_part. Through the middle layer loop, the int4 type bias and fp16 type quantization coefficient corresponding to each group can be read.

[0120] The innermost loop is the loop of group_size, which is used to process different rows. By using the feature of the processor core supporting dual-channel vector operations, the two channels can perform calculations on two adjacent rows respectively. Each loop can process weight_int4 of two adjacent rows within the same group. The next loop jumps to the next two adjacent rows, and the number of loops is group_size>>1. group_size>>1 means shifting right by one bit, which is equivalent to group_size / 2. The innermost loop processes group_size rows of weight_int4 within the same group, processing two rows at the same time; there is only one row of zero_int4 and scale_fp16 corresponding to the weight_int4 of a group.

[0121] In an embodiment of the present invention, the starting address and starting offset corresponding to the int4 type weights can be pre-stored. According to the starting address and starting offset corresponding to the int4 type weights, the int4 type weights contained in the current group can be read from the cache, and the int4 type weights of the current group can be stored; according to the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the next group of int4 type weights can be calculated. All int4 type weights in the current group can be read through the innermost loop.

[0122] S103: After reading out all int4 type weights in the current group, all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed according to the inverse quantization processing rules to obtain the inverse quantization result corresponding to the current group.

[0123] The dequantization processing rules may be set according to the data storage method of the int4 type.

[0124] By performing dequantization processing on all int4 type weights, int4 type biases and fp16 type quantization coefficients in the same group, a dequantization result can be obtained, and the dequantization result is a fp16 type weight.

[0125] The calculation formula for inverse quantization is weight_fp16=(weight_int4-zero_int4)*scale_fp16.

[0126] Since weight_int4 and zero_int4 are int4 type data, when performing dequantization, the data in the weight vector register (vr_weight_int4) and the data in vr_zero_int4 need to be parsed and converted to int8 type data in the same way. vr_weight_int4 represents a weight vector register, which is used to store int4 type weights.

[0127] In order to multiply with scale_fp16, when calculating the difference between weight_int4 and zero_int4, you need to convert the int8 data to fp16 data. The calculated difference is multiplied by scale_fp16 to get the final weight_fp16. The specific execution process of dequantization can be seen in Figure 3 About Figure 3 The content is described in detail later.

[0128] S104: When the current group is not the last group, the position information corresponding to each type of data of the next group is used as the latest position information of each type of data of the current group, and the position information corresponding to the int4 type offset of the current group is returned, and the int4 type offset is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache, until the inverse quantization calculation of all data in the cache is completed.

[0129] If the current group is not the last group, it means that there are still unprocessed groups. Since the addresses and offsets corresponding to various types of data in the next cycle are determined based on the accumulated addresses of this cycle in each loop, if the current group is not the last group, the addresses and offsets corresponding to various types of data in the next group can be used as the latest addresses and offsets of various types of data in the current group, and the address and offset corresponding to the int4 type offset of the current group are returned, and the int4 type offset is read from the cache; the fp16 type quantization coefficient is read from the cache according to the address and offset corresponding to the fp16 type quantization coefficient of the current group, until the inverse quantization calculation of all data in the cache is completed. Among them, various types of data include at least int4 type offset, fp16 type quantization coefficient and int4 type weight.

[0130] The above introduction describes that the inverse quantization calculation is implemented using vector operations, which is divided into three layers of loops. The outermost loop processes 128 elements each time. The middle loop processes one group each time. The innermost loop processes the int4 type weights within the same group. Each loop processes two rows of int4 type weights at the same time. Next, the reading process of various types of data will be introduced.

[0131] The reading of weight_int4 is implemented in the innermost loop. Before the loop starts, the starting addresses of the two vector channels reading weight_int4 are set to the starting addresses weight_addr of row 0 and weight_addr+(n_part>>7) of row 1 respectively. The two adjacent loops of the innermost loop process the 128 adjacent elements of the same two rows, and weight_int4 is stored continuously by row. Therefore, in the innermost loop, the offsets of the two channels reading weight_int4 are set to (n_part>>7)*2. After the innermost loop ends, the address of weight_int4 read has been accumulated to the first address of the two rows of weight_int4 starting with the next group (weight_addr+group_size*(n_part>>7) and weight_addr+(group_size+1)*(n_part>>7)). The same rule can be followed to continue reading until the middle loop is completed. At this time, the weight_int4 of all rows corresponding to the current n_part loop is processed, and the addresses of weight_int4 read by the two vector channels are accumulated to weight_addr+k_part*(n_part>>7) and weight_addr+(k_part+1)*(n_part>>7). Then jump to the next n_part loop.

[0132] In order to make the address of reading weight_int4 correct, the offset of the two vector channels reading weight_int4 needs to be set to -(k_part*(n_part>>7)-1) before the next n_part loop starts, so that the address of the next reading of weight_int4 is weight_addr+1 and weight_addr+(n_part>>7)+1, and the next group of 128 weight_int4 can be read out. The read data is saved in the weight vector register.

[0133] The reading of zero_int4 is implemented in the middle layer loop. Before the loop starts, the starting addresses of the two vector channels reading zero_int4 are set to the starting address zero_addr of row 0. Since the next middle layer loop processes the weight_int4 of the next group, which corresponds to the zero_int4 of the next row, the offset of reading zero_int4 is set to n_part>>7 inside the middle layer loop. At the end of the middle layer loop, the address of reading zero_int4 has been accumulated to zero_addr+(k_part / group_size)*(n_part>>7), and then jumps to the next n_part loop. In order to make the address of reading zero_int4 correct, the offset of reading zero_int4 needs to be set to -((k_part / group_size)*(n_part>>7)-1) before the next n_part loop starts. The address of reading zero_int4 next time becomes zero_addr+1. The read data is saved in the offset vector register.

[0134] The reading of scale_fp16 is implemented in the middle layer loop. Before the loop starts, the starting addresses of the two vector channels reading scale_fp16 are set to the starting address scale_addr of row 0. Since in the innermost loop, each reading of weight_int4 can read 128 int4 type weights, while the middle layer loop can only read 32 fp16 type scales at a time when reading scale_fp16, it is necessary to read scale_fp16 four times in a row. The address offsets of the first three reads are all set to 1. At the end of the third read, the address is accumulated to scale_addr+3, and the fourth read is read from this address. However, the next middle layer loop needs to read the scale_fp16 corresponding to the next group, and its address is scale_addr+(n_part>>5). Therefore, the offset address of the fourth read should be set to (n_part>>5)-3. The address after the fourth read needs to jump to scale_addr+(n_part)>>5. The address of the fourth read is scale_addr+3, so the offset needs to be set to the upper value.

[0135] At the end of the middle layer loop, the address of scale_fp16 has been accumulated to scale_addr+(k_part / group_size)*(n_part>>5), and then jumps to the next n_part loop. In order to make the address of scale_fp16 correct, the offset of scale_fp16 reading needs to be set to -((k_part / group_size)*(n_part>>5)-4) before the next n_part loop starts, and the address of scale_fp16 reading next time becomes scale_addr+4. Since four reads are involved, four coefficient vector registers can be set to store the corresponding data respectively. In practical applications, the read data can be saved in the first coefficient vector register (vr_scale_fp16_0), the second coefficient vector register (vr_scale_fp16_1), the third coefficient vector register (vr_scale_fp16_2), and the fourth coefficient vector register (vr_scale_fp16_3) in sequence. vr_scale_fp16_0 represents the first coefficient vector register, vr_scale_fp16_1 represents the second coefficient vector register, vr_scale_fp16_2 represents the third coefficient vector register, and vr_scale_fp16_3 represents the fourth coefficient vector register.

[0136] The storage of the dequantization results is implemented in the innermost loop. The innermost loop processes two rows of weight_int4 at the same time, and will also get two rows of weight_fp16. Before the start of the loop, the starting addresses of the two vector channels storing weight_fp16 are set to the starting address out_addr of the 0th row and the starting address out_addr+(n_part>>5) of the 1st row respectively. In the innermost loop, each loop processes 128 weight_int4s and gets 128 weight_fp16s, which requires 4 vector registers to save. When saving the weight_fp16 in the first 3 vector registers, the address offset is set to 1. After saving, the address is accumulated to out_addr+3, and the weight_fp16 in the 4th vector register is saved to this address. Since the next execution of the innermost loop will process the next two rows of weights, the calculation results should also be saved in the next two rows. Therefore, when saving weight_fp16 in the fourth vector register, the offset address is set to 2*(n_part>>5)-3, and the address jumps to out_addr+(n_part>>5).

[0137] In the embodiment of the present invention, the three-level cache structure of the processor is used to improve the data transmission efficiency. A specific int4 data storage method is designed according to the width of the vector register to improve the efficiency of int4 data analysis. According to the loop method introduced above, the inverse quantization calculation can be completed quickly and accurately.

[0138] After the inverse quantization calculation is completed, the int4 type bias, fp16 type quantization coefficient, int4 type weight and their corresponding inverse quantization results can be stored in the form of a correspondence list. When the inverse quantization operation is performed on the same type of data later, the inverse quantization result matching the current data can be directly called from the correspondence list.

[0139] In the embodiment of the present invention, by recording the dequantization results corresponding to common types of data, it is avoided to perform dequantization operations on the same data every time, thereby effectively improving the data processing efficiency.

[0140] It can be seen from the above technical solution that when the inverse quantization calculation is not completed, the int4 type bias is read from the cache according to the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group. When all the int4 type weights in the current group are not read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights. When all the int4 type weights in the current group are read out, it means that the data corresponding to the current group has been read out. At this time, all the int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group can be processed according to the inverse quantization processing rules to obtain the inverse quantization result corresponding to the current group. In the case where the current group is not the last group, the position information corresponding to each type of data of the next group is used as the position information of each type of data of the latest current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache until the dequantization calculation of all data in the cache is completed; wherein, each type of data includes at least the int4 type bias, the fp16 type quantization coefficient and the int4 type weight. The beneficial effect of the present invention is that the data required for performing the dequantization operation can be obtained in sequence in a cyclic manner, including the int4 type bias, the fp16 type quantization coefficient and the int4 type weight in the same group. According to the int4 type data storage method, the dequantization processing rules are set, and 4-bit dequantization can be implemented on a processor that does not support 4-bit data types, thereby giving full play to the advantages of 4-bit quantization occupying small storage space and high computational efficiency.

[0141] Figure 3 A flowchart of a method for performing dequantization processing on an int4 type weight, an int4 type bias, and an fp16 type quantization coefficient provided in an embodiment of the present invention, the method comprising:

[0142] S301: Convert the weight of the int4 type into the weight of the int8 type.

[0143] The int8 type weight may include a high-order int8 type weight and a low-order int8 type weight.

[0144] The weights of type int4 are stored in vr_weight_int4. When performing type conversion, the 512-bit data in vr_weight_int4 can be first parsed into 64 weights of type int8. Then, according to the vector right shift instruction, the high-order int8-type weights obtained by right-shifting each int8-type weight are saved in the weight high-order vector register (vr_weight_int8_hi). According to the vector left shift instruction, each int8-type weight is left-shifted; according to the vector right shift instruction, each int8-type weight after left shift is right-shifted, and the low-order int8-type weight obtained after right shift is saved in the weight low-order vector register (vr_weight_int8_lo). vr_weight_int8_hi represents the weight high-order vector register, and vr_weight_int8_lo represents the weight low-order vector register.

[0145] S302: Convert the int4 type offset to an int8 type offset.

[0146] The int8 type bias may include a high-order int8 type bias and a low-order int8 type bias.

[0147] The data in vr_zero_int4 is parsed and converted to int8 type using the same processing method as vr_weight_int4. The parsed data is saved in the bias high vector register (vr_ zero _int8_hi) and the bias low vector register (vr_ zero _int8_lo). vr_ zero _int8_hi represents the bias high vector register, which is used to save the 64th to 127th bias. vr_ zero _int8_lo represents the bias low vector register, which is used to save the 0th to 63rd bias.

[0148] When performing type conversion, the 512-bit data in the offset vector register can be parsed into 64 int8 offsets. According to the vector right shift instruction, the high-order int8 offsets obtained by right-shifting each int8 offset are saved in vr_ zero _int8_hi. According to the vector left shift instruction, each int8 offset is left-shifted; according to the vector right shift instruction, each int8 offset after left shift is right-shifted, and the low-order int8 offset obtained after right shift is saved in vr_ zero _int8_lo.

[0149] Figure 4A schematic diagram of int type data storage and parsing provided by an embodiment of the present invention. After zero_int4 is read out through the middle layer loop, the 128 int4 data stored in vr_zero_int4 need to be parsed out first. Because the processor used in the present invention does not support the int4 data type, the int4 type data needs to be converted to int8 type data for calculation.

[0150] First, use the vector shift instruction to parse the 512-bit data in vr_zero_int4 into 64 int8 data, and shift each int8 element right by 4 bits, and save the obtained data in vr_zero_int8_hi, thus obtaining 64 zeros stored in bit4~bit7, bit12~bit15...bit508~bit511. Then use the vector shift instruction to shift vr_zero_int4 left by 4 bits in int8 units, and the result is still retained in vr_zero_int4, thereby retaining the 64 zeros stored in bit0~bit3, bit4~bit7, bit8~bit11,... ..., bit504~bit507, bit508~bit511, but these 64 zeros are all stored in the upper 4 bits of each int8 element, and it is necessary to use the vector shift instruction to shift each int8 element right by 4 bits to move the valid data back to the lower 4 bits, and the result is saved in vr_zero_int8_lo.

[0151] In order to avoid adjusting the data order after data parsing is completed and to achieve the best efficiency, it is necessary to ensure that the data in vr_zero_int8_hi and vr_zero_int8_lo are stored sequentially, that is, the 0th to 63rd zeros are stored in vr_zero_int8_lo, and the 64th to 127th zeros are stored in vr_zero_int8_hi. Considering that the width of the vector register is 512 bits, when saving zero_int4, it is necessary to save 128 as a group, starting from the lowest bit, and save zero_0, zero_64, zero_1, zero_65...zero_63, zero_127.

[0152] S303: According to the vector subtraction instruction, calculate the first difference data of the int8 type of the low-order int8 type weight and the int8 type bias of the low-order int8 type.

[0153] To calculate the difference between weight and zero, you can use the vector subtraction instruction to calculate the difference between vr_weight_int8_lo and vr_zero_int8_lo, and the result is still saved in vr_weight_int8_lo. Use the vector subtraction instruction to calculate the difference between vr_weight_int8_hi and vr_zero_int8_hi, and the result is still saved in vr_weight_int8_hi.

[0154] For ease of distinction, the difference between vr_weight_int8_lo and vr_zero_int8_lo may be referred to as first difference data, and the difference between vr_weight_int8_hi and vr_zero_int8_hi may be referred to as second difference data.

[0155] S304: calling a data type conversion instruction to convert the lower 256 bits of the first difference data into the first lower bits of the fp16 type, and storing the first lower bits of the fp16 type.

[0156] In order to be able to multiply with the fp16 type scale, the int8 data in vr_weight_int8_lo needs to be converted to fp16 data first. The data type conversion instruction can be used to convert the 32 int8 data stored in the lower 256 bits of vr_weight_int8_lo into fp16 data, that is, the first low-order data, and save the first low-order data in the first fp16 weight register (vr_weight_fp16_lo_0). vr_weight_fp16_lo_0 represents the first fp16 weight register, which is used to store the 0th to 31st weights.

[0157] S305: According to the shift instruction, the first difference data is right-shifted by 256 bits, and the first difference data after the right-shift is converted into the first high-order data of the fp16 type, and the first high-order data of the fp16 type is stored.

[0158] Use the shift instruction to shift vr_weight_int8_lo right by 256 bits, thereby removing the lower 256 bits and retaining the upper 256 bits (the upper 256 bits are stored starting from the lowest bit), and then use the data type conversion instruction to convert the 32 int8 data stored in the upper 256 bits into the first high-order data. The first high-order data belongs to the fp16 type data, and the first high-order data is saved in the second fp16 weight register (vr_weight_fp16_lo_1). vr_weight_fp16_lo_1 represents the second fp16 weight register, which is used to save the 32nd to 63rd weights.

[0159] S306: According to the vector subtraction instruction, calculate the second difference data of the int8 type of the high-order int8 type weight and the int8 type bias of the high-order int8 type.

[0160] In the embodiment of the present invention, the difference between vr_weight_int8_hi and vr_zero_int8_hi may be calculated using the same method as the above-mentioned low-order int8 type weight and low-order int8 type bias.

[0161] S307: Calling a data type conversion instruction to convert the lower 256-bit data in the second difference data into the second lower data of the fp16 type, and storing the second lower data of the fp16 type.

[0162] After converting the lower 256 bits of the second difference data into the second lower data of the fp16 type, it is saved in the third fp16 weight register (vr_weight_fp16_hi_0). vr_weight_fp16_hi_0 represents the third fp16 weight register, which is used to save the 64th to 95th weights.

[0163] S308: According to the shift instruction, the second difference data is right-shifted by 256 bits, and the second difference data after the right-shift is converted into the second high-order data of the fp16 type, and the second high-order data of the fp16 type is stored.

[0164] The second highest bit data is stored in the fourth fp16 weight register (vr_weight_fp16_hi_1). vr_weight_fp16_hi_1 represents the fourth fp16 weight register, which is used to store the 96th to 127th weights.

[0165] S309: Calculate the product of the fp16 type quantization coefficient and the corresponding fp16 type data, use each product value as the dequantization result corresponding to each group, and store the dequantization result corresponding to each group.

[0166] Combined with the introduction of S104, it can be seen that there are four coefficient vector registers, namely the first coefficient vector register, the second coefficient vector register, the third coefficient vector register and the fourth coefficient vector register. There are also four fp16 weight registers corresponding to the quantization coefficients, namely the first fp16 weight register, the second fp16 weight register, the third fp16 weight register and the fourth fp16 weight register.

[0167] When performing a product operation on the fp16 type data recorded in the fp16 weight register and the fp16 type quantization coefficient recorded in each coefficient vector register, the product of the fp16 type quantization coefficient recorded in the first coefficient vector register and the first low-order data of the fp16 type recorded in the first fp16 weight register can be calculated, and the product value is used as the first group of corresponding dequantization results, and the first group of corresponding dequantization results is stored in the first dequantization register (vr_fp16_lo_0). vr_fp16_lo_0 represents the first dequantization register.

[0168] Calculate the product of the fp16 type quantization coefficient recorded in the second coefficient vector register and the first high-order data of the fp16 type recorded in the second fp16 weight register, use the product value as the second group of corresponding dequantization results, and store the second group of corresponding dequantization results in the second dequantization register (vr_fp16_lo_1). vr_fp16_lo_1 represents the second dequantization register.

[0169] Calculate the product of the fp16 type quantization coefficient recorded in the third coefficient vector register and the second lowest bit data of the fp16 type recorded in the third fp16 weight register, use the product value as the dequantization result corresponding to the third group, and store the dequantization result corresponding to the third group in the third dequantization register (vr_fp16_hi_0). vr_fp16_hi_0 represents the third dequantization register.

[0170] Calculate the product of the fp16 type quantization coefficient recorded in the fourth coefficient vector register and the second high-order data of the fp16 type recorded in the fourth fp16 weight register, use the product value as the dequantization result corresponding to the fourth group, and store the dequantization result corresponding to the fourth group in the fourth dequantization register (vr_fp16_hi_1). vr_fp16_hi_1 represents the fourth dequantization register.

[0171] For example, to calculate the product of weight and scale, use the vector multiplication instruction to calculate the product of vr_weight_fp16_lo_0 and vr_scale_fp16_0, save the result in vr_fp16_lo_0, and then save vr_weight_fp16_lo_0 to the L1 cache; use the same method to process vr_weight_fp16_lo_1 and vr_scale_fp16_1, vr_weight_fp16_hi_0 and vr_scale_fp16_2, vr_weight_fp16_hi_1 and vr_scale_fp16_3.

[0172] The present invention realizes the parsing and dequantization of int4 data based on a dual-channel vector processor, and can realize 4-bit dequantization on a processor that does not support 4-bit data types, giving full play to the advantages of 4-bit quantization in that it occupies a small storage space and has high computational efficiency. The three-level cache structure of the processor is utilized to improve the data transmission efficiency; a specific int4 data storage method is designed according to the width of the vector register, which improves the efficiency of int4 data parsing and avoids adjusting the data order after parsing. A complete processing flow based on dual-channel vector operations is given for grouped quantized two-dimensional int4 data.

[0173] Figure 5 A schematic diagram of the structure of a data dequantization device provided by an embodiment of the present invention includes a first reading unit 51, a second reading unit 52, a dequantization unit 53 and a serving unit 54;

[0174] The first reading unit 51 is used for reading the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the inverse quantization calculation is not completed; and reading the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group;

[0175] The second reading unit 52 is used to read the weights of the int4 type from the cache according to the position information corresponding to the weights of the int4 type when all the weights of the int4 type in the current group have not been read out;

[0176] The dequantization unit 53 is used to process all the int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group according to the dequantization processing rules after reading out all the int4 type weights in the current group, so as to obtain the dequantization result corresponding to the current group; wherein the dequantization processing rules are set according to the int4 type data storage method;

[0177] As unit 54, when the current group is not the last group, the position information corresponding to each type of data of the next group is used as the latest position information of each type of data of the current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read out from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read out from the cache until the inverse quantization calculation of all data in the cache is completed; wherein each type of data includes at least int4 type bias, fp16 type quantization coefficient and int4 type weight.

[0178] In some embodiments, the first reading unit is used to read the int4 type offset of the current group from the cache according to the starting address and the starting offset corresponding to the int4 type offset of the current group, and store the int4 type offset of the current group;

[0179] According to the accumulated address of the int4 type offset of the current group, the address and offset corresponding to the int4 type offset of the next group are determined;

[0180] According to the starting address and starting offset corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient of the current group is read out, and the fp16 type quantization coefficient of the current group is stored;

[0181] According to the accumulated addresses of the fp16 type quantization coefficients of the current group, the addresses and offsets corresponding to the fp16 type quantization coefficients of the next group are determined.

[0182] In some embodiments, the second reading unit is used to read the int4 type weights contained in the current group from the cache according to the starting address and starting offset corresponding to the int4 type weights, and store the int4 type weights of the current group;

[0183] According to the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the next group of int4 type weights are calculated.

[0184] In some embodiments, the inverse quantization unit includes a first conversion subunit, a second conversion subunit, a first subtraction subunit, a third conversion subunit, a first preservation subunit, a first shift subunit, a fourth conversion subunit, a second preservation subunit, a second subtraction subunit, a fifth conversion subunit, a third preservation subunit, a second shift subunit, a sixth conversion subunit, a fourth preservation subunit, and a multiplication subunit;

[0185] A first conversion subunit is used to convert the weight of the int4 type into the weight of the int8 type; wherein the weight of the int8 type includes the high-order int8 type weight and the low-order int8 type weight;

[0186] A second conversion subunit is used to convert the bias of the int4 type into the bias of the int8 type; wherein the bias of the int8 type includes a high-order int8-type bias and a low-order int8-type bias;

[0187] A first subtraction subunit, used for calculating, according to a vector subtraction instruction, first difference data of the int8 type of a low-order int8 type weight and a low-order int8 type bias;

[0188] The third conversion subunit is used to call the data type conversion instruction to convert the lower 256 bits of the first difference data into the first lower bits of the fp16 type;

[0189] A first storage subunit, used for storing the first low-order data of the fp16 type;

[0190] A first shift subunit, configured to right shift the first difference data by 256 bits according to a shift instruction;

[0191] A fourth conversion subunit, used for converting the first difference data after right shift into the first high-order data of fp16 type;

[0192] The second storage subunit is used to store the first high-order data of the fp16 type;

[0193] A second subtraction subunit, used for calculating, according to the vector subtraction instruction, a second difference data of the int8 type of a high-order int8 type weight and a high-order int8 type bias;

[0194] A fifth conversion subunit is used to call a data type conversion instruction to convert the lower 256-bit data in the second difference data into the second lower data of the fp16 type;

[0195] The third storage subunit is used to store the second low-order data of the fp16 type;

[0196] A second shift subunit, used for right shifting the second difference data by 256 bits according to the shift instruction;

[0197] A sixth conversion subunit, used for converting the second difference data after right shift into the second high-order data of fp16 type;

[0198] A fourth storage subunit is used to store the second high-order data of the fp16 type;

[0199] The multiplication subunit is used to calculate the product of the fp16 type quantization coefficient and the corresponding fp16 type data, use each product value as the dequantization result corresponding to each group, and store the dequantization result corresponding to each group.

[0200] In some embodiments, the first conversion subunit is used to parse 512 bits of data in the int4 type weight into 64 int8 type weights;

[0201] According to the vector right shift instruction, each int8 type weight is shifted right and the high-order int8 type weight obtained after the right shift is stored;

[0202] According to the vector left shift instruction, the weights of each int8 type are shifted left;

[0203] According to the vector right shift instruction, each int8 type weight after the left shift is right shifted, and the low-order int8 type weight obtained after the right shift is stored.

[0204] In some embodiments, the second conversion subunit is used to parse 512-bit data in the offset of the int4 type into 64 offsets of the int8 type;

[0205] According to the vector right shift instruction, each int8 type offset is shifted right and the high-order int8 type offset obtained after the right shift is stored;

[0206] According to the vector left shift instruction, each int8 type offset is shifted left;

[0207] According to the vector right shift instruction, each int8 type offset after the left shift is right shifted, and the low-order int8 type offset obtained after the right shift is stored.

[0208] In some embodiments, the multiplication subunit is used to calculate the product of the fp16 type quantization coefficient and the first low-order data of the fp16 type, use the product value as the first group of corresponding dequantization results, and store the first group of corresponding dequantization results;

[0209] Calculate the product of the quantization coefficient of the fp16 type and the first high-order data of the fp16 type, use the product value as the dequantization result corresponding to the second group, and store the dequantization result corresponding to the second group;

[0210] Calculate the product of the fp16 type quantization coefficient and the second lowest bit data of the fp16 type, use the product value as the dequantization result corresponding to the third group, and store the dequantization result corresponding to the third group;

[0211] The product of the fp16 type quantization coefficient and the second high-order data of the fp16 type is calculated, the product value is used as the dequantization result corresponding to the fourth group, and the dequantization result corresponding to the fourth group is stored.

[0212] It can be seen from the above technical solution that when the inverse quantization calculation is not completed, the int4 type bias is read from the cache according to the position information corresponding to the int4 type bias of the current group; the fp16 type quantization coefficient is read from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group. When all the int4 type weights in the current group are not read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights. When all the int4 type weights in the current group are read out, it means that the data corresponding to the current group has been read out. At this time, all the int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group can be processed according to the inverse quantization processing rules to obtain the inverse quantization result corresponding to the current group. In the case where the current group is not the last group, the position information corresponding to each type of data of the next group is used as the position information of each type of data of the latest current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache until the dequantization calculation of all data in the cache is completed; wherein, each type of data includes at least the int4 type bias, the fp16 type quantization coefficient and the int4 type weight. The beneficial effect of the present invention is that the data required for performing the dequantization operation can be obtained in sequence in a cyclic manner, including the int4 type bias, the fp16 type quantization coefficient and the int4 type weight in the same group. According to the int4 type data storage method, the dequantization processing rules are set, and 4-bit dequantization can be implemented on a processor that does not support 4-bit data types, thereby giving full play to the advantages of 4-bit quantization occupying small storage space and high computational efficiency.

[0213] Figure 5 The description of the features in the corresponding embodiments can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0214] Figure 6 A structural diagram of a data dequantization device provided by an embodiment of the present invention, such as Figure 6 As shown, the data dequantization device includes: a memory 60 for storing a computer program;

[0215] The processor 61 is used to implement the steps of the data dequantization method in the above embodiment when executing a computer program.

[0216] The data dequantization device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0217] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0218] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein the computer program is loaded and executed by the processor 61, and the relevant steps of the data dequantization method disclosed in any of the above embodiments can be implemented. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. The data 603 may include but is not limited to the address and offset corresponding to the int4 type bias of the current group, the address and offset corresponding to the fp16 type quantization coefficient of the current group, the int4 type bias, the fp16 type quantization coefficient, the address and offset corresponding to the int4 type weight, the int4 type weight, etc.

[0219] In some embodiments, the data dequantization device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0220] Those skilled in the art will understand that Figure 6The structure shown in the figure does not constitute a limitation on the data dequantization device, and may include more or fewer components than those shown in the figure.

[0221] It is understandable that if the data dequantization method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, disk or optical disk and other media that can store program codes.

[0222] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data dequantization method are implemented.

[0223] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data dequantization method when executed by a processor.

[0224] The above is a detailed introduction to a data dequantization method, device, equipment, computer-readable storage medium and computer program product provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0225] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0226] The above is a detailed introduction to a data dequantization method, device, equipment, computer-readable storage medium and computer program product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A data dequantization method, characterized in that: include: When the inverse quantization calculation is not completed, the int4 type offset is read from the cache according to the position information corresponding to the int4 type offset of the current group; According to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache; In the case that all the int4 type weights in the current group have not been read out, the int4 type weights are read from the cache according to the position information corresponding to the int4 type weights; When all int4 type weights in the current group are read out, all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed according to the inverse quantization processing rules to obtain the inverse quantization result corresponding to the current group; When the current group is not the last group, the position information corresponding to each type of data of the next group is used as the latest position information of each type of data of the current group, and the position information corresponding to the int4 type bias of the current group is returned, and the int4 type bias is read from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache until the inverse quantization calculation of all data in the cache is completed; wherein each type of data includes at least int4 type bias, fp16 type quantization coefficient and int4 type weight.

2. The data dequantization method according to claim 1, characterized in that: According to the position information corresponding to the int4 type offset of the current group, the int4 type offset is read from the cache; According to the position information corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient is read from the cache including: According to the starting address and starting offset corresponding to the int4 type offset of the current group, the int4 type offset of the current group is read from the cache, and the int4 type offset of the current group is stored; According to the accumulated address of the int4 type offset of the current group, the address and offset corresponding to the int4 type offset of the next group are determined; According to the starting address and starting offset corresponding to the fp16 type quantization coefficient of the current group, the fp16 type quantization coefficient of the current group is read out, and the fp16 type quantization coefficient of the current group is stored; According to the accumulated addresses of the fp16 type quantization coefficients of the current group, the addresses and offsets corresponding to the fp16 type quantization coefficients of the next group are determined.

3. The data dequantization method according to claim 1, characterized in that: In the case that all int4 type weights in the current group are not read out, according to the position information corresponding to the int4 type weights, the int4 type weights are read from the cache including: According to the starting address and starting offset corresponding to the int4 type weight, read the int4 type weight contained in the current group from the cache, and store the int4 type weight of the current group; According to the accumulated address of the int4 type weights of the current group, the address and offset corresponding to the next group of int4 type weights are calculated.

4. The data dequantization method according to claim 1, characterized in that: According to the dequantization processing rules, all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group are processed to obtain the dequantization results corresponding to the current group, including: Convert the int4 type weight to the int8 type weight; the int8 type weight includes the high-order int8 type weight and the low-order int8 type weight; Convert the int4 type bias to the int8 type bias; wherein the int8 type bias includes the high-order int8 type bias and the low-order int8 type bias; According to the vector subtraction instruction, calculate the first difference data of the int8 type of the weight of the low-order int8 type and the int8 type of the bias of the low-order int8 type; Calling a data type conversion instruction to convert the lower 256 bits of the first difference data into the first lower bits of the fp16 type, and storing the first lower bits of the fp16 type; According to the shift instruction, the first difference data is right-shifted by 256 bits, and the first difference data after the right shift is converted into the first high-order data of the fp16 type, and the first high-order data of the fp16 type is stored; According to the vector subtraction instruction, calculate the second difference data of the int8 type of the weight of the high-order int8 type and the bias of the high-order int8 type; Calling a data type conversion instruction to convert the lower 256-bit data in the second difference data into the second lower data of the fp16 type, and storing the second lower data of the fp16 type; According to the shift instruction, the second difference data is right-shifted by 256 bits, and the second difference data after the right-shift is converted into the second high-order data of the fp16 type, and the second high-order data of the fp16 type is stored; The product of the fp16 type quantization coefficient and the corresponding fp16 type data is calculated, each product value is used as the dequantization result corresponding to each group, and the dequantization result corresponding to each group is stored.

5. The data dequantization method according to claim 4, characterized in that: Converting int4 weights to int8 weights involves: Parse the 512-bit data in the int4 type weight into 64 int8 type weights; According to the vector right shift instruction, each int8 type weight is shifted right and the high-order int8 type weight obtained after the right shift is stored; According to the vector left shift instruction, the weights of each int8 type are shifted left; According to the vector right shift instruction, the weights of each int8 type after the left shift are right shifted, and the low-order int8 type weights obtained after the right shift are stored.

6. The data dequantization method according to claim 4, characterized in that: Converting an int4 type bias to an int8 type bias includes: Parse the 512-bit data in the int4 type offset into 64 int8 type offsets; According to the vector right shift instruction, each int8 type offset is shifted right and the high-order int8 type offset obtained after the right shift is stored; According to the vector left shift instruction, each int8 type offset is shifted left; According to the vector right shift instruction, each int8 type offset after the left shift is right shifted, and the low-order int8 type offset obtained after the right shift is stored.

7. The data dequantization method according to claim 4, characterized in that: Calculating the product of the fp16 type quantization coefficient and its corresponding fp16 type data, using each product value as the dequantization result corresponding to each group, and storing the dequantization result corresponding to each group includes: Calculate the product of the fp16 type quantization coefficient and the first low-order data of the fp16 type, use the product value as the first group of corresponding inverse quantization results, and store the first group of corresponding inverse quantization results; Calculate the product of the quantization coefficient of the fp16 type and the first high-order data of the fp16 type, use the product value as the dequantization result corresponding to the second group, and store the dequantization result corresponding to the second group; Calculate the product of the fp16 type quantization coefficient and the second lowest bit data of the fp16 type, use the product value as the dequantization result corresponding to the third group, and store the dequantization result corresponding to the third group; The product of the fp16 type quantization coefficient and the second high-order data of the fp16 type is calculated, the product value is used as the dequantization result corresponding to the fourth group, and the dequantization result corresponding to the fourth group is stored.

8. A data dequantization device, characterized in that: It includes a first reading unit, a second reading unit, a dequantization unit and an as unit; The first reading unit is used to read the int4 type bias from the cache according to the position information corresponding to the int4 type bias of the current group when the inverse quantization calculation is not completed; and read the fp16 type quantization coefficient from the cache according to the position information corresponding to the fp16 type quantization coefficient of the current group; The second reading unit is used to read the int4 type weights from the cache according to the position information corresponding to the int4 type weights when all the int4 type weights in the current group are not read out; The dequantization unit is used to process all int4 type weights, int4 type biases and fp16 type quantization coefficients in the current group according to the dequantization processing rules after reading out all int4 type weights in the current group, so as to obtain the dequantization result corresponding to the current group; The as unit is used to, when the current group is not the last group, use the position information corresponding to each type of data of the next group as the latest position information of each type of data of the current group, and return the position information corresponding to the int4 type bias of the current group, and read the int4 type bias from the cache; according to the position information corresponding to the fp16 type quantization coefficient of the current group, read the fp16 type quantization coefficient from the cache until the inverse quantization calculation of all data in the cache is completed; wherein each type of data includes at least int4 type bias, fp16 type quantization coefficient and int4 type weight.

9. A data dequantization device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the data dequantization method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data dequantization method according to any one of claims 1 to 7 are implemented.

Images