A method based on 4-bit quantization optimization

Abstract

The application provides a method based on 4-bit quantization optimization, wherein a is a convolution calculation result, b is a quantized multiplication coefficient, shift_left is a first shift coefficient, shift_right is a second shift coefficient, res is a final result, data batch processing is performed by using instructions in a simd instruction set, and a quantization processing formula is as follows: S1, a maximum value is obtained: a1=max(a,0)(1); S2, a multiplication and shift result is obtained: ab=a1*b>>(15-shift_left)(2); S3, a result of arbitrary shift after multiplication is obtained: ab_shift=ab>>shift_right(3); ab_rem2=[ab-(ab_shift<<shift_right)]<1(4), dividend=1<< (15-shift_left)(5), res0=ab_shift+rem(7), S4, a minimum value is obtained: res=min(res0,15)-8(8), in formula (2), a1 and b are converted into 32 bits from 16 bits, and the ab value is 32 bits; the instructions include a maximum value instruction, a multiplication and shift instruction, a shift rounding instruction, or a specified selection instruction, a multiplication instruction and a shift instruction for realizing arbitrary shift after multiplication, and a minimum value instruction.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method based on 4-bit quantization optimization. Background Technology

[0002] In integrated circuit technology, existing chip manufacturers develop their own chips to keep pace with the times. However, in chip applications, each chip design presents its own specific needs and challenges. For example, chips produced by Beijing Junzheng Integrated Circuit Co., Ltd. (hereinafter referred to as Beijing Junzheng), such as the Beijing Junzheng T30 and T31 models, and other T and X series chips, support the SIMD instruction set. However, the T30 and T31 models have 128-bit registers, and the number of registers is limited to 32. Using more than 32 registers will cause previously loaded register data to be stored in memory. When the data is reused later, it needs to be reloaded, resulting in low efficiency. This problem has become an unavoidable consideration in optimization design.

[0003] Therefore, the drawback of the existing technology is that using C programs directly on existing chips, such as those produced by Beijing Junzheng, is very slow.

[0004] In addition, the following are commonly used terms in the prior art:

[0005] 1. SIMD instructions: Single Instruction Stream, Multiple Data Stream, meaning a single instruction can execute multiple data streams, thus improving program processing speed. In simpler terms, it's a vector-based computation. Different chips use different instruction sets.

[0006] 2. Feature Map: The result obtained after input data is processed by convolution is called a feature map (or output data). The result generated after data is processed by a fully connected layer is also called a feature map (or output data). The size of a feature map is generally expressed as length × width × depth, or 1 × depth. Depth is also called a channel. Summary of the Invention

[0007] To address the problems in the prior art, the purpose of this application is to achieve a significant increase in computing speed by employing this method. It solves the problem of existing chips having 128-bit registers and a limited number of registers. In particular, the method of this invention is suitable for vector instruction operations, and it has a significant effect on solving the problems of the Beijing Junzheng T30 and T31 chips.

[0008] Specifically, this invention provides a method based on 4-bit quantization optimization. In this method, let the convolution calculation result be a, the quantization multiplication coefficient be b, the first shift coefficient be shift_left, the second shift coefficient be shift_right, and the final result be res. Instructions from the SIMD instruction set are used to process the data in batches. The quantization processing formula is expressed as follows:

[0009] S1, obtain the maximum value: a1=max(a,0) (1)

[0010] S2, obtain the result of multiplication and shift: ab = a1 * b >> (15 - shift_left) (2)

[0011] S3, obtains the result of the multiplication and arbitrary shift:

[0012] ab_shift=ab>>shift_right (3)

[0013] ab_rem2=[ab-(ab_shift<<shift_right)]<<1 (4)

[0014] dividend=1<<(15-shift_left) (5)

[0015]

[0016] res0 = ab_shift + rem (7)

[0017] S4, obtain the minimum value: res = min(res0, 15) - 8 (8)

[0018] In formula (2), a1 and b need to be converted, transforming the original 16 bits into 32 bits. At this time, the value of ab is also 32 bits (the actual effective data is 16 bits). The instructions include the maximum value instruction, multiplication shift instruction, shift and rounding instruction, or, in order to achieve arbitrary shift after multiplication, a combination of specified selection instruction, multiplication instruction, shift instruction, and minimum value instruction. Formulas (3)-(7) are calculations with rounding shift function. Chips with SIMD instruction set, such as the Beijing Junzheng T30 and T31 models, have instructions with rounding shift. If there is no such instruction, it is necessary to implement it according to the steps of formulas (3)-(7).

[0019] The quantization multiplication coefficient b, the first shift coefficient shift_left, and the second shift coefficient shift_right are data obtained through training the model (belonging to the model's parameters). This data matches the number of channels in the feature map; there are as many sets of such data as there are channels.

[0020] The method described above can be divided into two cases when using the SIMD instruction set for calculation. One case is when shift_left is all 0: In SIMD instructions, the maximum value instruction, multiplication shift instruction, shift rounding instruction and minimum value instruction are used to perform the above calculation. However, the shift instruction in the multiplication shift instruction can only shift right by 15 bits, so it is only suitable for the case where shift_left is all 0.

[0021] Another scenario is when shift_left is not zero: In SIMD instructions, when using the maximum value instruction, multiplication shift instruction, and minimum value instruction, since shift_left is not always zero, meaning the shift after multiplication is not always 15, it is impossible to use the multiplication shift instruction to perform multiplication and shift by 15 bits. Therefore, it is necessary to use a combination of selection instructions, multiplication instructions, and shift instructions to achieve the function of arbitrary shift after multiplication. That is, two data with actual effective data of 16 bits are converted to 32 bits before multiplication, and then multiplied. The result of multiplication is then shifted to achieve the above calculation.

[0022] One scenario involves all `shift_left` values ​​being 0. Further, let `vmul` be the register storing the multiplication coefficients, and `vshift2` be the register storing the second shift coefficients. `vmul` and `vshift2` can only store eight 16-bit data values ​​at a time. The registers for convolution accumulation are `vsum0` and `vsum1`, and register `v_tmp` stores 0, 15, or 8. Since there are 32 registers, multiple registers are used in the convolution calculation. To use registers efficiently, the three variables share a register by repeatedly loading immediate values. In the implementation of convolution calculations, batch processing is generally used to accelerate processing, resulting in more generated results. A batch of generated convolution results might contain 64 data points, occupying 8 registers. Some registers storing constants are continuously occupied, leaving even fewer registers available for subsequent quantization. If more than 32 registers are used for assignment, the previously assigned register data will be saved to a cache. When the register is used again later, the data is loaded from the cache, leading to a sharp decrease in speed. To improve speed, we design an algorithm to avoid repeatedly saving and loading register data.

[0023] a) Perform operations where the value of vsum is not less than 0, and store the result in vsum:

[0024] Load immediate data instruction: Load immediate value 0, store 16 8-bit data bytes or 8 16-bit data bytes into v_tmp, represented as:

[0025] v_tmp = ingenic_li_h(0);

[0026] Using the maximum value instruction: Input variables vsum and v_tmp, output variable vsum. vsum stores 8 integers (int16_t). Both vsum and v_tmp store 8 integers (int16_t). This can be represented as:

[0027] vsum=ingenic_max_h(vsum,v_tmp);

[0028] b) Load vmul and vshift2 data:

[0029] Using the `simd` instruction to load data: 128 bits of data, starting from memory location 0 (where `indata` points), are loaded into the variable `vmul`; then, another 128 bits of data, starting from memory location 16 (where `indata` points), are loaded into the variable `vshift2`; finally, `indata = indata + 16`; this is represented as follows:

[0030] vmul=ingenic_load(indata,0);

[0031] vshift2=ingenic_load(indata,16);

[0032] indata = indata + 16;

[0033] c) Perform the multiplication and shift by 15 bits, and store the result in vsum:

[0034] Using multiplication and shift instructions: Input variables vsum and vmul, output variable vsum, where vsum and vmul store 8 int16_t data; represented as follows:

[0035] vsum=ingenic_mulq_h(vsum,vmul);

[0036] d) Operations involving rounding and shifting, and storing the result in vsum:

[0037] Using the right shift instruction: Each element in the variable `vsum` is shifted right by `vshift` by 2 positions. Elements not shifted by more than 2 positions are rounded to the nearest integer. This is represented as:

[0038] vsum=ingenic_rvshift_right(vsum,vshift2);

[0039] e) Perform operations on vsum values ​​no greater than 15 and store the results in vsum:

[0040] Using the load immediate value instruction: Load immediate value 15, represented as:

[0041] v_tmp=ingenic_li_b(15)

[0042] Using the minimum value instruction: Input variables vsum and v_tmp, output variable vsum. vsum stores 8 int16_t data, and vsum and v_tmp both store 8 int16_t data, represented as:

[0043] vsum=ingenic_min_h(vsum,v_tmp);

[0044] f) Assign vsum to vsum0, replace vsum with vsum1, and repeat a), b), c), d), and e) to get the result of vsum1;

[0045] g) Merge vsmu0 and vsum1 into a single register, which stores 16 8-bit data entries. vri is the data location selected by the instruction to achieve this merging. Use the specified selection instruction to select 4, 8, or 16 data entries from the variables vsmu0 and vsmu1 according to the number set in vri. This instruction requires a permanent register vri, represented as:

[0046] vsum0=ingenic_choise(vsmu0,vsmu1,vri);

[0047] Using the instruction to load immediate data: Load immediate value 8, store 16 8-bit data, represented as:

[0048] v_tmp=ingenic_li_b(8);

[0049] Using the subtraction instruction: Input variables are mu0 and v_tmp, output variable is sum0. sum0 stores 8 integers (int16_t). Both mu0 and v_tmp store 8 integers (int16_t). This can be represented as:

[0050] vsum0=ingenic_sub_h(vsmu0,v_tmp);

[0051] h) Save the results to savedata:

[0052] The save data instruction uses the register variable vsum0 to store data, and a pointer to the data savedata. Data in the register is stored starting from the pointer's location. Specifically, 128 bits of data from vsum0 are stored in savedata, starting from memory location 0. This is represented as:

[0053] ingenic_save(vsum0,savedata,0);

[0054] savedata = savedata + 16.

[0055] The other case is when shift_left is not zero. In the method for the case where shift_left is all zeros, the operation in step c) is replaced by the following function (not a simd instruction):

[0056] vsum=mul_shift(vsum,vmul,vshift1);

[0057] The other steps remain unchanged to ensure that shift_left is not always 0.

[0058] The other case is where shift_left is not zero. Further, it includes the following design method for arbitrary shifting after multiplication:

[0059] Let the register storing the multiplication coefficients be vmul, and the register storing the first shift coefficients be vshift1. vmul and vshift1 can only store eight 16-bit data at a time. The register storing the result of the operation that is not less than 0 is vsum, which is then converted into 32-bit data and stored in two registers vsumt0 and vsumt1, where vsumt0 and v_tmp each store four 32-bit data. The final output result is vsum, which stores these eight 16-bit data. During the intermediate calculations, vmul and vshift1 need to be converted into 32-bit data. Temporary registers vsumt0 and v_tmp store the converted data at positions 0, 2, 4, 6 or 1, 3, 5, 6. Before loading vshift1, (15 - shift_left) is calculated and then loaded into vshift1. Here, the data loaded into vshift1 is (15 - shift_left), and (15 - shift_left) >= 0.

[0060] Let the function that performs the multiplication and arbitrary shift be vsum = mul_shift(vsum, vmul, vshift1).

[0061] The design method further includes:

[0062] <1> The data at positions 0, 2, 4, and 6 of vsum is converted to 32 bits. vri0 is the data at the specific position selected by the instruction and stored in vsumt0. Since vsum is data greater than or equal to 0, the 16-bit data is directly stored in the lower 16 bits of the 32-bit array, with the data remaining unchanged. Using the specified selection instruction, it is represented as follows:

[0063] vsumt0=ingenic_choise(vsum,vsmu0,vri0);

[0064] <2> Convert the data at positions 0, 2, 4, and 6 of vmul to 32 bits; use <1> The result of vri0 in the file is saved to v_tmp.

[0065] Using a specified selection command, it is represented as:

[0066] v_tmp=ingenic_choise(vmul,vmul,vri0);

[0067] <3> Perform a multiplication operation and store the result in vsumt0.

[0068] Using the multiplication instruction, it can be represented as:

[0069] vsumt0=ingenic_mul_d(vsumt0,v_tmp);

[0070] <4> Convert the data at positions 0, 2, 4, and 6 of vshift1 to 32 bits, using... <1> The result of vri0 in the file is saved to v_tmp.

[0071] Using a specified selection command, it is represented as:

[0072] v_tmp=ingenic_choise(vshift1,vshift1,vri0);

[0073] <5> Perform a shift operation on vsumt0 and save the result back to vsumt0.

[0074] Using the right shift instruction, any bits not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is performed, represented as:

[0075] vsumt0=ingenic_vshift_right_d(vsumt0,v_tmp);

[0076] <6> Perform a left shift operation of 15 bits on vsumt0, and save the result back to vsumt0.

[0077] Using the left shift instruction, any bits not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is performed, represented as:

[0078] vsumt0=ingenic_shift_left_d(vsumt0,15);

[0079] <7> The data at positions 1, 3, 5, and 6 of vsum are converted to 32 bits. vri1 is the data at the specific position selected by the instruction, and the result is saved to vsum, which contains four 32-bit data entries. Using a specified selection instruction, four data entries are selected from the variables vsum and vsum (a register variable must be input in the instruction; two identical register variables are acceptable) according to the numbers set in vri1. When using this instruction, a permanent register vri1 is required for the instruction to select the data at the specific position. The generated result is stored in vsum, represented as follows:

[0080] vsum=ingenic_choise(vsum,vsum,vri1);

[0081] <8> Convert the data at positions 1, 3, 5, and 6 of vmul to 32 bits, using... <6> The result of vri1 in the file is saved to v_tmp.

[0082] Using the specified selection instruction, four data points are selected from the variables vmul and vmul (the instruction must specify register variables, which can be two identical register variables) according to the number set in vri1. When using this instruction, a permanent register vri1 is required to hold the data at the specific location selected by the instruction, represented as:

[0083] v_tmp=ingenic_choise(vmul,vmul,vri1);

[0084] <9> Perform a multiplication operation and store the result in vsum.

[0085] Using the multiplication instruction, input variables vsum and v_tmp, the output variable is vsum. vsum stores four int32_t values, and both vsum and v_tmp store four int32_t values, represented as:

[0086] vsum=ingenic_mul_d(vsum,v_tmp);

[0087] <10> Convert the data at positions 1, 3, 5, and 6 of vshift1 to 32 bits, using... <6> The result of vri1 in the file is saved to v_tmp;

[0088] Using the specified selection instruction, four data items are selected from variables vshift1 and vshift1 according to the numbers set in vri1. When using this instruction, a permanent register vri1 is required to hold the data at the specific location selected by the instruction, as shown below:

[0089] v_tmp=ingenic_choise(vshift1,vshift1,vri1);

[0090] <11> Perform a shift operation on vsum and save the result back to vsum.

[0091] Using the right shift instruction, bits that are not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is represented as follows:

[0092] vsum=ingenic_vshift_right_d(vsum,v_tmp);

[0093] <12> The operation of adding vsumt0 to vsum is performed, and the result is stored in vsum. The addition operation merges the four 32-bit data (actual effective data is 16 bits, vsumt0 puts the 16-bit data into the high 16 bits of the 32-bit array, and vsum puts the 16-bit data into the low 16 bits of the 32-bit array) in the two registers.

[0094] Using the addition instruction, the input variables are sumt0 and sum, and the output variable is sum. sumt0 and sum store four int32_t data units. The high 16 bits and low 16 bits of each int32_t data unit in sum are each 16-bit data units. Here, two 16-bit data units are represented using 32 bits, so sum actually stores eight int16_t data units, as shown below:

[0095] vsum=ingenic_add_d(vsumt0,vsum).

[0096] The method is applicable to chips with the SIMD instruction set.

[0097] The chips include Beijing Junzheng T30 and T31 models.

[0098] Therefore, the advantage of this application lies in providing a method based on 4-bit quantization optimization. This method utilizes instructions from the SIMD instruction set to implement quantization optimization and employs a design method involving arbitrary shifting after multiplication, thereby improving computational speed. Using conventional computational design methods would result in a large computational load and numerous intermediate transformations. Shifting operations are a full rounding method, requiring data to be rounded to the nearest whole number, which is computationally inefficient using existing techniques. Specifically, for the Beijing Junzheng T30 and T31 chips, adopting this method and utilizing relevant SIMD instructions avoids these transformations found in existing technologies, leading to higher efficiency. Furthermore, it enables batch data processing, fully utilizing cached data and significantly improving processing speed. Attached Figure Description

[0099] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0100] Figure 1 This is a flowchart of the method of the present invention.

[0101] Figure 2 This is a schematic diagram of the design method when all values ​​are 0 in the method of this invention. Detailed Implementation

[0102] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0103] like Figure 1 As shown, the method of this invention relates to a method based on 4-bit quantization optimization. In this method, let the convolution calculation result be a, the quantization multiplication coefficient be b, the first shift coefficient be shift_left, the second shift coefficient be shift_right, and the final result be res. Instructions from the SIMD instruction set are used to process the data in batches. The quantization processing formula is expressed as follows:

[0104] S1, obtain the maximum value: a1=max(a,0) (1)

[0105] S2, obtain the result of multiplication and shift: ab = a1 * b >> (15 - shift_left) (2)

[0106] S3, obtains the result of the multiplication and arbitrary shift:

[0107] ab_shift=ab>>shift_right (3)

[0108] ab_rem2=[ab-(ab_shift<<shift_right)]<<1 (4)

[0109] dividend=1<<(15-shift_left) (5)

[0110]

[0111] res0 = ab_shift + rem (7) S4, obtain the minimum value: res = min(res0, 15) - 8 (8)

[0112] In formula (2), a1 and b need to be converted, transforming the original 16 bits into 32 bits. At this time, the value of ab is also 32 bits (the actual effective data is 16 bits). The instructions include the maximum value instruction, multiplication shift instruction, shift and rounding instruction, or, in order to achieve arbitrary shift after multiplication, a combination of specified selection instruction, multiplication instruction, shift instruction, and minimum value instruction. Formulas (3)-(7) are calculations with rounding shift function. In the Beijing Junzheng T30 and T31 model chips, there are instructions with rounding shift. If there is no such instruction, it is necessary to implement it according to the steps (3)-(7).

[0113] The quantization multiplication coefficient b, the first shift coefficient shift_left, and the second shift coefficient shift_right are data obtained through training the model (belonging to the model's parameters). This data matches the number of channels in the feature map; there are as many sets of such data as there are channels.

[0114] The method described above can be divided into two cases when using the SIMD instruction set for calculation. One case is when shift_left is all 0: In SIMD instructions, the maximum value instruction, multiplication shift instruction, shift rounding instruction and minimum value instruction are used to perform the above calculation. However, the shift instruction in the multiplication shift instruction can only shift right by 15 bits, so it is only suitable for the case where shift_left is all 0.

[0115] Another scenario is when shift_left is not zero: In SIMD instructions, when using the maximum value instruction, multiplication shift instruction, and minimum value instruction, since shift_left is not always zero, meaning the shift after multiplication is not always 15, it is impossible to use the multiplication shift instruction to perform multiplication and shift by 15 bits. Therefore, it is necessary to use a combination of selection instructions, multiplication instructions, and shift instructions to achieve the function of arbitrary shift after multiplication. That is, two data with actual effective data of 16 bits are converted to 32 bits before multiplication, and then multiplied. The result of multiplication is then shifted to achieve the above calculation.

[0116] Furthermore, for ease of understanding, the specific implementation of the method of this application can also be described as follows:

[0117] 1. SIMD instruction algorithm.

[0118] 1) Introduction to SimD Commands

[0119] The simd commands involved are as follows.

[0120] The method uses the following SIMD instruction set:

[0121] A) Maximum value instruction, represented as:

[0122] vrd=ingenic_max_h(vrs,vrt);

[0123] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 8 int16_t data, and vrs and vrt store 8 int16_t data.

[0124] Equivalent operations:

[0125] vrd0:=(vrs0>vrt0)? vrs0:vrt0;

[0126] vrd1:=(vrs1>vrt1)? vrs1:vrt1;

[0127] ...

[0128] vrd7:=(vrs7>vrt7)? vrs7:vrt7;

[0129] B) Minimum value instruction, expressed as:

[0130] vrd=ingenic_min_h(vrs,vrt);

[0131] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 8 int16_t data, and vrs and vrt store 8 int16_t data.

[0132] Equivalent operations:

[0133] vrd0:=(vrs0) <vrt0)?vrs0:vrt0;

[0134] vrd1:=(vrs1) <vrt1)?vrs1:vrt1;

[0135] ...

[0136] vrd7:=(vrs7) <vrt7)?vrs7:vrt7;

[0137] C) Multiplication and shift instructions are represented as:

[0138] vrd=ingenic_mulq_h(vrs,vrt);

[0139] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 8 int16_t data, and vrs and vrt store 8 int16_t data.

[0140] Equivalent operations:

[0141] vrd0:=(vrs0*vrt0)>>15;

[0142] vrd1:=(vrs1*vrt1)>>15;

[0143] ...

[0144] vrd7:=(vrs7*vrt7)>>15;

[0145] D) Multiplication instructions are represented as:

[0146] vrd=ingenic_mul_d(vrs,vrt);

[0147] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 4 int32_t data, while vrs and vrt store 4 int32_t data.

[0148] Equivalent operations:

[0149] vrd0:=vrs0*vrt0;

[0150] vrd1:=vrs1*vrt1;

[0151] ...

[0152] vrd3:=(vrs3*vrt3;

[0153] E) Subtraction instruction, represented as:

[0154] vrd=ingenic_sub_h(vrs,vrt);

[0155] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 8 int16_t data, and vrs and vrt store 8 int16_t data.

[0156] Equivalent operations:

[0157] vrd0:=vrs0-vrt0;

[0158] vrd1:=vrs1-vrt1;

[0159] ...

[0160] vrd7:=vrs7-vrt7;

[0161] F) Addition instruction, represented as:

[0162] vrd=ingenic_add_d(vrs,vrt);

[0163] The input variables are vrs and vrt, and the output variable is vrd; vrd stores 4 int32_t data, while vrs and vrt store 4 int32_t data.

[0164] Equivalent operations:

[0165] vrd0:=vrs0+vrt0;

[0166] vrd1:=vrs1+vrt1;

[0167] ...

[0168] vrd3:=vrs3+vrt3;

[0169] G) Shift instructions: Shift each element in the variable by i bits or vri bits; for 8-bit data shift, add _b after the instruction; for 16-bit data shift, add _h after the instruction; for 32-bit data shift, add _d after the instruction.

[0170] In the instruction description, this is an instruction with rounding shift.

[0171] Naming conventions: r stands for round, v is a shifted data that is similar to an array, using the v in vector, and the others are just their English meanings.

[0172] The left shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as:

[0173] vrd=ingenic_shift_left(vrs,i);

[0174] The right shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as:

[0175] vrd=ingenic_shift_right(vrs,i);

[0176] The left shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as:

[0177] vrd=ingenic_vshift_left(vrs,vri);

[0178] The right shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as:

[0179] vrd=ingenic_vshift_right(vrs,vri);

[0180] The left shift SIMD instruction rounds up any bits that are not shifted sufficiently. This is represented as:

[0181] vrd=ingenic_rvshift_left(vrs,vri);

[0182] The right shift SIMD instruction rounds up any bits that are not shifted sufficiently.

[0183] vrd=ingenic_rvshift_right(vrs,vri);

[0184] H) Select instruction: Selects 4, 8, or 16 data items from the variables vrs and vrt according to the numbers set in vri. When using this instruction, a permanent register vri is required to select the data at a specific location; represented as...

[0185] vrd=ingenic_choise(vrs,vrt,vri);

[0186] I) Load Data Instruction: The input is the data to be loaded. Currently, it's a pointer to the data. Starting from the location pointed to by this data in memory, 128 bits of data are loaded. If it's 8-bit data, 16 bits are loaded; if it's 16-bit data, 8 bits are loaded. The data is loaded into the variable `vrd`. `m` is calculated in bytes, i.e., 8 bits per unit. The 128-bit data is loaded starting from the location `m` pointed to by the data `indata` in memory, represented as...

[0187] vrd=ingenic_load(indata,m);

[0188] J) Load immediate data instruction: Loads an immediate value i, where i is a constant and cannot be a variable;

[0189] Store 16 8-bit data, represented as

[0190] vrd = ingenic_li_b(i);

[0191] Store 8 16-bit data, represented as

[0192] vrd = ingenic_li_h(i);

[0193] K) Save data command: represented as

[0194] ingenic_save(vrd,savedata,m);

[0195] The register variable vrd stores the data, and the pointer savedata stores the data. The data in the register is stored starting from the pointer position. The 128-bit data in vrd is stored in savedata. m is calculated in bytes, that is, 8 bits per unit. The 128-bit data is stored starting from the position m pointed to by savedata in memory.

[0196] 2. Quantification

[0197] In the quantization formula, let the convolution result be 'a', the quantization multiplication coefficient be 'b', the first shift coefficient be 'shift_left', the second shift coefficient be 'shift_right', and the final result be 'res'. The calculation formula is as follows.

[0198] a1=max(a,0);................................(1)

[0199] ab=a1*b>>(15-shift_left);............(2)

[0200] ab_shift=ab>>shift_right;......................(3)

[0201] ab_rem2=[ab-(ab_shift<<shift_right)]<<1;......(4)

[0202] dividend=1<<(15-shift_left);............(5)

[0203]

[0204] res0=ab_shift+rem,........................(7)

[0205] res=min(res0,15)-8;......................(8)

[0206] In formula (2), a1 and b need to be converted from 16 bits to 32 bits; otherwise, the result calculated after multiplication may overflow and cause errors. At this time, the value of ab is also 32 bits.

[0207] 3. SIMD instruction optimization algorithm

[0208] For the formula above, ordinary calculations would be computationally intensive and require numerous conversions. Shift operations are a rounding-off method, requiring rounding of the data using steps (4), (5), (6), and (7). This results in very low computational efficiency. This method, based on instructions from the Beijing Junzheng T30 and T31 chips, avoids these conversions, achieving higher efficiency. It also allows for batch data processing, fully utilizing cached data and significantly improving processing speed.

[0209] Here, the quantization multiplication coefficient is 'b', the first shift coefficient is 'shift_left', and the second shift coefficient is 'shift_right'. These are data obtained through training, and this data matches the number of channels in the feature map; the number of such data sets corresponds to the number of channels. Generally, 'shift_left' is 0, which can be simplified as a special case. When 'shift_left' is not 0, the calculation proceeds differently. Therefore, in subsequent calculations, two cases will be considered: one where all 'shift_left' values ​​provided by the current model are 0, and another where 'shift_left' contains non-zero values.

[0210] 1) shift_left is all 0, such as Figure 2 As shown,

[0211] The T30 and T31 chips offer four instruction sets: maximum value, multiplication-shift, shift-round, and minimum value. These four instructions can perform the calculations described above. However, the multiplication-shift instruction can only shift right by 15 bits, making it only suitable for cases where shift_left is all zeros.

[0212] Let register vmul store the multiplication coefficients and vshift2 store the second shift coefficients. vmul and vshift2 can only store eight 16-bit data bits at a time. The registers for convolution accumulation are vsum0 and vsum1, and register v_tmp stores 0, 15, or 8. Since there are 32 registers, many registers are used in convolution accumulation. To use registers efficiently, the three variables share a single register by repeatedly loading immediate values.

[0213] a) Perform operations where the value of vsum is not less than 0, and store the result in vsum.

[0214] v_tmp = ingenic_li_h(0);

[0215] vsum=ingenic_max_h(vsum,v_tmp);

[0216] b) Load vmul and vshift2 data.

[0217] vmul=ingenic_load(indata,0);

[0218] vshift2=ingenic_load(indata,16);

[0219] indata+=16;

[0220] c) Perform the multiplication and shift by 15 bits, and store the result in vsum.

[0221] vsum=ingenic_mulq_h(vsum,vmul);

[0222] d) Operations involving rounding and shifting, and storing the result in vsum

[0223] vsum=ingenic_rvshift_right(vsum,vshift2);

[0224] e) Perform calculations where the value of vsum is not greater than 15, and store the result in vsum.

[0225] v_tmp=ingenic_li_b(15);

[0226] vsum=ingenic_min_h(vsum,v_tmp);

[0227] f) Assign vsum1 to vsum0, and repeat a), b), c), and d). Obtain the result of vsum1.

[0228] g) Merge vsmu0 and vsum1 into a single register that stores 16 8-bit data entries. vri is the instruction that selects the specific location of the data to be merged into a single register.

[0229] vsum0=ingenic_choise(vsmu0,vsmu1,vri);

[0230] v_tmp=ingenic_li_b(8);

[0231] vsum0=ingenic_sub_h(vsmu0,v_tmp);

[0232] h) Save the results to savedata.

[0233] ingenic_save(vsum0,savedata,0);

[0234] savedata+=savedata+16.

[0235] 2) shift_left is not all zeros.

[0236] Because `shift_left` is not all zeros, meaning the shift after multiplication is not all 15, the `ingenic_mulq_h` instruction cannot be used. Therefore, several instructions need to be used in combination to achieve arbitrary shifting after multiplication. Multiplying two data points that are actually 16 bits in size may result in overflow, so the data needs to be converted to 32 bits before multiplication. Then, the result of the multiplication is shifted. Let's assume the design for arbitrary shifting after multiplication is as follows:

[0237] Let vmul be the register storing the multiplication coefficients, and vsum be the register storing the convolution calculation result and the calculator storing the multiplication shift result. The register storing the first shift coefficient is vshift1. vmul and vshift1 can only store eight 16-bit data values ​​at a time. Some calculation results are also stored in register vsum, converted to 32-bit data and stored in two registers vsumt0 and vsumt1, where vsumt0 and v_tmp each store four 32-bit data values. The final output result is vsum, which stores these eight 16-bit data values. Intermediate calculations require converting vmul and vshift1 to 32-bit, with temporary registers vsumt0 and v_tmp storing the converted data at positions 0, 2, 4, 6 or 1, 3, 5, 6. Before loading vshift1, (15 - shift_left) is calculated and then loaded into vshift1. Here, the data loaded into vshift1 is (15 - shift_left), and (15 - shift_left) >= 0.

[0238] Let the function that performs the multiplication and arbitrary shift be vsum = mul_shift(vsum, vmul, vshift1).

[0239] <1> Convert the data at positions 0, 2, 4, and 6 of vsum to 32 bits. vri0 is the instruction to select the specific data positions, converting the data at positions 0, 2, 4, and 6 of vsum to 32-bit data. Since vsum is data greater than or equal to 0, the 16-bit data can be directly stored in the lower 16 bits of the 32-bit array, without changing the data itself.

[0240] vsumt0=ingenic_choise(vsmu,vsmu,vri0);

[0241] <2> Convert the data at positions 0, 2, 4, and 6 of vmul to 32 bits. Use <1> vri0 in the middle. The result is saved to v_tmp

[0242] v_tmp=ingenic_choise(vmul,vmul,vri0);

[0243] <3> Perform multiplication and store the result in vsumt0.

[0244] vsumt0=ingenic_mul_d(vsumt0,v_tmp);

[0245] <4> Convert the data at positions 0, 2, 4, and 6 of vshift1 to 32 bits. Use <1> vri0 in the middle. The result is saved to v_tmp

[0246] v_tmp=ingenic_choise(vshift1,vshift1,vri0);

[0247] <5> Perform a shift operation on vsumt0 and save the result back to vsumt0;

[0248] vsumt0=ingenic_vshift_right_d(vsumt0,v_tmp);

[0249] <6> Perform a left shift operation of vsumt0 by 15 bits, and save the result to vsumt0.

[0250] vsumt0=ingenic_shift_left_d(vsumt0,15);

[0251] <7> The data at positions 1, 3, 5, and 6 of vsum is converted to 32 bits. vri1 is the instruction to select the data at a specific position, and the result is saved to vsum. Here, vsum stores four 32-bit data.

[0252] vsum=ingenic_choise(vsum,vsum,vri1);

[0253] <8> Convert the data at positions 1, 3, 5, and 6 of vmul to 32 bits. Use <6> vri1 in the middle.

[0254] The result is saved to v_tmp

[0255] v_tmp=ingenic_choise(vmul,vmul,vri1);

[0256] <9> Perform multiplication and store the result in vsum0.

[0257] vsum=ingenic_mul_d(vsum,v_tmp);

[0258] <10> Convert the data at positions 1, 3, 5, and 6 of vshift1 to 32 bits. Use <6> vri1 in the file. The result is saved to v_tmp.

[0259] v_tmp=ingenic_choise(vshift1,vshift1,vri1);

[0260] <11> Perform a shift operation on vsum0 and save the result to vsum0.

[0261] vsum=ingenic_vshift_right_d(vsum,v_tmp);

[0262] <12> The operation `vsum` is performed on `vsum0`, and the result is saved to `vsum`.

[0263] vsum=ingenic_add_d(vsumt0,vsum);

[0264] This optimized design method can make great use of registers.

[0265] In other words, the operation in c) where 1) shift_left is all 0 can be replaced by the following, expressed as:

[0266] vsum=mul_shift(vsum,vmul,vshift1);

[0267] With the rest of the process unchanged, it is possible to achieve a situation where shift_left is not all zeros.

[0268] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method based on 4-bit quantization optimization, characterized in that, In the method described, let the convolution calculation result be a, the quantization multiplication coefficient be b, and the first shift coefficient be... The second shift coefficient is The final result is res. The method described, when used with the SimD instruction set, can be divided into two cases. One is In the case where all values ​​are 0: SIMD instructions use the maximum value instruction, multiplication-shift instruction, shift-round instruction, and minimum value instruction to calculate the quantization formula. However, the multiplication-shift instruction can only shift right by 15 bits, so it is only suitable for... The case where all values ​​are 0; Another one is There are cases where the result is not zero: In SIMD instructions, when using the maximum value instruction, multiplication shift instruction, and minimum value instruction, because... If the result is not all zeros, meaning the shift after multiplication is not all 15, then the multiplication-shift instruction cannot be used to perform the multiplication and shift operation by 15 bits. Therefore, it is necessary to use a combination of selection, multiplication, and shift instructions to achieve the function of arbitrary shift after multiplication. That is, before multiplying two data that are actually 16 bits, convert the data to 32 bits, then multiply them, and then shift the result of the multiplication to achieve the above calculation. Data is processed in batches using instructions from the SIMD instruction set. The quantization formula is expressed as follows: S1, obtain the maximum value: (1) S2, obtain the result of the multiplication and shift: (2) S3, obtains the result of the multiplication and arbitrary shift: (3) (4) (5) (6) (7) S4, obtain the minimum value: (8) In formula (2), a1 and b need to be converted to 32 bits from the original 16 bits. At this time, the value of ab is also 32 bits, and the actual effective data is 16 bits. The instructions include the maximum value instruction, multiplication shift instruction, shift rounding instruction, or in order to achieve arbitrary shift after multiplication, the specified selection instruction, multiplication instruction, shift instruction, minimum value instruction are used in combination. Formulas (3)-(7) are calculations with shift function rounding.

2. The method based on 4-bit quantization optimization according to claim 1, characterized in that, The quantization multiplication coefficient b, the first shift coefficient Second shift coefficient This data is obtained through training the model and consists of the model's parameters. This data matches the number of channels in the feature map; the number of channels corresponds to the number of such data sets.

3. The method based on 4-bit quantization optimization according to claim 1, characterized in that, The method uses the following SIMD instruction set: A) Maximum value instruction, represented as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores 8... The data, stored in VRS and VRRT, consists of 8... data; Equivalent operations: ; ; …… ; B) Minimum value instruction, expressed as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores 8... The data, stored in VRS and VRRT, consists of 8... data; Equivalent operations: ; ; …… ; C) Multiplication and shift instructions, represented as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores 8... The data, stored in VRS and VRRT, consists of 8... data; Equivalent operations: ; ; …… ; D) Multiplication instruction, represented as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores four... The data, stored in VRS and VRRT, consists of four... data; Equivalent operations: ; ; …… ; E) Subtraction instruction, represented as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores 8... The data, stored in VRS and VRRT, consists of 8... data; Equivalent operations: ; ; …… ; F) Addition instruction, represented as: ; Input variables are vrs and vrt, and the output variable is vrd; vrd stores four... The data, stored in VRS and VRRT, consists of four... data; Equivalent operations: ; ; …… ; G) Shift instructions: Shift each element in the variable by i bits or vri bits; for 8-bit data shifts, add the following after the instruction: ;16-bit data shift operation, add after the instruction 32-bit data shift operations, add after the instruction ; The left shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as: ； The right shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as: ； The left shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as: ； The right shift SIMD instruction rounds down any bits not shifted, without rounding. This is represented as: ； The left shift SIMD instruction rounds up any bits that are not shifted sufficiently. This is represented as: ； The right shift SIMD instruction rounds up any bits that are not shifted sufficiently. ； H) Select instruction: Selects 4, 8, or 16 data items from the variables vrs and vrt according to the numbers set in vri. When using this instruction, a permanent register vri is required to select the data at a specific location; represented as... ； I) Load Data Instruction: The input is the data to be loaded. Currently, it's a pointer to the data. Starting from the location pointed to by this data in memory, 128 bits of data are loaded. If it's 8-bit data, 16 bits are loaded; if it's 16-bit data, 8 bits are loaded. The data is loaded into the variable `vrd`. `m` is calculated in bytes, i.e., 8 bits per unit. The 128-bit data is loaded starting from the location `m` pointed to by the data `indata` in memory, represented as... ； J) Load immediate data instruction: Loads an immediate value i, where i is a constant and cannot be a variable; Store 16 8-bit data, represented as ； Store 8 16-bit data, represented as ； K) Save data command: represented as ； The register variable vrd stores the data, and the pointer savedata stores the data. The data in the register is stored starting from the position pointed to by this pointer. The 128-bit data in vrd is stored in savedata. m is calculated in bytes, that is, 8 bits as a unit. The 128-bit data is stored starting from the position m pointed to by savedata in memory.

4. The method based on 4-bit quantization optimization according to claim 3, characterized in that, The one is The case where all values ​​are 0 further includes: Let the register storing the multiplication coefficients be vmul, and the register storing the second shift coefficients be vshift2. vmul and vshift2 can only store eight 16-bit data values ​​at a time; the registers for convolution accumulation are vsum0 and vsum1. It stores 0, 15, or 8; since there are 32 registers, the three variables use one register by repeatedly loading immediate values; a) Perform operations where the value of vsum is not less than 0, and store the result in the register vsum: Load immediate data instruction: Load immediate value 0, store 8 16-bit data into , is represented as: ； Using the maximum value command: Input variable vsum, The output variable is vsum, which stores 8 values. Data, vsum and It stores 8 The data is represented as: ; b) Load vmul and vshift2 data: Using the `simd` command to load data: 128 bits of data, starting from memory location 0 (pointed to by `indata`), are loaded into the variable `vmul`; then, another 128 bits of data, starting from memory location 16 (pointed to by `indata`), are loaded into the variable `vshift2`; finally, ... ; indicates the following: ； ； ； c) Perform the multiplication and shift by 15 bits, and store the result in vsum: Using multiplication and shift instructions: Input variables vsum and vmul, output variable vsum. vsum and vmul store 8 bits. Data; represented as follows: ; d) Operations involving rounding and shifting, and storing the result in vsum: Using the right shift instruction: Shift each element in the variable vsum by vshift2 positions. Round off any elements that are not shifted further. This is represented as: ; e) Perform operations on vsum values ​​no greater than 15 and store the results in vsum: Using the load immediate value instruction: Load immediate value 15, represented as: ； Use the minimum value instruction: Input variable vsum, The output variable is vsum, which stores 8 values. Data, vsum and It stores 8 The data is represented as: ; f) Assign vsum to vsum0, replace vsum with vsum1, and repeat a), b), c), d), and e) to get the result of vsum1; g) Merge vsmu0 and vsum1 into a single register that stores 16 8-bit data bytes. The vri parameter specifies the data location for the instruction to perform the merging. Use the specified selection instruction: select 4, 8, or 16 data items from the variables vsmu0 and vsmu1 according to the numbers set in the vri. When using this instruction, a permanent register vri is required, represented as: ； Using the instruction to load immediate data: Load immediate value 8, store 16 8-bit data, represented as: ； Use the subtraction command: Input variable vsmu0, The output variable is vsum0, which stores 8 values. Data, vsmu0 and It stores 8 The data is represented as: ； h) Save the results to savedata: The save data instruction uses the register variable vsum0 to store data, and a pointer to the data savedata. Data in the register is stored starting from the pointer's location. Specifically, 128 bits of data from vsum0 are stored in savedata, starting from memory location 0. This is represented as: ； 。 5. The method based on 4-bit quantization optimization according to claim 4, characterized in that, The other one is There are cases where the result is not zero, which refers to the aforementioned type. In the method for the case where all values ​​are 0, the operation in step c) is replaced by the following function: ； In this setup, let vmul be the register storing the multiplication coefficients, vshift1 be the register storing the first shift coefficients, and vsum be the register storing the convolution calculation result and the multiplication and shift result; the other steps remain unchanged to achieve... Cases where not all values ​​are 0.

6. The method based on 4-bit quantization optimization according to claim 5, characterized in that, The other one is There are cases where the result is not zero. Furthermore, the design method for arbitrary shifting after multiplication is as follows: The register storing the multiplication coefficients is vmul, and the register storing the convolution result is vsum. The register storing the first shift coefficient is vshift1. Some operation results are also stored in register vsum, converted to 32-bit data and stored in two registers, vsumt0 and vsum. Each input consists of four 32-bit data segments; the final output is vsum, which stores these eight 16-bit data segments. The intermediate calculations require converting vmul and vshift1 to 32 bits using temporary registers vsumt0. Stores the data at positions 0, 2, 4, 6 or 1, 3, 5, 6 after conversion; before loading vshift1, After calculation, load the data into vshift1; the data loaded into vshift1 here is... ,and ; Let the function that performs arbitrary shifts after multiplication be: .

7. The method based on 4-bit quantization optimization according to claim 6, characterized in that, The design method further includes: <1> The data at positions 0, 2, 4, and 6 of vsum is converted to 32 bits. vri0 is the instruction that selects the specific position of the data, which is then stored in vsumt0. Since vsum is data greater than or equal to 0, the 16-bit data is directly stored in the lower 16 bits of the 32-bit array, and the data remains unchanged. Using a specified selection command, it is represented as: ； <2> Convert the data at positions 0, 2, 4, and 6 of vmul to 32 bits; use <1> vri0 in the middle, the result is saved to , Using a specified selection command, it is represented as: ； <3> Perform a multiplication operation and store the result in vsumt0. Using the multiplication instruction, it can be represented as: ; <4> Convert the data at positions 0, 2, 4, and 6 of vshift1 to 32 bits, using... <1> vri0 in the middle, the result is saved to , Using a specified selection command, it is represented as: ； <5> Perform a shift operation on vsumt0 and save the result back to vsumt0. Using the right shift instruction, any bits not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is performed, represented as: ; <6> Perform a left shift operation of 15 bits on vsumt0, and save the result back to vsumt0. Using the left shift instruction, any bits not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is performed, represented as: ; <7> Convert the data at positions 1, 3, 5, and 6 of vsum to 32 bits. vri1 is the instruction to select the specific data position. Convert the last four data bits of vsmu0 to 32 bits and save the result to vsum0. Using the specified selection instruction, four data items are selected from the variable vsum according to the numbers set in vri1. When using this instruction, a permanent register vri1 is required to hold the data at the specific location selected by the instruction, as shown below: ； <8> Convert the data at positions 1, 3, 5, and 6 of vmul to 32 bits, using... <6> vri1 in the middle, the result is saved to , Using the specified selection instruction, four data points are selected from the variable vmul according to the numbers set in vri1. When using this instruction, a permanent register vri1 is required to select the data at the specific location, as shown below: ； <9> Perform a multiplication operation and store the result in vsum. Use the multiplication instruction to input the variable vsum. The output variable is vsum, which stores four values. Data, vsum and It stores 4 The data is represented as: ; <10> Convert the data at positions 1, 3, 5, and 6 of vshift1 to 32 bits, using... <6> vri1 in the middle, the result is saved to ; Using the specified selection instruction, four data items are selected from the variable vshift1 according to the numbers set in vri1. When using this instruction, a permanent register vri1 is required to hold the data at the specific location selected by the instruction, as shown below: ； <11> Perform a shift operation on vsum0 and save the result back to vsum0. Using the right shift instruction, bits that are not shifted sufficiently are discarded without rounding. A 32-bit data shift operation is represented as follows: ； <12> The operation of adding vsum to vsumt0 results in vsum. The addition operation merges the four 32-bit data in the two registers, and the actual effective data is 16 bits. vsumt0 puts the 16-bit data into the high 16 bits of the 32-bit register, and vsum puts the 16-bit data into the low 16 bits of the 32-bit register. Using the addition instruction, input variables vsumt0 and vsum, the output variable is vsum. vsumt0 and vsum store four values. Data, each in vsum The high 16 bits and low 16 bits of the data are each 16 bits. Here, 32 bits are used to represent and store two 16-bit data, but vsum actually stores 8 of them. The data is represented as: 。 8. The method based on 4-bit quantization optimization according to claim 1, characterized in that, The method is applicable to chips with the SIMD instruction set.

Images