Mixed bit width calculation unit and quantization method based on DSP (Digital Signal Processor)

Abstract

The invention discloses a DSP (Digital Signal Processor)-based mixed bit width calculation unit and a quantization method, and the mixed bit width calculation unit comprises a pre-addition module which comprises an execution input processing part, a bit operation part and an accumulation part; and the multiplication and accumulation module comprises a multiplication part and an accumulation part. The quantization method is used for quantizing different layers of a CNN into different bit widths, and comprises the following steps: obtaining weight data; and representing the 32-bit floating-point number corresponding to the weight data by using an inner product of K base vectors and binary coding vectors. By using the method, LUT resources can be fully utilized, 1b-8b weight calculation is supported, and the calculation efficiency is increased along with reduction of the weight bit width. The method can be widely applied to the field of neural network accelerators.

Description

Technical Field

[0001] The present invention relates to the field of neural network accelerators, and in particular, to a mixed-bitwidth calculation unit based on DSP and a quantization method. Background Art

[0002] In the field of artificial intelligence, convolutional neural networks (CNNs) have achieved excellent results in many tasks, such as image classification, object detection, etc., due to their powerful feature extraction and learning capabilities. However, with the continuous expansion of network scale and the increase in model complexity, their demand for computing resources is also growing. Especially when deploying CNN models on resource-constrained embedded devices, how to improve hardware efficiency while ensuring network accuracy has become a key problem to be solved urgently.

[0003] The mixed-bitwidth quantization network provides an effective solution to this key problem. It can achieve a good balance between model size and hardware efficiency, enabling CNN models to run efficiently on embedded devices. However, in practical applications, the deployment of mixed-bitwidth accelerators based on the Field Programmable Gate Array (FPGA) platform faces many challenges.

[0004] Currently, the mixed-bitwidth accelerators on the FPGA platform mainly use the following methods to construct the Mixed-Width Multiply-Accumulate (MWMAC) unit that supports mixed-bitwidth calculation: the method based on the Digital Signal Processor (DSP). However, currently, the mixed-bitwidth calculations supported by the DSP-based MWMAC are limited in combination, and the DSP resources cannot be efficiently utilized, which greatly restricts the performance of the mixed-bitwidth quantization network. Summary of the Invention

[0005] In view of this, in order to solve the technical problems that the existing mixed-bitwidth calculation unit cannot fully utilize DSP resources and the supported mixed-bitwidth calculation combinations are limited, in a first aspect, the present invention proposes a mixed-bitwidth calculation unit based on DSP, and the mixed-bitwidth calculation unit includes:

[0006] A pre-addition module, including an input processing part, a bit operation part, and an accumulation part;

[0007] The input processing part: accepts input activation data;

[0008] The bit operation part: performs a bit-level operation on the input activation and the sign vector Sign_V convert (the elements of which only contain -1 and +1);

[0009] Accumulation part: Use a three-level adder tree to perform parallel accumulation on the bit operation results;

[0010] Multiplication and accumulation module, including a multiplication part and an accumulation part;

[0011] Multiplication part: Receive the output result of the pre-addition module and perform a multiplication operation with the Basis;

[0012] Accumulation part: Perform an accumulation operation to generate the final output.

[0013] In a second aspect, a quantization method is provided. Based on the above-mentioned hybrid bit-width calculation unit, different layers of the CNN are quantized into different bit-widths, including:

[0014] Load weights from the CNN;

[0015] Represent the 32-bit floating-point number corresponding to the weight data as the inner product of K basis vectors and binary coding vectors.

[0016] Based on the above solution, the present invention provides a hybrid bit-width calculation unit and quantization method based on DSP, which can support 1b to 8b weight calculations, can efficiently utilize the DSP and LUT resources of the FPGA platform, and the proposed hybrid bit-width calculation unit can be used in a convolutional neural network accelerator for the FPGA platform. Description of the Drawings

[0017] Figure 1 is a structural block diagram of a hybrid bit-width calculation unit based on DSP of the present invention;

[0018] Figure 2 is the internal calculation structure diagram of DSP in a specific embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of specific data analysis for quantizing weights into 4b by the quantization method in a specific embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of the process of the quantization method in a specific embodiment of the present invention. Detailed Description of the Invention

[0021] In view of the problems existing in the existing LUT-based and DSP-based MWMAC units on the FPGA platform, such as the inability to fully utilize DSP resources and the limited combination of supported mixed-bitwidth calculations. The present invention proposes a DSP-based multiply-accumulate (Mixed-Width Multiply-Accumulate, MWMAC) unit that supports mixed-bitwidth calculations. This MWMAC is constructed by DSPs and can make full use of LUT resources, support 1b to 8b weight calculations, and the calculation efficiency increases as the weight bitwidth decreases. (The calculation efficiency of 4b weights is twice that of 8b weights, and the calculation efficiency of 2b weights is four times that of 8b weights).

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application.

[0023] It should be noted that for the sake of convenience of description, only the parts related to the relevant invention are shown in the accompanying drawings. Without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0024] It should be understood that the "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0025] As shown in the present application and the claims, unless the context clearly indicates an exception, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of the clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. An element defined by the statement "comprising one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

[0026] In the description of the embodiments of the present application, "a plurality" means two or more than two. The following terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0027] In addition, flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of this application. It should be understood that the preceding or subsequent operations do not necessarily need to be executed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. At the same time, other operations can also be added to these processes, or one or several steps can be removed from these processes.

[0028] Referring to Figure 1 , which is a schematic structural diagram of an optional example of a hybrid bit-width calculation unit based on DSP proposed by the present invention. The hybrid bit-width calculation unit proposed in this embodiment may include but is not limited to the following modules:

[0029] The pre-addition module. Eight input activations first perform bit operations with Sign_V convert and the obtained results are accumulated in a three-stage addition tree. This part of the operation is called pre-addition operation;

[0030] The multiplication and accumulation module. The result output by the pre-addition module is finally sent to the DSP to perform multiplication and accumulation with the corresponding Basis.

[0031] Figure 2 shows the internal calculation structure of the DSP. The output result of the pre-addition unit and the Basis are sent to the multiplier input terminals A and B of the DSP, and then multiplication is performed.

[0032] There are 4 elements in Basis, which means that 8 input activations and 4 weights need to perform 4 high-bit-width multiplication operations, so it means that the calculation can be completed in 4 clocks; similarly, if there is only 1 element in Basis, then these 8 input activations and 8 weights only need to perform 1 multiplication operation and only 1 clock is required to complete the calculation. Therefore, the MWMAC designed by us can support hybrid bit-width calculation from 1b to 8b, and ensure that the change of the weight bit-width from 1b to 8b can bring a linear change in the calculation efficiency. That is, the calculation efficiency of 4b weights is twice that of 8b weights, and the calculation efficiency of 2b weights is 4 times that of 8b weights. <00ffff

[0033] The number of elements in the Basis vector is the same as the bit-width of the weight data, ranging from 1 to 8. Since each filter shares a set of the same Basis, it means that the Basis only needs to be input into the calculation unit once, and all the Sign_Vectors of this filter use this Basis. Therefore, the storage and transmission of the Basis only occupy a very small part of the resources and can be ignored. Considering that the elements in the Sign_Vector are all composed of -1 and 1, in actual storage, in this embodiment, 0 is used to represent -1, which can further reduce the storage resources. Therefore, in fact, each Sign_Vector only needs to be stored with a bit-width of 8b.

[0034] For example Figure 3 as shown, this example demonstrates quantizing the weights to 4 bits. As Figure 2 shown, there is a set of input activations A = {A0, A1, A2, A3, A4, A5, A6, A7} and a set of unquantized weights W = {W0, W1, W2, W3, W4, W5, W6, W7}, and 8 multiplications are required to obtain the output result Res1. Assume that each multiplication operation requires one clock cycle to execute for hardware implementation; thus it takes 8 clock cycles to obtain Res1.

[0035] Figure 3 In it, Sign_Vinit is the initial Sign_Vector before conversion; Sign_Vconvert points to the converted Sign_Vector. After conversion, Sign_Vconver and the input activations at the corresponding positions perform low-bit multiplication and accumulation. Sign_Vinit, Sign_Vector, and Sign_Vconver are all the corresponding sign vectors, and the result of the previous operation is multiplied by the corresponding basis at the high bit to obtain Res2. Since the elements in Sign_Vector only contain -1 and +1, simple bitwise operations can directly implement low-bit-width multiplication. It can be observed that Res2 only requires four high-width multiplications to obtain the desired result. Assume that one multiplication is executed in one clock cycle, then only four clock cycles are required to obtain the output result Res2.

[0036] The present invention also proposes a mixed-width quantization (MWQ) method for CNN weights, which quantizes different layers of the CNN into different bit widths.

[0037] MWQ quantization is to quantize the weight data into the inner product of multiple basis and vector according to the quantized bit width, and the elements of the vector only contain -1 and 1. For example, some convolutional layers can be quantized to 4b, and some convolutional layers can be quantized to 8b.

[0038] The MWQ method is to quantize 32-bit floating-point numbers into 8-bit fixed-point integers INT8 or 4-bit fixed-point integers INT4.

[0039] Specifically, the 32-bit floating-point number w is represented by the inner product of K basis vectors 〖basis = [b1, b1,..., b k 〗 and the binary-encoded vector 〖vector = [v1, v2,..., v k 〗^T, where b i∈{-1, 1}. The inner product of the basis (when INT8, N = 8; when INT4, N = 4) after being converted to an N-bit fixed-point number and the vector can represent an INT8 or INT4 fixed-point number.

[0040] As shown below, this representation can express any integer w between 0 and 2 K-1 inclusive.

[0041]

[0042] To learn the quantizers while keeping them compatible with bitwise operations, we can simply learn the basis vectors consisting of K scalars. Specifically, our quantizers are adaptively learned during network training. Each call updates the basis and the vector according to the gradient. The network training process is multi-round, that is, the process of quantizer learning gradually optimizes the combination of the basis and the vector to continuously approximate the given weight w. Optimize the left part into a trainable basis, that is, the quantizer function we can learn is in the form of:

[0043] Y out (w, basis) = basis T v l

[0044] where basis ∈ R K is the trainable floating-point basis, v1 ∈ {-1, 1} is an enumeration of 2 bit cases from [-1, …, -1, -1] to [1, …, 1, 1]. To learn the quantizers while keeping them compatible with bitwise operations, we can simply learn the basis vectors consisting of K scalars. For network training, mainly iterate to train the basis and the vector, quantize the input according to the current quantization parameters, and use this gradient information to update the basis and the vector, so as to achieve learning-based low-bit quantization, calculate the gradient and optimize the quantized output.

[0045] The following shows the quantization effect of simulating a 3×3 convolution kernel using 4b MWQ quantization, as Figure 4 shown. It can be seen that the quantized values are not much different from those before quantization, and the quantized values can be obtained from the inner product of the basis and the vector. It is not difficult to verify that the first value here -14 ≈ -1 * 4 + 1 * 6 + (-1) * 7 + (-1) * 8. This splits the quantized value into the product of -1 and 1 and the corresponding basis.

[0046] In recent years, convolutional neural networks (CNNs) have achieved remarkable success in artificial intelligence applications, exceeding human-level accuracy in many tasks. However, directly deploying CNN models on resource-constrained embedded devices poses challenges due to their large parameter sizes and high energy consumption. Hybrid bit-width networks provide a better balance between model size and hardware efficiency by adopting different bit-widths within different layers. For almost all CNNs, more than ninety percent of the computational operations are multiply-accumulate (MAC). Therefore, MAC units play a very crucial role in the design of CNN accelerators. On the FPGA platform, MWMAC units in accelerators based on hybrid bit-width networks are mainly constructed by LUT-based methods or DSP-based methods. However, LUT-based MWMAC units cannot fully utilize DSPs, and DSP-based MWMAC units support limited combinations of bit-width calculations, such as only supporting 8b weight and 4b weight calculations and being unable to efficiently support 5b weight or 3b weight calculations. Therefore, this patent designs a DSP-based MWMAC unit using the proposed MWQ algorithm, which supports a combination of weight bit-width calculations from 1b to 8b and ensures efficient utilization of DSP and LUT resources. Through synthesis and implementation in Vivado2022.2, Table 1 shows the hardware resource usage of a single MWMAC; Table 2 shows the computational cycles and throughput of a single MWMAC unit at different bit-widths at 200 MHz.

[0047] Table 1 Hardware Resource Usage of a Single MWMAC

[0048] Hardware resource usage LUT FF DSP MWMAC 51 144 1

[0049] Table 2 Computational Cycles and Throughput of a Single MWMAC at Different Bit-Widths and 200 MHz

[0050]

[0051] The above has specifically described the preferred embodiments of the present invention. However, the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A hybrid bit-width calculation unit based on DSP, characterized in that, It includes a pre-addition module, a multiplication and accumulation module, where: The pre-addition module is used to perform bit operations on the input activation and the sign vector Sign_Vconvert, and accumulate the results obtained from the bit operations in a three-stage adder tree; The multiplication and accumulation module is used to multiply and accumulate the result output by the pre-addition module with the corresponding Basis.

2. The hybrid bit-width calculation unit based on DSP according to claim 1, wherein The number of elements in Basis is consistent with the bit width of the weight data.

3. The hybrid bit-width calculation unit based on DSP according to claim 2, wherein Symbol vector Sign_V convert The elements within it are composed of -1 and 1.

4. A quantization method, characterized in that, It is used to quantize different layers of the CNN into different bit widths, including: Obtain the weight data; Represent the 32-bit floating-point number corresponding to the weight data as the inner product of K basis vectors and a binary encoding vector.

5. The quantization method according to claim 4, wherein The representation method of the floating-point number is as follows: where w represents a floating-point number and v k represents the k-th basis vector.

6. The quantization method according to claim 5, wherein The learning process of the quantizer is represented as follows: Y out (w, basis) = basis T v l where w represents a floating point number, basis represents a learnable floating point basis; v l represents the corresponding sign vector.

Images