Object detection accelerator and its binary quantization training method

Abstract

The present invention discloses a target detection accelerator and a binary quantization training method thereof, belonging to the field of neural network accelerator quantization and hardware deployment. The multiplication module in the convolution calculation unit is replaced by a lookup table module, which reduces the amount of multiplication calculation in the convolution process, and fully utilizes the characteristics of weight binarization to reduce the redundancy in the calculation process. An efficient binary neural network accelerator hardware data flow module is designed. According to the characteristics of the binary target detection neural network model, the input data reuse method is adopted to maximize the reuse of the pre-calculated results obtained in the lookup table of the convolution pre-calculation unit, thereby minimizing the redundancy in the accelerator calculation process. In the design of the hardware data flow, the advantage of the low bit width of the weight is fully utilized to reduce the operating power consumption of the accelerator. The accelerator is binary quantized and trained with a preset data set, and the weight values ​​in the network are quantized to 1 bit, which reduces the computational complexity and storage requirements of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerator quantization and hardware deployment, and in particular to a target detection accelerator and a binary quantization training method thereof. Background Art

[0002] Convolutional Neural Networks (CNNs) have achieved significant success in many computer vision tasks, including image recognition and object detection. YOLOv2-Tiny's low model complexity and balanced performance, combined with the accuracy and speed required for object detection, make it well-suited for object detection on FPGA (Field Programmable Gate Array) platforms. Due to the limited bandwidth and resources of FPGA platforms, it's necessary to reduce the number of parameters and computational overhead in object detection neural networks. Common approaches to this include model pruning and parameter quantization.

[0003] Network pruning achieves network compression by removing weights or convolution kernels that contribute less to the network, but this requires the introduction of additional encoding and decoding units, increasing the complexity of hardware design. Compared to network pruning, network quantization offers advantages as a network compression algorithm. Network quantization reduces the network parameter size by reducing the bit width without introducing additional modules. Quantization-aware training quantizes and updates the 32-bit floating-point parameters in the network during training, allowing learning to compensate for quantization-related losses during training. Quantization-aware training can compress network parameters to lower bit widths, such as ternary and binarized networks.

[0004] Binarization quantization can significantly reduce the number of parameters in target detection neural networks, reducing FPGA bandwidth usage and on-chip BRAM usage, thereby reducing FPGA power consumption when deploying target detection accelerators. However, the binary quantization method is not very effective when deployed on large-scale target detection neural networks. Although the binarized neural network significantly reduces the network model and saves accelerator computation time, the binary neural network still has a certain amount of computational redundancy when deployed on hardware, and there is room for further optimization in the hardware deployment of binary networks. Summary of the Invention

[0005] The present invention provides a target detection accelerator and a binary quantization training method thereof. A binary quantization method is adopted to optimize a large-scale YOLOv2-Tiny convolutional neural network model while ensuring detection accuracy, reduce network parameters, and design a corresponding hardware accelerator. When designing the hardware accelerator, an efficient lookup table convolution calculation unit is designed to reduce computational redundancy when deploying the binary YOLOv2-Tiny convolutional neural network on an FPGA.

[0006] A first embodiment of the present invention provides a target detection accelerator, comprising: a convolution pre-calculation unit, a convolution table lookup calculation unit, and an accelerator convolution calculation data flow module;

[0007] The convolution pre-calculation unit is used to pre-calculate the multiplication and accumulation results of the three activation input values ​​and the four weight vectors of 1, -1, -1; 1, -1, 1; 1, 1, -1; 1, 1, 1 through a lookup table, obtain the AND calculation results of the input feature map and the corresponding weights, and put the pre-calculated results into the lookup table;

[0008] The convolution table lookup calculation unit is used to first use the last two weights of a group of three weights as indexes of the coding weight to perform a table lookup operation during the convolution operation, and invert the table lookup result through the inverter, and put the inverted result and the original result into the selector as input signals, and select the table lookup result or the inverted table lookup result by judging the sign of the first weight. The three groups of weights are passed through the table lookup unit and the selector to obtain three results, and the three results are subjected to two-stage addition to obtain the final convolution calculation result;

[0009] The accelerator convolution calculation data flow module adopts input multiplexing as the data multiplexing method of the accelerator. The input channel slicing size is 32, and the output channel slicing size is 8. The input feature map of the 32 input channels is calculated by the pre-calculation unit to obtain the pre-calculation results of the input feature map and the corresponding weights and put them into the lookup table. The pre-calculation results and the corresponding 8 groups of 32-channel convolution kernels are convolved with the lookup table to obtain the final convolution calculation result. After the convolution operation is performed on the input feature map of the 32 input channels and the 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. The 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are operated through an addition tree. The addition tree obtains the output result from the 32 input results through a five-level adder. The final output result of the addition tree is subjected to BN layer multiplication and addition operations to obtain the final output feature map result.

[0010] Optionally, in one embodiment of the present invention, the convolution pre-calculation unit includes a first pre-calculation difference module, a second pre-calculation summation module, a third pre-calculation difference module, a fourth pre-calculation summation module, a fifth pre-calculation difference module, a sixth pre-calculation summation module and a lookup table storage module, and the convolution pre-calculation unit is further configured to:

[0011] Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference calculation result of the other activation input value is performed by the third pre-calculation difference calculation module. The three activation input values ​​and the corresponding output results of the weights 1, -1, -1 are stored in the first part of the lookup table through the lookup table storage module;

[0012] Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference result and the other activation input value are summed by the fourth pre-calculation summation module. The three activation input values ​​and the corresponding output results of weights 1, -1, and 1 are stored in the second part of the lookup table by the lookup table storage module;

[0013] Two of the three activation input values ​​are summed by the second pre-calculation summation module, and then the sum is subtracted from the other activation input value by the fifth pre-calculation subtraction module. The three activation input values ​​and the corresponding output results of weights 1, 1, and -1 are stored in the third part of the lookup table by the lookup table storage module;

[0014] Two of the three activation input values ​​are summed by the second pre-calculation and summation module and then summed with the other activation input value by the sixth pre-calculation and summation module. The obtained three activation input values ​​and the corresponding output results of weights 1, 1, 1 are connected to the lookup table storage module and stored in the fourth part of the lookup table.

[0015] Optionally, in one embodiment of the present invention, the convolution table lookup calculation unit includes: a first table lookup module, a second inversion module, a third selection module, a fourth table lookup summation module and a fifth table lookup summation module; the convolution table lookup calculation unit is specifically configured to:

[0016] During the convolution operation, the last two weights in a group of three weights are first used as the index of the coding weight to access the first table lookup module for table lookup, and the table lookup result is connected to the second inversion module for inversion through the inverter. The inverted result and the original result are connected to the third selection module as input signals for selection. The positive or negative sign of the first weight in a group of three weights is used to determine whether to select the table lookup result or the inverted table lookup result. If the sign of the weight is positive, the table lookup result is selected as the output of the selector. If the sign of the weight is negative, the inverted table lookup result is selected as the output of the selector. The three groups of weights pass through three groups of first table lookup modules, second inversion modules and third selection modules to obtain three results. The three results are connected to the fourth table lookup summation module and the fifth table lookup summation module for summation, and the final convolution calculation result is obtained after two-stage addition.

[0017] Optionally, in one embodiment of the present invention, the accelerator convolution calculation data flow module includes: a third addition tree module, a fourth multiplication module and a fifth data flow summation module, the third addition tree module includes a first convolution summation module, a second convolution summation module, a third convolution summation module, a fourth convolution summation module and a fifth convolution summation module, and the accelerator convolution calculation data flow module is specifically used to:

[0018] After the convolution operation is performed on the input feature maps of the 32 input channels and the 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. The 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are summed through the third addition tree module. The addition tree is composed of 32 input data passing through 5-stage adders to obtain the output result. The 32 input data of the first-stage adder are added through 16 of the first convolution summation modules to obtain the output of 16 first-stage adders. The 16 input data of the second-stage adder are added through 8 The second convolution summation module obtains 8 output results, the 8 input data of the third-level adder pass through 4 of the third convolution summation modules to obtain 4 output results, the 4 input data of the fourth-level adder pass through 2 of the fourth convolution summation modules to obtain 2 output results, the 2 input data of the fifth-level adder pass through 1 of the fifth convolution summation modules to obtain the final output result of the addition tree, the final output result of the addition tree passes through the fourth multiplication module of the BN layer to obtain the multiplication result, and the multiplication result passes through the fifth data stream summation module to obtain the final output feature map result.

[0019] A second aspect of the present invention provides a binary quantization training method for an object detection accelerator, which is used for the object detection accelerator described in the above embodiment, and includes the following steps:

[0020] Pre-train the YOLOv2-Tiny convolutional neural network model using a preset dataset to obtain 32-bit floating-point network parameters;

[0021] The pre-trained YOLOv2-Tiny convolutional neural network model is trained with a binary weight network method to quantize the weights of the YOLOv2-Tiny convolutional neural network model to 1 bit.

[0022] The target detection accelerator and its binary quantization training method proposed in the present invention have the following beneficial effects:

[0023] 1. We designed a binary YOLOv2-Tiny object detection neural network model and trained it using the PASCAL VOC 2007+2012 dataset. This quantized the weights in the YOLOv2-Tiny object detection neural network to 1 bit, reducing the model's computational complexity and storage requirements. The trained network achieved a mean average accuracy (mAP) of 50.22%, only 6.88% lower than the full-precision network.

[0024] 2. A convolution calculation unit based on a lookup table is designed. The multiplication module in the convolution calculation unit is replaced by a lookup table module, which effectively reduces the amount of multiplication calculation in the convolution process and makes full use of the characteristics of weight binarization to reduce the redundancy in the calculation process.

[0025] 3. An efficient binary neural network accelerator convolution calculation data flow was designed. According to the characteristics of the binary YOLOv2-Tiny convolutional neural network model, the input data reuse method was used to maximize the reuse of the pre-calculated results obtained in the lookup table of the convolution pre-calculation unit, thereby minimizing the redundancy in the accelerator calculation process. In the design of the convolution calculation data flow, the advantage of low bit width of the weight was fully utilized to reduce the operating power consumption of the accelerator.

[0026] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of the hardware architecture of a pre-calculation unit provided according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the hardware structure of a convolution calculation unit based on a lookup table according to an embodiment of the present invention;

[0030] Figure 3A schematic diagram of the hardware structure of an accelerator convolution calculation data flow module provided according to an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the hardware structure of a third addition tree module in the accelerator convolution calculation data flow provided according to an embodiment of the present invention;

[0032] Figure 5 This is a flow chart of a binary quantization training method for an object detection accelerator provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0034] The target detection accelerator proposed in an embodiment of the present invention includes: a convolution calculation unit including a pre-calculation unit, a convolution table lookup calculation unit and an accelerator convolution calculation data flow module.

[0035] The convolution calculation unit consists of a convolution pre-calculation unit and a convolution lookup table calculation unit. The convolution pre-calculation unit uses a lookup table to pre-calculate the multiplication and accumulation results of the three activation input values ​​with the four weight vectors: 1, -1, -1; 1, -1, 1; 1, 1, -1; and 1, 1, 1. This calculation then calculates the AND result of the input feature map and the corresponding weights and stores the pre-calculated result in the lookup table.

[0036] The convolution lookup table calculation unit is used to first use the last two weights in a group of three weights as the index of the coding weight to perform a lookup table operation during the convolution operation, and invert the lookup table result through the inverter. The inverted result and the original result are put into the selector as input signals, and the lookup table result or the inverted lookup table result is selected by judging the positive or negative sign of the first weight. The three groups of weights obtain three results through the lookup table unit and the selector, and the three results are added in two stages to obtain the final convolution calculation result.

[0037] The accelerator's convolution calculation data flow module uses input multiplexing as the accelerator's data multiplexing method. The input channel slice size is 32, and the output channel slice size is 8. The pre-calculation unit calculates the input feature map of the 32 input channels, and the pre-calculated results of the input feature map and the corresponding weights are placed in a lookup table. The pre-calculated results are then convolved with the corresponding 8 groups of 32-channel convolution kernels to obtain the final convolution calculation results. After the convolution operation is performed on the input feature map of the 32 input channels and the 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. The 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are operated on by an addition tree. The addition tree outputs the 32 input results through a five-level adder. The final output of the addition tree is subjected to BN layer multiplication and addition operations to obtain the final output feature map result.

[0038] The designed pre-calculation unit calculates the multiplication and accumulation results of the three activation input values ​​and the four weight vectors 1, -1, -1; 1, -1, 1; 1, 1, -1; 1, 1, 1 by designing a lookup table, and puts these four results into the lookup table.

[0039] like Figure 1 As shown, the convolution pre-calculation unit (1) includes a first pre-calculation difference module (1.1), a second pre-calculation summation module (1.2), a third pre-calculation difference module (1.3), a fourth pre-calculation summation module (1.4), a fifth pre-calculation difference module (1.5), a sixth pre-calculation summation module (1.6) and a lookup table storage module (1.7), and the convolution pre-calculation unit is further used to,

[0040] Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference is calculated with the other activation input value by the third pre-calculation difference calculation module. The three activation input values ​​and the corresponding output results of weights 1, -1, -1 are stored in the first part of the lookup table through the lookup table storage module;

[0041] Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference is summed with the other activation input value by the fourth pre-calculation summation module. The three activation input values ​​and the corresponding output results of the weights 1, -1, and 1 are stored in the second part of the lookup table by the lookup table storage module;

[0042] Two of the three activation input values ​​are summed by the second pre-calculation summation module, and then the sum is subtracted from the other activation input value by the fifth pre-calculation subtraction module. The three activation input values ​​and the corresponding output results of weights 1, 1, and -1 are stored in the third part of the lookup table through the lookup table storage module;

[0043] Two of the three activation input values ​​are summed by the second pre-calculation summation module and then summed with the other activation input value by the sixth pre-calculation summation module. The obtained three activation input values ​​and the corresponding output results of weights 1, 1, 1 are stored in the fourth part of the lookup table through the lookup table storage module.

[0044] like Figure 2 As shown, in the embodiment of the present invention, the convolution table lookup calculation unit (2) includes: a first table lookup module (2.1), a second inversion module (2.2), a third selection module (2.3), a fourth table lookup summation module (2.4) and a fifth table lookup summation module (2.5); the convolution table lookup calculation unit is specifically used to,

[0045] During the convolution operation, the last two weights in a group of three weights are first used as the index of the coding weight to access the first lookup table module for table lookup, and the table lookup result is connected to the second inversion module for inversion through the inverter. The inverted result and the original result are connected as input signals to the third selection module for selection. The positive or negative sign of the first weight in a group of three weights is used to determine whether to select the table lookup result or the inverted table lookup result. If the sign of the weight is positive, the table lookup result is selected as the output of the selector. If the sign of the weight is negative, the inverted table lookup result is selected as the output of the selector. The three groups of weights pass through three groups of first lookup table modules, second inversion modules and third selection modules to obtain three results. The three results are connected to the fourth table lookup summation module and the fifth table lookup summation module for summation, and the final convolution calculation result is obtained after two-stage addition.

[0046] In the convolution calculation unit based on the lookup table, 9 multiplication modules are replaced by 3 lookup table modules, which can effectively reduce the amount of multiplication calculation in the convolution process and make full use of the characteristics of weight binarization to reduce redundancy in the calculation process.

[0047] like Figure 3 and Figure 4 As shown, the accelerator convolution calculation data flow module includes: a third addition tree module (3), a fourth multiplication module (4.1) and a fifth data flow summation module (4.2), and the third addition tree module includes a first convolution summation module (3.1), a second convolution summation module (3.2), a third convolution summation module (3.3), a fourth convolution summation module (3.4) and a fifth convolution summation module (3.5).

[0048] The accelerator's convolution calculation data flow module uses input multiplexing as its data multiplexing method. The input channel slice size is 32, and the output channel slice size is 8. The first pre-calculation unit calculates the input feature maps of the 32 input channels, and the pre-calculated results of the input feature maps and corresponding weights are stored in a lookup table. The pre-calculated results are then used with the corresponding eight groups of 32-channel convolution kernels for convolution table lookup calculations. During the convolution table lookup operation, the last two weights of the three weights in the convolution kernel are used as encoding weights and indexed into the first lookup table module for lookup. The lookup result is then connected to the second inversion module for inversion via an inverter. The inverted result and the original result are then connected as input signals to the third selection module for selection. The sign of the first weight in the group of three weights determines whether to select the lookup result or the inverted one. The three groups of weights are passed through three groups of the first lookup table module, the second inversion module, and the third selection module to obtain three results. These three results are then summed in the fourth and fifth lookup table summation modules, and the final convolution result is obtained through two-stage addition.

[0049] After the convolution operation is performed on the input feature map of 32 input channels and 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. These 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are summed through the third addition tree module. The addition tree is composed of 32 input data passing through 5-level adders to obtain the output result. The 32 input data of the first-level adder are added through 16 first convolution summation modules to obtain the output of 16 first-level adders. The 16 input data of the second-level adder are summed through 8 The second convolution summation module obtains 8 output results, the 8 input data of the third-level adder pass through 4 third convolution summation modules to obtain 4 output results, the 4 input data of the fourth-level adder pass through 2 fourth convolution summation modules to obtain 2 output results, the 2 input data of the fifth-level adder pass through 1 fifth convolution summation module to obtain the final output result of the addition tree, the final output result of the addition tree passes through the fourth multiplication module of the BN layer to obtain the multiplication result, and the multiplication result passes through the fifth data stream summation module to obtain the final output feature map (partial sum) result.

[0050] Specifically, the convolution calculation unit based on the lookup table is designed to include a convolution pre-calculation unit and a convolution lookup table calculation unit. The hardware architecture of the designed pre-calculation unit is as follows: Figure 1As shown, two of the three input values ​​a0 and a1 of the pre-calculation unit are subjected to difference calculation by the first pre-calculation difference calculation module (1.1), and the result is then subjected to difference calculation by the third pre-calculation difference calculation module (1.3) to obtain the output results a0-a1-a2 of a0, a1, a2 and the corresponding weights 1, -1, -1, which are then connected to the lookup table storage module (1.7) and stored in the first part of the LUT; two of the three input values ​​a0 and a1 of the pre-calculation unit are subjected to difference calculation by the first pre-calculation difference calculation module (1.1), and the result is then subjected to summation by the fourth pre-calculation summation module (1.4) to obtain the output results a0-a1+a2 of a0, a1, a2 and the corresponding weights 1, -1, 1, which are then connected to the lookup table storage module (1.7) and stored in the second part of the LUT. In the embodiment of the present invention, two of the three input values ​​of the pre-calculation unit, a0 and a1, are summed by the second pre-calculation summing module (1.2), and the result is then subtracted from a2 by the fifth pre-calculation subtraction module (1.5) to obtain a0, a1, a2 and the corresponding output results a0+a1-a2 of weights 1, 1, -1, which are then connected to the lookup table storage module (1.7) and stored in the third part of the LUT; two of the three input values ​​of the pre-calculation unit, a0 and a1, are summed by the second pre-calculation summing module (1.2), and the result is then summed from a2 by the sixth pre-calculation summing module (1.6) to obtain a0, a1, a2 and the corresponding output results a0+a1+a2 of weights 1, 1, 1, which are then connected to the lookup table storage module (1.7) and stored in the fourth part of the LUT. The pre-calculation unit designed in the design has a lookup table. It first calculates the multiplication and accumulation results of the activation input values ​​a0, a1, a2 with the four weight vectors 1, -1, -1; 1, -1, 1; 1, 1, -1; 1, 1, 1, and then puts the four results a0-a1-a2, a0-a1+a2, a0+a1-a2, and a0+a1+a2 into the lookup table. The convolution calculation unit based on the lookup table is as follows: Figure 2 As shown, in the pre-calculation unit 1, the input feature map and the corresponding weight convolution operation result are placed in the lookup table. During the convolution operation, the last two w1 and w2 of a group of three weights w0, w1, and w2 are first used as coding weights as indexes to access the first lookup module (2.1) for lookup. Then, the lookup result is connected to the second inversion module (2.2) and inverted through the inverter. The inverted result and the original result are connected as input signals to the third selection module (2.3) for selection. The sign of w0 determines whether to select the lookup result or the inverted lookup result. If the sign of w0 is positive, the lookup result is selected as the output of the selector. If the sign of w0 is negative, the inverted lookup result is selected as the output of the selector. The three groups of weights are processed through three first table lookup modules (2.1), second negation modules (2.2) and third selection modules (2.3) to obtain three results. The three results are then connected to a fourth table lookup summation module (2.4) and a fifth table lookup summation module (2.5) for summation, and the final convolution calculation result is obtained through two-stage addition.

[0051] The hardware architecture of the accelerator convolution calculation data flow module is as follows Figure 3 As shown, an input multiplexing method is used as the data multiplexing method of the accelerator, the input channel slice size is 32, the output channel slice size is 8, the input feature map of the 32 input channels is calculated by the first pre-calculation unit (1), and the pre-calculation result of the input feature map and the corresponding weight is put into the lookup table. The pre-calculation result and the corresponding 8 groups of 32-channel convolution kernels are used for convolution table calculation. When performing the convolution table lookup operation, the last two w1 and w2 of the three weights w0, w1, and w2 in the convolution kernel are first used as encoding weights as indexes to access the first lookup module (2.1) for table lookup. Then, the lookup result is connected to the second inversion module (2.2) and inverted through the inverter. The inverted result and the original result are used as input signals to access the third selection module (2.3) for selection. The positive or negative sign of w0 determines whether to select the lookup result or the inverted lookup result. The three sets of weights are fed into the table lookup unit and the selector to obtain three results. The three results are fed into the fourth table lookup summation module (2.4) and the fifth table lookup summation module (2.5) for summation. The final convolution calculation result is obtained through two-stage addition. After the convolution operation is performed on the input feature maps of the 32 input channels and the 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. These 256 convolution calculation results are divided into 8 groups, and the 32 convolution calculation results in each group are summed through the third addition tree module (3). The hardware architecture of the third addition tree module (3) is as follows: Figure 4 As shown, the third addition tree module (3) is from s0 to s 31 A total of 32 inputs are fed into a 5-stage adder to obtain the output result. The first-stage adder s0 to s 31 These 32 inputs are added together through 16 first-level convolution summation modules (3.1) to produce the outputs of 16 first-level adders. The 16 inputs of the second-level adder are added through 8 second-level convolution summation modules (3.2) to produce 8 outputs. The 8 inputs of the third-level adder are added through 4 third-level convolution summation modules (3.3) to produce 4 outputs. The 4 inputs of the fourth-level adder are added through 2 fourth-level convolution summation modules (3.4) to produce 2 outputs. The 2 inputs of the fifth-level adder are added through 1 fifth-level convolution summation module (3.5) to produce the final output of the addition tree. The final output of the addition tree is connected to the BN layer m0 input parameter and then connected to the fourth multiplication module (4.1) to obtain the multiplication result. The multiplication result and the BN layer n0 input parameter are connected to the fifth data stream summation module (4.2) to obtain the final output feature map (partial sum).

[0052] The following describes a binary quantization training method for an object detection accelerator proposed in an embodiment of the present invention.

[0053] Figure 5This is a flow chart of a binary quantization training method for an object detection accelerator provided according to an embodiment of the present invention.

[0054] like Figure 5 As shown, the binary quantization training method for the target detection accelerator is used for the target detection accelerator proposed in the above embodiment. The binary quantization training method for the target detection accelerator includes the following steps:

[0055] In step S101, the YOLOv2-Tiny convolutional neural network model is pre-trained using a preset data set to obtain 32-bit floating-point network parameters.

[0056] In step S102, the pre-trained YOLOv2-Tiny convolutional neural network model is subjected to quantization perception training using a binary weight network method to quantize the weight of the YOLOv2-Tiny convolutional neural network model to 1 bit.

[0057] In an embodiment of the present invention, the YOLOv2-Tiny network is pre-trained using the PASCAL VOC2007 and PASCAL VOC2012 datasets to obtain 32-bit floating-point network parameters. The pre-trained network is then subjected to quantization-aware training using the Binary-Weight-Network (BWN) method, quantizing the network weights to 1.

[0058] During the weight binarization process, since the changes in parameters caused by gradient descent are tiny, the changes in the binarized weights can be ignored and training optimization cannot be performed. The binarized weights are only used as an intermediate process for updating the weights, and the full-precision weights W are still used during the update.

[0059] The weight of each kernel in the single-weight binarization network (BWN) is represented by only two values, as shown in formula (1):

[0060]

[0061] in Represents binary weights. For a binary weighted network, the convolutional layer formula is as follows: is the weighted average of the i-th channel, and x is the input activation convolved with the i-th weight kernel:

[0062]

[0063] The training process of a single-weight binarized network (BWN) is as follows: During training, the binarized network needs to store two sets of convolution kernel parameters in memory. One set is the full-precision convolution kernel parameters, which are used to calculate the gradients of each parameter and update the full-precision parameters; the other set is the binary convolution kernel parameters, which are actually involved in the convolution operation of the binarized network. During training, the binary convolution kernel parameters are still floating-point data, and the two values ​​are either 1.0 or -1.0, plus a floating-point parameter.

[0064] Binarized weight training is divided into three steps: forward propagation, reverse gradient descent, and parameter update. In the backpropagation process, the weights need to be updated, so the derivative of the loss function with respect to the weights needs to be calculated. However, binarized functions such as the sign function are usually non-differentiable. Since the derivative of the sign function is almost always 0, and even some derivative values ​​of the function disappear, a direct estimator is needed to solve the gradient problem caused by the backpropagation process. The direct estimator uses the function shown in formula (3) to replace the binarized function in the backpropagation process for approximation. The derivative is 1 in the range of [-1, 1], and the parameters outside the range are meaningless in the binarization and the derivative is 0.

[0065]

[0066] The target detection accelerator and its binary quantization training method in the embodiment of the present invention use a preset data set to perform binary quantization training on the YOLOv2-Tiny target detection neural network, quantizing the weight values ​​in the network to 1 bit, thereby reducing the computational complexity and storage requirements of the model. A lookup table-based convolution calculation unit is designed, and the multiplication module in the convolution calculation unit is replaced by a lookup table module, effectively reducing the amount of multiplication calculations in the convolution process and fully utilizing the characteristics of weight binarization to reduce redundancy in the calculation process. Finally, an efficient binary neural network accelerator hardware data flow module is designed. Based on the characteristics of the binary YOLOv2-Tiny target detection neural network model, input data reuse is used to maximize the reuse of the pre-calculated results obtained in the lookup table of the convolution pre-calculation unit, thereby minimizing redundancy in the accelerator's calculation process. In the design of the hardware data flow, the advantage of low bit width of weights is fully utilized to reduce the operating power consumption of the accelerator.

[0067] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0069] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A target detection accelerator, characterized in that: include: Convolution pre-calculation unit, convolution table lookup calculation unit and accelerator convolution calculation data flow module; The convolution pre-calculation unit is used to add three activation input values ​​to 1, -1, -1 through a lookup table; 1，-1，1；1，1，-1； The multiplication and accumulation results of the four weight vectors 1, 1, and 1 are pre-calculated to obtain the AND calculation results of the input feature map and the corresponding weights, and the pre-calculated results are placed in the lookup table; The convolution table lookup calculation unit is used to first use the last two weights of a group of three weights as indexes of the coding weight to perform a table lookup operation during the convolution operation, and invert the table lookup result through the inverter, and put the inverted result and the original result into the selector as input signals, and select the table lookup result or the inverted table lookup result by judging the sign of the first weight. The three groups of weights are passed through the table lookup unit and the selector to obtain three results, and the three results are subjected to two-stage addition to obtain the final convolution calculation result; The accelerator convolution calculation data flow module adopts input multiplexing as the data multiplexing mode of the accelerator, the input channel slicing size is 32, and the output channel slicing size is 8. The input feature map of the 32 input channels is calculated by the pre-calculation unit to obtain the pre-calculation results of the input feature map and the corresponding weights and put them into the lookup table. The pre-calculation results and the corresponding 8 groups of 32-channel convolution kernels are convolved with the lookup table to obtain the final convolution calculation result. After the input feature map of the 32 input channels and the 8 groups of 32-channel convolution kernels complete the convolution operation, 256 convolution calculation results are obtained. The 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are operated through an addition tree. The addition tree obtains the output result from the 32 input results through a five-level adder. The final output result of the addition tree obtains the final output feature map result after the BN layer multiplication and addition operation.

2. The accelerator according to claim 1, characterized in that The convolution pre-calculation unit includes a first pre-calculation difference module, a second pre-calculation summation module, a third pre-calculation difference module, a fourth pre-calculation summation module, a fifth pre-calculation difference module, a sixth pre-calculation summation module and a lookup table storage module, and the convolution pre-calculation unit is further used to: Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference calculation result of the other activation input value is performed by the third pre-calculation difference calculation module. The three activation input values ​​and the corresponding output results of the weights 1, -1, -1 are stored in the first part of the lookup table through the lookup table storage module; Two of the three activation input values ​​are subjected to difference calculation by the first pre-calculation difference calculation module, and then the difference result and the other activation input value are summed by the fourth pre-calculation summation module. The three activation input values ​​and the corresponding output results of weights 1, -1, and 1 are stored in the second part of the lookup table by the lookup table storage module; Two of the three activation input values ​​are summed by the second pre-calculation summation module, and then the sum is subtracted from the other activation input value by the fifth pre-calculation subtraction module. The three activation input values ​​and the corresponding output results of weights 1, 1, and -1 are stored in the third part of the lookup table by the lookup table storage module; Two of the three activation input values ​​are summed by the second pre-calculation and summation module and then summed with the other activation input value by the sixth pre-calculation and summation module. The obtained three activation input values ​​and the corresponding output results of weights 1, 1, 1 are connected to the lookup table storage module and stored in the fourth part of the lookup table.

3. The accelerator according to claim 2, characterized in that The convolution table lookup calculation unit includes: a first table lookup module, a second inversion module, a third selection module, a fourth table lookup summation module and a fifth table lookup summation module; the convolution table lookup calculation unit is specifically used to: During the convolution operation, the last two weights in a group of three weights are first used as the index of the coding weight to access the first table lookup module for table lookup, and the table lookup result is connected to the second inversion module for inversion through the inverter. The inverted result and the original result are connected to the third selection module as input signals for selection. The positive or negative sign of the first weight in a group of three weights is used to determine whether to select the table lookup result or the inverted table lookup result. If the sign of the weight is positive, the table lookup result is selected as the output of the selector. If the sign of the weight is negative, the inverted table lookup result is selected as the output of the selector. The three groups of weights pass through three groups of first table lookup modules, second inversion modules and third selection modules to obtain three results. The three results are connected to the fourth table lookup summation module and the fifth table lookup summation module for summation, and the final convolution calculation result is obtained after two-stage addition.

4. The accelerator according to claim 3, characterized in that The accelerator convolution calculation data flow module includes: a third addition tree module, a fourth multiplication module and a fifth data flow summation module, the third addition tree module includes a first convolution summation module, a second convolution summation module, a third convolution summation module, a fourth convolution summation module and a fifth convolution summation module, and the accelerator convolution calculation data flow module is specifically used to: After the convolution operation is performed on the input feature maps of the 32 input channels and the 8 groups of 32-channel convolution kernels, 256 convolution calculation results are obtained. The 256 convolution calculation results are divided into 8 groups. The 32 convolution calculation results in each group are summed through the third addition tree module. The addition tree is composed of 32 input data passing through 5-stage adders to obtain the output result. The 32 input data of the first-stage adder are added through 16 of the first convolution summation modules to obtain the output of 16 first-stage adders. The 16 input data of the second-stage adder are added through 8 The second convolution summation module obtains 8 output results, the 8 input data of the third-level adder pass through 4 of the third convolution summation modules to obtain 4 output results, the 4 input data of the fourth-level adder pass through 2 of the fourth convolution summation modules to obtain 2 output results, the 2 input data of the fifth-level adder pass through 1 of the fifth convolution summation modules to obtain the final output result of the addition tree, the final output result of the addition tree passes through the fourth multiplication module of the BN layer to obtain the multiplication result, and the multiplication result passes through the fifth data stream summation module to obtain the final output feature map result.

5. A binary quantization training method for a target detection accelerator, used for the target detection accelerator according to any one of claims 1 to 4, characterized in that: The following steps are involved: Pre-train the YOLOv2-Tiny convolutional neural network model using a preset dataset to obtain 32-bit floating-point network parameters; The pre-trained YOLOv2-Tiny convolutional neural network model is trained with a binary weight network method to quantize the weights of the YOLOv2-Tiny convolutional neural network model to 1 bit.

Images