A sparse method and accelerator design for target detection network

Abstract

The application discloses a kind of sparsification method and accelerator design for target detection network, for YOLOv2-tiny target detection network, under the condition of guaranteeing certain accuracy, the network is sparsified, and it has hardware friendliness, the designed accelerator can efficiently support the network model operation after sparsification, the designed computing unit has two kinds of calculation modes, can make full use of the network sparsity brought by this sparsification method.The core of the sparsification method is to remove the values with small absolute values in the convolution kernel unit, and to encode the positions of the remaining non-zero values, providing the computing unit with feature map data and weight matching during calculation, reducing a large amount of redundant calculation, saving storage space, improving running speed, so that the target detection algorithm can run efficiently on the resource-limited FPGA platform.

Description

Technical Field

[0001] This invention relates to a sparsification method and accelerator design for target detection neural networks, belonging to the field of sparsified accelerators. Background Technology

[0002] Object detection technology has broad application prospects in fields such as autonomous driving, security monitoring, medical image analysis, and drone tracking and positioning. Convolutional neural networks (CNNs) have achieved excellent results in object detection, but they involve large amounts of data, are computationally intensive, and have high demands on hardware memory, bandwidth, and computing resources. To enable deployment on lower-cost hardware and improve operational efficiency, many model compression methods have been proposed, such as model sparsification and quantization, to reduce the model's storage and computational requirements.

[0003] Sparsity pruning can effectively compress the network structure while maintaining a certain level of accuracy, reducing the number of model parameters and the required computation. Currently, sparsity pruning schemes can be divided into two types: structured sparsity and unstructured sparsity. Structured sparsity disrupts the original network structure and has a lower degree of sparsity, but it is very hardware-friendly. Unstructured sparsity has a higher degree of sparsity, but it is more difficult to implement in hardware due to the irregularity of the non-zero weight distribution and a series of problems such as computational load balancing.

[0004] Although YOLOv2-tiny (You Only Look Once, the second-generation lightweight version of the YOLO object detection network) is a relatively simplified object detection network model, its parameter count is still as high as 63.5MB, and the computational requirements are still very large. Efficient deployment on embedded platforms with limited hardware resources still faces many difficulties, and further compression of its network model is needed. Summary of the Invention

[0005] Technical Problem: To address the above-mentioned problems, this invention discloses a sparsification method and accelerator design for target detection networks. It not only efficiently compresses the target detection network model but also takes into account hardware friendliness. The hardware can support its operation with simple optimization. The designed hardware accelerator makes full use of the sparsity of the model, reduces storage and computation, and improves detection efficiency and hardware utilization.

[0006] Technical solution: The present invention provides a sparsification method for object detection networks as follows: The sparsification method uses the kernel size of the object detection convolutional neural network as a group of 3×3, sorts the absolute values ​​of the weights, resets the weights with absolute values ​​below the threshold to 0 as required, performs 4:9 and 2:9 sparsification in layers, and records and encodes the positions of non-zero values; using the two sparsification modes of 4:9 and 2:9, the non-zero values ​​are multiplied and accumulated.

[0007] The sparsification method uses 3×3 convolutional kernels as grouping units and performs N:9 sparsification to reduce the design difficulty of hardware load balancing. The sparsification ratio of each convolutional layer can be set and adjusted according to different requirements for accuracy and computing speed, which is flexible.

[0008] Specifically, the sparsification method involves a YOLOv2-tiny object detection network with nine convolutional layers. The first, second, and third layers (shallow convolutional layers) are sparsified using a 4:9 ratio, meaning the absolute values ​​of the weights are sorted from largest to smallest, retaining the first four weights with the largest absolute values. The third, fourth, fifth, sixth, seventh, and eighth layers (deep convolutional layers) are sparsified using a 2:9 ratio, meaning the absolute values ​​of the weights are sorted from largest to smallest, retaining the first two weights with the largest absolute values. The ninth convolutional layer has a size of 1×1 and is not sparsified.

[0009] This invention presents an accelerator designed using a sparsity method for target detection networks. The accelerator includes an input feature map buffer unit, a non-zero value location index buffer unit, a weight buffer unit, a non-zero value location decoding unit, a computation unit, a pooling and ReLU unit, an output buffer unit, an off-chip storage and control unit, and an accumulation unit. The outputs of the off-chip storage and control unit are connected to the input feature map buffer unit, the non-zero value location index buffer unit, and the weight buffer unit, respectively. The outputs of the input feature map buffer unit and the weight buffer unit are connected to the computation unit. The output of the non-zero value location index buffer unit is connected to the non-zero value location decoding unit, and the obtained location information is output to the computation unit. The output of the computation unit is output through the accumulation unit and connected to the output buffer unit. The output of the output buffer unit is connected to the pooling and ReLU unit, and the output of the pooling and ReLU unit after processing is connected to the off-chip storage and control unit.

[0010] The computational unit section comprises multiple computational units forming a computational array. Each computational unit section includes a feature map input port, a position information input port, a weight input port, a mode control input port, a first convolution calculation output port, a second convolution calculation output port, a first input feature map shift register, a second input feature map shift register, a third input feature map shift register, an input feature map selection unit, an input weight register, a mode selection control unit, a multiplier, and an adder. The output of the feature map input port is connected to the first input feature map shift register, the output of the position information input port is connected to the input feature map selection unit, the output of the weight input port is connected to the input weight register, the output of the mode control input port is connected to the mode selection control unit, and the output of the mode selection control unit is connected to the first convolution calculation output port and the second convolution calculation output port, respectively.

[0011] The feature map input port inputs the feature Figure 3 The input features are grouped into the first input feature map shift register, and then continuously shifted to the second and third input feature map shift registers. The input feature map is matched with the output of the input weight register by the input feature map selection unit. In 4:9 mode calculation, the four weights come from the same convolution kernel; in 2:9 mode calculation, the four weights are grouped in pairs from two different convolution kernels. After matching, a multiplication operation is performed, and the resulting signals are added pairwise to obtain the first and second multiplication accumulation results. These results are then selected and output by the mode selection control unit. The selected outputs for both 4:9 and 2:9 mode convolution calculations, and the 2:9 mode convolution calculation result, are output through the first and second convolution calculation output ports, respectively.

[0012] The computation array, composed of the computational units, employs a hybrid data stream reuse strategy. For the YOLOv2-tiny object detection network, the first convolutional layer has relatively little weight data, so a weight reuse approach is used; the remaining layers have a larger amount of weight data, so a feature map reuse approach is used.

[0013] The steps to sparsify the YOLOv2-tiny object detection network and run it on an accelerator are as follows:

[0014] Step 1: Pre-train the YOLOv2-tiny object detection network to obtain a pre-trained model. Using 3×3 convolutional kernels as units, compare the absolute values ​​of the weights and reset the weights below the threshold to 0. At the same time, use encoding to record the position information of the weights.

[0015] Step 2: Retrain the sparsified model and prevent the updates of weights that have been set to zero based on the position information of the encoded records;

[0016] Step 3: After obtaining the sparse model through the first two steps, extract the weights and record the indices of the non-zero weight positions;

[0017] Step 4: After completing the above steps, the accelerator performs calculations: First, the input feature map from outside the accelerator is transmitted to the input feature map buffer unit via the AXI4 bus. The non-zero value position index data is transmitted to the non-zero value position index buffer unit via the AXI4 bus. The weight data is transmitted to the weight read / write buffer unit via the AXI4 bus. The off-chip storage and control unit configures the storage format of the input feature map and the calculation mode of the calculation unit. The input feature map and weights are input into the calculation unit in the form required for calculation. The non-zero value position index is input to the non-zero value position decoding unit to obtain position information and then input into the calculation unit. The calculation results are accumulated in the accumulation unit, temporarily stored in the output buffer unit, and input in batches into the pooling and ReLU units for max pooling and ReLU operations. The output results are transmitted to the memory outside the accelerator via the AXI4 bus.

[0018] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following advantages:

[0019] 1. The sparsity scheme based on convolution kernel size constraint ensures a high sparsity rate, reduces model storage and computation, increases the regularity of weights after sparsification, improves hardware friendliness, and is easy to implement in hardware.

[0020] 2. The accelerator designed based on the selected sparsity scheme can make full use of the sparsity brought about by network sparsity and is compatible with both 4:9 and 2:9 modes. In the 2:9 mode, the computational parallelism can be doubled, which improves the utilization of computing resources and can achieve higher throughput and operating efficiency under the same hardware resource conditions.

[0021] 3. The computing unit, designed based on the sparsity scheme and operation mode, supports two computing modes: 4:9 and 2:9. After the input calculator module is filled, the convolution calculation result can be output in each cycle. Under the same conditions, the use of computing resources such as multipliers is saved.

[0022] This invention, starting from the perspective of compressing the target detection network model and improving the accelerator throughput, designs a hardware-friendly sparsification method for the target detection network, and designs an accelerator that can fully utilize its sparsity for the sparsification scheme, which greatly reduces the storage and computation of the network model and improves the throughput and computing resource utilization. Attached Figure Description

[0023] Figure 1This is a schematic diagram of a 4:9 sparsity reduction method for target detection networks according to an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of a 2:9 sparsity reduction method for target detection networks according to an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the overall structure of a sparse accelerator design according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a computing unit structure that supports two sparsity modes, 4:9 and 2:9, according to an embodiment of the present invention.

[0027] The diagram includes: input feature map cache unit 1, non-zero value position index cache unit 2, weight cache unit 3, non-zero value position decoding unit 4, calculation unit 5, pooling and ReLU unit 6, output cache unit 7, off-chip storage and control unit 8, accumulation unit 9; feature map input port 21, position information input port 22, weight input port 23, mode control input port 24, first convolution calculation output port 25, second convolution calculation output port 26, first input feature map shift register unit 27, second input feature map shift register unit 28, third input feature map shift register unit 29, input feature map selection unit 210, input weight register unit 211, and mode selection control unit 212. Detailed Implementation

[0028] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0029] First, the weights of the original YOLOv2-tiny network are extracted and sparsified. Using 3×3 convolutional kernels as units, the absolute values ​​of the weights are compared, and weights with smaller absolute values ​​are pruned. YOLOv2-tiny has nine convolutional layers. The shallower convolutional layers (layers 1, 2, and 3) are sparsified using a 4:9 split. The absolute values ​​of the weights are sorted from largest to smallest, and the first four weights with the largest absolute values ​​are retained. Figure 1 The deep convolutional layers (layers 3, 4, 5, 6, 7, and 8) are sparsified using a 2:9 ratio. The weights are then sorted by absolute value from largest to smallest, retaining the first four weights with the largest absolute values. Figure 2 Simultaneously, the positions of non-zero weights are encoded. The kernel size of the 9th convolutional layer is 1×1, and no sparsification is performed.

[0030] The hardware accelerator described herein has the following overall structure: Figure 3As shown, the system includes an input feature map buffer unit 1, a non-zero value position index buffer unit 2, a weight buffer unit 3, a non-zero value position decoding unit 4, a calculation unit 5, a pooling and ReLU unit 6, an output buffer unit 7, an off-chip storage and control unit 8, and an accumulation unit 9. The off-chip storage and control unit 8 is connected to the input feature map buffer unit 1, the non-zero value position index buffer unit 2, the weight buffer unit 3, and the pooling and ReLU unit 6. The outputs of the input feature map buffer unit 1 and the weight buffer unit 3 are connected to the calculation unit 5. The output of the non-zero value position index buffer unit 2 is connected to the non-zero value position decoding unit 4, and the obtained position information is output to the calculation unit 5. The output of the calculation unit 5 is processed by the accumulation unit 9 and then connected to the output buffer unit 7. The output buffer unit is connected to the pooling and ReLU unit 6, and after processing, the output is sent to the off-chip storage and control unit 8.

[0031] The computational unit section 5 comprises multiple units, forming a computational array. Each computational unit section 5 includes a feature map input port 21, a position information input port 22, a weight input port 23, a mode control input port 24, a first convolution calculation output port 25, a second convolution calculation output port 26, a first input feature map shift register 27, a second input feature map shift register 28, a third input feature map shift register 29, an input feature map selection unit 210, an input weight register 211, a mode selection control unit 212, a multiplier, and an adder. The output of the feature map input port 21 is connected to the first input feature map shift register 27, the output of the position information input port 22 is connected to the input feature map selection unit 210, the output of the weight input port 23 is connected to the input weight register 211, the output of the mode control input port 24 is connected to the mode selection control unit 212, and the output of the mode selection control unit 212 is connected to the first convolution calculation output port 25 and the second convolution calculation output port 26, respectively.

[0032] Feature map input port 21 input features Figure 3The input features are grouped into the first input feature map shift register 27 and continuously shifted to the second input feature map shift register 28 and the third input feature map shift register 29. The input feature map is matched with the weights (output of the input weight register 211) through the input feature map selection unit 210. In 4:9 mode calculation, the four weights come from the same convolution kernel. In 2:9 mode calculation, the four weights are grouped in pairs from two convolution kernels. After matching, a multiplication operation is performed. The signals after the multiplication operation are added in pairs to obtain the first multiplication accumulation result 213 and the second multiplication accumulation result 214. These are then selected and output by the mode selection control unit 212. The selected output 4:9 / 2:9 mode convolution calculation result 215 and the second 2:9 mode convolution calculation result 216 are then output through the first convolution calculation output port 25 and the second convolution calculation output port 26, respectively.

[0033] The computing array composed of multiple computing units 5 adopts a hybrid data reuse strategy. For the YOLOv2-tiny object detection network, the weight data of the first convolutional layer is relatively small, so the weight reuse method is adopted. The weight data of the remaining layers is much larger than the input feature map data, so the feature map reuse method is adopted.

[0034] The steps for implementing sparsification of the YOLOv2-tiny target detection network and its operation in the accelerator, as described in the target detection network sparsification method and accelerator design, are as follows:

[0035] Step 1: Pre-train the YOLOv2-tiny object detection network to obtain a pre-trained model. Using 3×3 convolutional kernels as units, compare the absolute values ​​of the weights and reset weights below a threshold to 0. Apply different sparsity ratios to different convolutional layers, such as 4:9 sparsity to shallow convolutional layers. Figure 1 ), and perform 2:9 sparsification on deep convolutional layers ( Figure 2 At the same time, the position information of the weights is recorded using encoding.

[0036] Step 2: Retrain the sparsified model and prevent the updates of weights that have been set to zero based on the position information of the encoded records.

[0037] Step 3: After obtaining the sparse model through the first two steps, extract the weights and record the indices of the non-zero weight positions.

[0038] Step 4: After completing the above steps, the accelerator performs calculations: First, the input feature map from outside the accelerator is transmitted to the input feature map buffer unit 1 via the AXI4 bus. Non-zero value location index data is transmitted to the non-zero value location index buffer unit 2 via the AXI4 bus. Weight data is transmitted to the weight read / write buffer unit 3 via the AXI4 bus. The external storage and control unit 8 configures the input feature map storage format and the calculation mode of the calculation unit 5, and inputs the input feature map and weights into the calculation unit 5 in the required form. The non-zero value location index is input to the non-zero value location decoding unit to obtain position information, which is then input into the calculation unit 5. The calculation results are accumulated in the accumulation unit 9, temporarily stored in the output buffer unit 7, and input in batches to the pooling and ReLU unit 6 for max pooling and ReLU operations. The output results are transmitted to the external memory of the accelerator via the AXI4 bus.

Claims

1. An accelerator designed using a sparsity reduction method for target detection networks, characterized in that: This sparsification method groups the kernels of the object detection convolutional neural network into 3×3 groups, sorts the absolute values ​​of the weights, resets the weights with absolute values ​​below the threshold to 0 as required, performs 4:9 and 2:9 sparsification in layers, and records and encodes the positions of non-zero values; it uses two sparsification modes, 4:9 and 2:9, to perform multiplication and accumulation operations on non-zero values. Specifically, the sparsification method involves the YOLOv2-tiny object detection network, which has nine convolutional layers. Layers 1, 2, and 3 (shallow convolutional layers) are sparsified in a 4:9 ratio, meaning the weights are sorted from largest to smallest absolute value, retaining the first four weights with the largest absolute values. Layers 3, 4, 5, 6, 7, and 8 (deep convolutional layers) are sparsified in a 2:9 ratio, meaning the weights are sorted from largest to smallest absolute value, retaining the first two weights with the largest absolute values. The 9th convolutional layer is 1×1 in size and is not sparsified. The accelerator includes an input feature map buffer unit (1), a non-zero value position index buffer unit (2), a weight buffer unit (3), a non-zero value position decoding unit (4), a calculation unit (5), a pooling and ReLU unit (6), an output buffer unit (7), an off-chip storage and control unit (8), and an accumulation unit (9). The output of the off-chip storage and control unit (8) is connected to the input feature map buffer unit (1), the non-zero value position index buffer unit (2), and the weight buffer unit (3), respectively. The outputs of the input feature map buffer unit (1) and the weight buffer unit (3) are connected to the calculation unit (5). The output of the non-zero value position index buffer unit (2) is connected to the non-zero value position decoding unit (4), and the obtained position information is output to the calculation unit (5). The output result of the calculation unit (5) is output to the output buffer unit (7) via the accumulation unit (9). The output of the output buffer unit (7) is connected to the pooling and ReLU unit (6), and the output of the pooling and ReLU unit (6) after processing is connected to the off-chip storage and control unit (8).

2. The accelerator designed according to the sparsity reduction method for target detection networks as described in claim 1, characterized in that: The aforementioned computing unit section (5) comprises multiple units, forming a computing array. Each computing unit section (5) includes a feature map input port (21), a position information input port (22), a weight input port (23), a mode control input port (24), a first convolution calculation output port (25), a second convolution calculation output port (26), a first input feature map shift register (27), a second input feature map shift register (28), a third input feature map shift register (29), an input feature map selection unit (210), and an input weight register unit. (211), mode selection control unit (212), multiplier, adder; wherein the output of feature map input port (21) is connected to the first input feature map shift register unit (27), the output of position information input port (22) is connected to the input feature map selection unit (210), the output of weight input port (23) is connected to the input weight register unit (211), the output of mode control input port (24) is connected to the mode selection control unit (212), and the output of mode selection control unit (212) is connected to the first convolution calculation output port (25) and the second convolution calculation output port (26), respectively.

3. The accelerator designed according to the sparsity reduction method for target detection networks as described in claim 2, characterized in that: The feature maps input by the feature map input port (21) are input in groups of three into the first input feature map shift register (27), and continuously shifted to the second input feature map shift register (28) and the third input feature map shift register (29). The input feature map is matched with the output of the input weight register (211) through the input feature map selection unit (210). When performing 4:9 mode calculation, the four weights come from the same convolution kernel. When performing 2:9 mode calculation, the four weights are grouped in pairs and come from two convolution kernels. After matching, a multiplication operation is performed. The signals after the multiplication operation are added in pairs and then added in pairs to obtain the first multiplication accumulation part result (213) and the second multiplication accumulation part result (214). The mode selection control unit (212) selects and outputs the selected output 4:9 mode convolution calculation result (215) and 2:9 mode convolution calculation result (216) through the first convolution calculation output port (25) and the second convolution calculation output port (26), respectively.

4. The accelerator designed according to the sparsity reduction method for target detection networks as described in claim 3, characterized in that: The steps to sparsify the YOLOv2-tiny object detection network and run it on an accelerator are as follows: Step 1: Pre-train the YOLOv2-tiny object detection network to obtain a pre-trained model. Using 3×3 convolutional kernels as units, compare the absolute values ​​of the weights and reset the weights below the threshold to 0. At the same time, use encoding to record the position information of the weights. Step 2: Retrain the sparsified model and prevent the updates of weights that have been set to zero based on the position information of the encoded records; Step 3: After obtaining the sparse model through the first two steps, extract the weights and record the indices of the non-zero weight positions; Step 4: After completing the above work, the accelerator is executed to perform calculations: First, the input feature map outside the accelerator is transmitted to the input feature map buffer unit (1) through the AXI4 bus, the non-zero value position index data is transmitted to the non-zero value position index buffer unit (2) through the AXI4 bus, and the weight data is transmitted to the weight read and write buffer unit (3) through the AXI4 bus; the off-chip storage and control unit (8) configures the storage form of the input feature map and the calculation mode of the calculation unit (5), and inputs the input feature map and weight into the calculation unit (5) in the form required for calculation, inputs the non-zero value position index into the non-zero value position decoding unit to obtain the position information, and inputs it into the calculation unit (5); the calculation result is accumulated in the accumulation unit (9), temporarily stored in the output buffer unit (7), and input into the pooling and ReLU unit (6) in batches for max pooling and ReLU operations, and the output result is transmitted to the memory outside the accelerator through the AXI4 bus.

Images