Target detection-oriented lightweight network hardware acceleration method and target detection method
By replacing the YOLOv4-tiny network with deep separable convolution module, BN layer sparse pruning and Roofline model optimization, algorithm optimization is accelerated, and the problems of real-time and power consumption in object detection are solved, and efficient object detection is achieved network.
Patent Information
- Application Number
- CN202411240617.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2025-01-10
AI Technical Summary
In actual object detection, poor real-time performance and excessive power consumption are problems.
Lightweight improvements to the YOLOv4-tiny network include replacing conventional convolution modules with depth separable convolution modules, BN layer sparse processing and pruning operations, combining the Roofline model selection acceleration algorithm to optimize convolution, depth separable convolution, and depth separable convolution , pooling and upsampling modules.
The real-time and energy efficiency of target detection have been significantly improved, the number of parameters has been reduced by 40%, the number of calculations has been reduced by 36%, the inference speed has been increased by about 40%, the power consumption has been greatly reduced, and the adaptation of embedded edge computing with limited resources has been adapted to the resource.
Smart Images
Figure CN119295722A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a lightweight network hardware acceleration method for target detection and a target detection method, belonging to the technical field of target detection. Background Art
[0002] With the rapid development of deep learning technology and network computing power, the number of model parameters and the amount of computation are increasing rapidly. However, due to the limitations of hardware resources and system space in a large number of application scenarios, models with large number of parameters and computation cannot be run. Therefore, lightweight design and hardware acceleration of models have become a hot topic in recent research. Object detection based on deep learning has broad application prospects in the fields of intelligent driving, remote sensing and telemetry, medical diagnosis, etc. In order to extract features from images and realize object detection tasks, convolutional structures are widely used in detection networks. However, convolutional layers often have a large number of parameters and computation. Although they can improve accuracy, they will affect the memory usage and forward reasoning performance of the entire neural network.
[0003] The deep separable convolution structure can reduce the number of parameters and computations of ordinary convolution. MobileNet uses a large number of deep separable convolution structures to solve the problem of difficult deployment of neural networks in embedded devices. The SqueezeNet network uses 11 convolution kernels instead of 33 convolution kernels, and delays the downsampling time in the network to obtain a larger feature map, which reduces the number of parameters while ensuring the accuracy of the network. The CUDA general computing architecture further improves the speed of convolutional neural network training and reasoning, but the cost is high and the energy consumption is large. ASIC, as a customized integrated circuit chip, is also an important hardware platform for accelerating convolutional neural networks. For example, Huawei Ascend 310 using the Da Vinci architecture, Cambrian ShiDianNao, Horizon Journey, Cambrian MLU series chips, the above chips are widely used in tasks such as image recognition and natural language processing. However, the use of ASIC, a hardware acceleration method, has a huge initial investment cost, poor flexibility and lack of versatility.
[0004] Field Programmable Gate Array (FPGA) has shown excellent characteristics in the field of target detection and in building neural network accelerators due to its flexible architecture and low power consumption. However, traditional FPGA solutions based on traditional convolution algorithms are often limited by the number of on-chip computing units.
[0005] The information disclosed in this background technology section is only intended to increase the understanding of the overall background of the utility model, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to ordinary technicians in this field. Summary of the invention
[0006] The technical problem to be solved by the present invention is: how to solve the problems of poor real-time performance and excessive power consumption in actual target detection.
[0007] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.
[0008] In a first aspect, the present invention provides a lightweight network hardware acceleration method for target detection, comprising:
[0009] The YOLOv4-tiny network is improved by using lightweight modules to obtain the target detection network;
[0010] The scaling coefficient of the BN layer in the target detection network is sparsely processed to obtain the BN layer sparse factor;
[0011] According to the sparse factor of the BN layer, the modules in the target detection network whose parameter quantity exceeds the threshold are determined, and pruning operations are performed to obtain the pruned target detection network;
[0012] Divide the pruned object detection network into a convolution accelerator module, a depth-separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module;
[0013] According to the Roofline model, the acceleration algorithm is selected, and the convolution accelerator module, depthwise separable convolution accelerator module, pooling accelerator module and upsampling accelerator module are optimized to obtain the optimized object detection network.
[0014] Furthermore, the method of improving the YOLOv4-tiny network using a lightweight module includes:
[0015] Use the partial convolution module Pconv to replace the conventional convolution module in the residual module of the backbone network Backbone;
[0016] Use the depth-wise separable convolution module DSConv to replace the conventional convolution modules in the neck and head detection;
[0017] Among them, the depth-wise separable convolution module DSConv includes DW convolution and PW convolution, the DW convolution is used to extract spatial features in each channel, and the PW convolution is used for cross-channel feature fusion and feature map expansion.
[0018] Furthermore, before the scaling coefficient of the BN layer in the target detection network is sparsely processed, the target detection network is iteratively trained multiple times based on the Adam optimizer according to the remote sensing image data set to obtain a trained target detection network.
[0019] Furthermore, the module whose parameter quantity exceeds the threshold includes a CBL module consisting of a convolutional layer, a batch normalization layer and a LeakyReLu activation layer.
[0020] Furthermore, the method of determining the modules in the target detection network whose parameter quantity exceeds the threshold value according to the BN layer sparse factor, performing pruning operations, and obtaining the pruned target detection network includes:
[0021] By setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function;
[0022] Combine the channel weight of the BN layer with the scaling factor of the BN layer, perform sparse regularization on the scaling factor of the BN layer, make the scaling factor of the BN layer gradually sparse, use the new training loss function to perform multiple iterative training, and obtain the scaling factor of the BN layer;
[0023] The scaling factor of the BN layer is used to form the sparse factor of the BN layer;
[0024] According to the sparse factor of the BN layer, the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are determined, and the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are pruned;
[0025] Set multiple pruning rates to perform pruning operations on the target detection network;
[0026] Evaluate the impact of different pruning rates on the performance of the target detection network and obtain the pruning rate evaluation results;
[0027] Fine-tuning is performed based on the pruning rate evaluation results to recover part of the accuracy lost due to pruning and obtain the pruned object detection network.
[0028] Furthermore, by setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function, wherein the new training loss function is expressed as:
[0029] ;
[0030] In the formula, L is the loss value of the new training loss function, x and y are the input and target of the training respectively; is the trainable weight of the convolutional layer; It is the prediction function of the forward propagation of the trained target detection network; is the scaling factor of the BN layer in the trained target detection network; is the penalty coefficient; It is the original training loss function of the trained target detection network.
[0031] Furthermore, the acceleration algorithm is selected according to the Roofline model, and the convolution accelerator module, the depthwise separable convolution accelerator module, the pooling accelerator module and the upsampling accelerator module are optimized, including:
[0032] Select the Winograd algorithm to optimize the convolution accelerator module;
[0033] Select the addition tree to optimize the depthwise separable convolution accelerator module;
[0034] Select pipeline to optimize the pooling accelerator module and the upsampling accelerator module.
[0035] Furthermore, the method of selecting the Winograd algorithm to optimize the convolution accelerator module includes:
[0036] Obtain an input feature map, divide each channel feature map of the input feature map into slices according to a predetermined size, and generate P slices for each channel of the input feature map;
[0037] According to the Winograd algorithm, the P slices generated by each channel of the input feature map are accumulated and calculated, the results of the accumulated calculation are inversely transformed, and the convolution accelerator module is optimized.
[0038] Furthermore, after the YOLOv4-tiny network is improved by using the lightweight module, the depthwise separable convolution module DSConv is accelerated by increasing the interface width and the number of interfaces, including:
[0039] The DW convolution hardware accelerator is accelerated by output parallelism;
[0040] The PW convolution hardware accelerator is accelerated by using input and output parallelism.
[0041] In a second aspect, the present invention provides a target detection method, which uses the method described in the first aspect for detection.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] The present invention proposes a lightweight network hardware acceleration method for target detection. By optimizing the YOLOv4-tiny network architecture and introducing the BN layer sparse pruning technology, the parameter quantity and computational complexity of the target detection network are effectively reduced. At the same time, the present invention also divides the lightweight target detection network into a convolution accelerator module, a depth-separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module, and selects an acceleration algorithm based on the Roofline model to accelerate the above modules, significantly improving the real-time performance and energy efficiency of target detection.
[0044] Compared with YOLOv7-tiny, the target detection network provided by the present invention not only improves the detection accuracy to 0.8mAP, but also reduces the number of parameters by 40%, the amount of calculation by 36%, the inference speed by about 40%, and the power consumption by 40%. It is perfectly adapted to resource-constrained embedded edge computing and promotes the widespread application of lightweight neural networks in the field of real-time target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 is a schematic diagram of the overall architecture of a target detection network provided by an embodiment of the present invention;
[0046] Figure 2 It is a schematic diagram of a process of improving a YOLOv4-tiny network by using a lightweight module provided by an embodiment of the present invention;
[0047] Figure 3 It is a schematic diagram of the internal structure of pruning a target detection network provided by an embodiment of the present invention;
[0048] Figure 4 Schematic diagram of the pruning process of the target detection network provided by an embodiment of the present invention;
[0049] Figure 5 Schematic diagram of the module composition of the target detection network provided by an embodiment of the present invention;
[0050] Figure 6 is a schematic diagram of a Roofline model provided by an embodiment of the present invention;
[0051] Figure 7 is a schematic diagram of GPU speed reasoning provided by an embodiment of the present invention;
[0052] Figure 8 It is a schematic diagram of CPU speed reasoning provided by an embodiment of the present invention;
[0053] Fig. 9 is a schematic diagram comparing parameter quantities of different pruning rate models provided by an embodiment of the present invention;
[0054] Fig.10 is a schematic diagram comparing the computational complexity of different pruning rate models provided by an embodiment of the present invention;
[0055] Fig.11 It is a comparative schematic diagram of a Roofline model for CPU speed inference of different pruning rate models provided by an embodiment of the present invention;
[0056] Fig.12 is a comparative schematic diagram of a Roofline model for GPU speed inference of different pruning rate models provided by an embodiment of the present invention;
[0057] Fig.13 It is a schematic diagram comparing the computing performance that can be achieved on ZYNQ by different pruning rate models provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0058] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other, wherein the original improved models marked in the drawings of the present specification are all YOLOv4-tiny networks.
[0059] Example 1
[0060] The present invention improves the target detection network from two dimensions of model lightweight architecture design and hardware acceleration based on a field programmable gate array (FPGA) to achieve hardware acceleration.
[0061] This embodiment introduces a lightweight network hardware acceleration method for target detection, including the following steps:
[0062] Step 1: If Figure 2 As shown in the figure, the YOLOv4-tiny network is improved by using a lightweight module to obtain a target detection network.
[0063] like Figure 1 As shown, in order to achieve lightweight architecture design and hardware acceleration of the target detection network model, the present invention uses a lightweight module to improve the YOLOv4-tiny network according to the depthwise separable convolution design principle and the partial convolution design principle, and then designs a lightweight target detection network architecture.
[0064] The method of improving the YOLOv4-tiny network by using a lightweight module includes:
[0065] Use the partial convolution module Pconv to replace the conventional convolution module in the residual module of the backbone network Backbone;
[0066] Use the depth-wise separable convolution module DSConv to replace the conventional convolution modules in the neck and head detection;
[0067] Among them, the depth-wise separable convolution module DSConv includes DW convolution and PW convolution, the DW convolution is used to extract spatial features in each channel, and the PW convolution is used for cross-channel feature fusion and feature map expansion.
[0068] Step 2: Before the scaling factor of the BN layer in the target detection network is sparsely processed, the target detection network is trained based on the Adam optimizer according to the remote sensing image dataset, including:
[0069] Set the initial learning rate of the Adam optimizer to 0.01;
[0070] Set 8 samples per batch for training and monitor the loss and accuracy during training;
[0071] Use the original training loss function of the target detection network to perform multiple iterative training to obtain a trained target detection network.
[0072] Among them, the remote sensing image dataset NWPU VHR-10 released by Northwestern Polytechnical University is used as the training set of the target detection network. The remote sensing dataset contains 10 categories of ground objects and a total of 3651 ground objects. The 10 categories of ground objects include airplanes, ships, storage tanks, baseball fields, basketball courts, tennis courts, ground tracks, ports, bridges, and vehicles.
[0073] In the training process of the target detection network, this embodiment jointly optimizes the weight of the BN layer and the scaling factor of the BN layer, and implements sparse regularization on the scaling factor of the BN layer. Pruning is performed on the weights of the convolutional layer and the BN layer channels with smaller scaling factors. Since the joint training method can automatically identify and remove unimportant channels, the generalization performance of the network will not be negatively affected during pruning. In this way, the network structure can be effectively optimized and the performance of the target detection network can be improved.
[0074] The computational load and parameter count of the target detection network are 5.18 GFLOPs and 3838970 respectively. The comparison between the computational load and parameter count of the YOLOv4-tiny network and the computational load and parameter count of the YOLOv7-tiny network is shown in Table 1.
[0075]
[0076] It can be seen from Table 1 that the computational amount and parameter amount of the target detection network are 62% and 60% of the computational amount and parameter amount of the YOLOv4-tiny network, respectively, and the computational amount and parameter amount of the target detection network are 64% and 61% of the computational amount and parameter amount of the YOLOv7-tiny network, respectively, indicating that the present invention can achieve lightweight while also ensuring the accuracy of the target detection network.
[0077] Step 3: Sparsely process the scaling coefficient of the BN layer in the target detection network to obtain the BN layer sparse factor.
[0078] By setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function;
[0079] Combine the channel weight of the BN layer with the scaling factor of the BN layer, perform sparse regularization on the scaling factor of the BN layer, make the scaling factor of the BN layer gradually sparse, use the new training loss function to perform multiple iterative training, and obtain the scaling factor of the BN layer;
[0080] The scaling factor of the BN layer is used to form the sparse factor of the BN layer;
[0081] By setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function, wherein the new training loss function is expressed as:
[0082] (1);
[0083] In the formula, L is the loss value of the new training loss function, x and y are the input and target of the training respectively; is the trainable weight of the convolutional layer; It is the prediction function of the forward propagation of the trained target detection network; is the scaling factor of the BN layer in the trained target detection network; is the penalty coefficient; It is the original training loss function of the trained target detection network.
[0084] Step 4: According to the sparse factor of the BN layer, determine the modules in the target detection network whose parameters exceed the threshold, perform pruning operations, and obtain the pruned target detection network.
[0085] The module whose parameter quantity exceeds the threshold value includes a CBL module composed of a convolution layer, a batch normalization layer and a LeakyReLu activation layer. The CBL module is represented as:
[0086] (2);
[0087] In the formula, represents the output of the CBL module; LeakyReLu represents the activation function; represents the scaling factor of the learnable parameters of the BN layer; Conv() represents the convolution operation; Represents the input of the CBL module; Represents the weight parameter of the convolution kernel; Represents the mean of the convolution operation results; Represents the variance of the convolution operation result; Represents a number greater than 0 and avoids the denominator being 0; Represents the translation coefficient of the learnable parameters of the BN layer.
[0088] From formula (2), we can see that when the scaling factor of the BN layer As it gradually approaches 0, the output of the CBL module Will gradually approach the translation coefficient of the BN layer . This means that the input of the CBL module In this case the output of the CBL module There is almost no effect. At this time, it is considered that The corresponding BN layer channels are unimportant. Therefore, the convolutional layer channels corresponding to these unimportant BN layer channels will not have a significant impact on the output, and channel pruning can be performed.
[0089] According to the sparse factor of the BN layer, the modules in the target detection network whose parameters exceed the threshold are determined and pruning operations are performed, including:
[0090] Combining the channel weight of the BN layer with the scaling coefficient of the BN layer, performing sparse regularization on the scaling coefficient of the BN layer, making the scaling coefficient of the BN layer gradually sparse, and obtaining the scaling coefficient of the BN layer;
[0091] The scaling factor of the BN layer is used to form the sparse factor of the BN layer;
[0092] According to the sparse factor of the BN layer, the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are determined, and the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are pruned;
[0093] Set multiple pruning rates to perform pruning operations on the target detection network;
[0094] Evaluate the impact of different pruning rates on the performance of the target detection network and obtain the pruning rate evaluation results;
[0095] Fine-tuning is performed based on the pruning rate evaluation results to recover part of the accuracy lost due to pruning and obtain the pruned object detection network.
[0096] The above process is as follows Figure 3 , Figure 4 As shown, this embodiment uses a sparse penalty coefficient α=0.002 for sparse training. The pruning rates are set to 19%, 39%, and 50% respectively to prune and fine-tune the target detection network, and the final detection accuracies are 0.82, 0.83, and 0.83 respectively. Compared with the YOLOv4-tiny network, pruning reduces the number of parameters and the amount of calculation, and also obtains a higher detection accuracy than YOLOv4-tiny through network fine-tuning. At the same time, the gap between the detection accuracy and the YOLOv4-tiny network is further narrowed, and the pruned target detection network is finally obtained.
[0097] Step 5: Fig. 9 , Fig.10 As shown in the figure, the three models of the pruned target detection network with high, medium and low pruning ratios are compared in terms of parameter quantity and computational complexity combined with the Roofline model theory, the impact of different pruning rates on the performance of the target detection network is evaluated, and the optimal target detection network is selected and deployed on the embedded platform ZYNQ 7020.
[0098] The pruning rate of the target detection network is set to 19% as a low-ratio pruning model, the pruning rate of the target detection network is set to 39% as a medium-ratio pruning model, and the pruning rate of the target detection network is set to 59% as a high-ratio pruning model.
[0099] The low, medium, and high ratio pruning models are deployed on different hardware for GPU speed testing, and the GPU inference speed and CPU inference speed of the low, medium, and high ratio pruning models are analyzed.
[0100] Among them, the GPU speed test was completed on the NVIDIA GPU GeForce 2080, and the achievable computing performance was 9.83TFLOP / S and the bandwidth was 448.0GB / S.
[0101] CPU inference speed is completed by Intel(R) Xeon(R) CPU E5-2678 v3, with a computing performance of 480GFLOP / S and a bandwidth of 68.2GB / S.
[0102] Depend on Figure 7 , Figure 8 As shown in the figure, during the GPU speed test, the model with smaller computational complexity and parameter amount is slower on the GPU. The reason is that the GPU has strong computing performance, and the test model has relatively low computational memory access. In the GPU's Roofline model, it falls into the bandwidth bottleneck area and cannot give full play to the powerful computing power of the GPU.
[0103] In the CPU test speed, models with lower computational complexity and parameter count have faster inference speeds because all models fall into the computing bottleneck area of the CPU Roofline model.
[0104] Therefore, according to the relevant theory of the Roofline model, in the computing bottleneck area, the lower the model's computational load, the faster the inference speed. Therefore, it is necessary to select a suitable network model based on the specific hardware deployment platform to give full play to the hardware's performance and achieve network acceleration.
[0105] In summary, although the inference speed of the high-ratio pruning model on the GPU is poor, since the YOLOv4-tiny network and the target detection network are deployed on the embedded platform ZYNQ 7020 with limited hardware resources, and the computational access-to-memory ratio of the YOLOv4-tiny network and the target detection network is much greater than 2.75, the YOLOv4-tiny network and the target detection network both fall into the computing bottleneck area of the ZYNQ 7020 Roofline model.
[0106] like Fig.11 , Fig.12 and Fig.13 As shown in the figure, the high-ratio pruning model with the lowest computational effort can theoretically achieve the fastest inference speed. Compared with other models, the high-ratio pruning model has higher accuracy and lower parameters, which can effectively save the storage space and hardware on-chip resources of embedded devices, achieving the design requirements of high accuracy, fast inference speed and low memory usage. In summary, the high-ratio pruning model is deployed as an inference network on the embedded platform ZYNQ 7020.
[0107] Step 6: Figure 5 As shown in Figure 1, the target detection network with a pruning rate of 59% is divided into a convolution accelerator module, a depth-separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module.
[0108] Step 7: Select the acceleration algorithm according to the Roofline model, optimize the convolution accelerator module, depthwise separable convolution accelerator module, pooling accelerator module and upsampling accelerator module to obtain the optimized object detection network.
[0109] This embodiment first establishes Figure 6 The Roofline model shown in FIG. 1 selects a suitable network acceleration algorithm for hardware acceleration and deploys it on hardware, wherein the Roofline model is expressed by the formula:
[0110] (3);
[0111] In the formula, AP represents the maximum performance that the target detection network can obtain under specific hardware conditions, and the unit is GFLOP / S. CTC Ratio represents the computational access-to-memory ratio of the target detection network, which is used to express the average amount of computation that can be actually supported per off-chip access, that is, the data reuse of the target detection network. CRoof represents the maximum computational performance of the target detection network. BW represents the bandwidth of the target detection network, which is used to express the maximum number of memory exchanges that the hardware of the target detection network can perform per second.
[0112] It can be seen from formula (3) that the maximum performance that the hardware of the target detection network can exert does not exceed the performance upper limit of the target detection network hardware itself and the minimum value of the product of the computing memory access ratio and the bandwidth.
[0113] The computational access ratio CTC Ratio of the target detection network is expressed as:
[0114] (4);
[0115] Wherein, Computational Cost represents the computational amount of the target detection network, which is used to represent the number of floating-point operations required for one reasoning of the target detection network, and measures the time complexity of the target detection network; DRAM byte Access represents the memory access amount of the target detection network, which is used to represent the total amount of memory exchange during one reasoning, and measures the space complexity of the target detection network.
[0116] Among them, the maximum computational performance CRoof of the target detection network is expressed as:
[0117] (5);
[0118] In the formula, Indicates the specific number of digital signal processors DSP in the hardware platform of the target detection network; Indicates the number of digital signal processors DSP required for each multiplication and addition calculation; Indicates the clock frequency of the target detection network hardware.
[0119] Since each high-performance bandwidth interface HP in the hardware platform communication interface of the target detection network can complete the transmission of 64 bits of data under a unit clock, and one byte has 8 bits of data, the bandwidth of the target detection network is expressed as:
[0120] (6);
[0121] In the formula, Indicates the number of HP communication interfaces owned by the hardware platform of the target detection network.
[0122] According to the hardware resource parameters of the ZYNQ 7020 platform, the hardware platform of the target detection network has 220 digital signal processors DSP and 4 high-performance bandwidth interfaces HP. It can be seen that when the clock frequency of the target detection network is 100MHz, the computing power of the ZYNQ 7020 platform is 8.8GFLOPS and the bandwidth is 3.2GB / s.
[0123] The method for selecting an acceleration algorithm according to a Roofline model and optimizing a convolution accelerator module, a depthwise separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module includes:
[0124] Select the Winograd algorithm to optimize the convolution accelerator module;
[0125] Select the addition tree to optimize the depthwise separable convolution accelerator module;
[0126] Select pipeline to optimize the pooling accelerator module and the upsampling accelerator module.
[0127] The Winograd algorithm can improve the calculation speed of the target detection network by reducing the number of multiplications. The core idea of the Winograd algorithm is to convert the convolution operation into a series of small-scale matrix multiplication operations to reduce the amount of calculation.
[0128] The process of implementing convolution using Winograd is mainly divided into four steps: matrix change of input feature map, matrix change of BN layer weights of target detection network, matrix multiplication, and output result conversion.
[0129] In order to further accelerate the algorithm's reasoning process, this embodiment, based on the Winograd algorithm, proposes an optimization strategy for the characteristic that the weight parameters in the target detection network remain unchanged during the forward reasoning process. The core of the optimization strategy is to convert the weights in the target detection network in advance before formally performing forward reasoning, which can significantly reduce the time overhead required for weight conversion in the forward reasoning stage, thereby improving the overall reasoning efficiency.
[0130] Due to the relatively limited storage hardware resources of the field programmable gate array FPGA, it is impossible to store all the weight data and feature map data of the convolutional layer in the on-chip cache at one time.
[0131] Therefore, before actually performing the convolution operation, this embodiment needs to slice the feature map, that is, only process the data of some channels and some areas of the feature map each time. The slicing processing method ensures that the FPGA can efficiently perform convolution operations even under resource-constrained conditions, further promoting the improvement of algorithm reasoning speed.
[0132] Among them, the method of selecting the Winograd algorithm to optimize the convolution accelerator module includes:
[0133] Get a channel number Width Gao Wei The input feature map is selected each time In the channel The feature map slices of size are tiled and selected at the same time convolution kernel The convolution kernel of channels is obtained by Winograd convolution. Width Gao Wei The output feature map block is repeated until all channels of the entire input feature map are output to complete the convolution operation.
[0134] Among them, each channel of the input feature map produces P slices, , H , W are the length and width of the input feature map respectively, , m represents the output length of the convolution result, and r represents the convolution kernel size.
[0135] According to the Winograd algorithm, the P slices generated by each channel of the input feature map are accumulated and calculated, the results of the accumulated calculation are inversely transformed, and the convolution accelerator module is optimized.
[0136] Step 8: After the YOLOv4-tiny network is improved by using the lightweight module, the depthwise separable convolution module DSConv is accelerated by increasing the interface width and the number of interfaces, including:
[0137] The DW convolution hardware accelerator is accelerated by output parallelism;
[0138] The PW convolution hardware accelerator is accelerated by using input and output parallelism.
[0139] A deep separable convolution structure is used in the target detection network to realize the lightweight architecture of the network. The deep separable convolution module DSConv includes DW convolution and PW convolution. The DW convolution is used to extract spatial features in each channel, and the PW convolution is used for cross-channel feature fusion and feature map expansion.
[0140] In the actual operation of this embodiment, the hardware accelerator of DW convolution and the hardware accelerator of PW convolution are respectively written to achieve acceleration, and the variable mode is set to distinguish the hardware accelerator of DW convolution from the hardware accelerator of PW convolution, so as to control the flow of data.
[0141] Example 2
[0142] Based on the same inventive concept as Example 1, this example introduces a target detection method, which uses the method described in Example 1 to perform detection.
[0143] For the specific function implementation, please refer to the relevant contents in the method of Example 1 and will not be elaborated here.
[0144] In summary, the present invention proposes a lightweight network hardware acceleration method for target detection, which effectively reduces the number of parameters and computational complexity of the target detection network by optimizing the YOLOv4-tiny network architecture and introducing the BN layer sparse pruning technology. At the same time, the present invention also divides the lightweight target detection network into a convolution accelerator module, a depth-separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module, and selects an acceleration algorithm based on the Roofline model to accelerate the above modules, significantly improving the real-time performance and energy efficiency of target detection.
[0145] Compared with YOLOv7-tiny, the target detection network provided by the present invention not only improves the detection accuracy to 0.8mAP, but also reduces the number of parameters by 40%, the amount of calculation by 36%, the inference speed by about 40%, and the power consumption by 40%. It is perfectly adapted to resource-constrained embedded edge computing and promotes the widespread application of lightweight neural networks in the field of real-time target detection.
[0146] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0147] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0150] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.
Claims
1. A lightweight network hardware acceleration method for target detection, characterized in that: include: The YOLOv4-tiny network is improved by using lightweight modules to obtain the target detection network; The scaling coefficient of the BN layer in the target detection network is sparsely processed to obtain the BN layer sparse factor; According to the sparse factor of the BN layer, the modules in the target detection network whose parameter quantity exceeds the threshold are determined, and pruning operations are performed to obtain the pruned target detection network; Divide the pruned object detection network into a convolution accelerator module, a depth-separable convolution accelerator module, a pooling accelerator module, and an upsampling accelerator module; According to the Roofline model, the acceleration algorithm is selected, and the convolution accelerator module, depthwise separable convolution accelerator module, pooling accelerator module and upsampling accelerator module are optimized to obtain the optimized object detection network.
2. The lightweight network hardware acceleration method for target detection according to claim 1 is characterized in that: The method for improving the YOLOv4-tiny network using a lightweight module includes: Use the partial convolution module Pconv to replace the conventional convolution module in the residual module of the backbone network Backbone; Use the depth-wise separable convolution module DSConv to replace the conventional convolution modules in the neck and head detection; Among them, the depth-wise separable convolution module DSConv includes DW convolution and PW convolution, the DW convolution is used to extract spatial features in each channel, and the PW convolution is used for cross-channel feature fusion and feature map expansion.
3. The lightweight network hardware acceleration method for target detection according to claim 1 or 2, characterized in that: Before the scaling coefficient of the BN layer in the target detection network is sparsely processed, the target detection network is iteratively trained multiple times based on the Adam optimizer according to the remote sensing image data set to obtain a trained target detection network.
4. The lightweight network hardware acceleration method for target detection according to claim 1 is characterized in that: The module whose parameter quantity exceeds the threshold includes a CBL module consisting of a convolutional layer, a batch normalization layer and a LeakyReLu activation layer.
5. The lightweight network hardware acceleration method for target detection according to claim 1 is characterized in that: The method of determining the modules in the target detection network whose parameter quantity exceeds the threshold value according to the BN layer sparse factor, performing pruning operation, and obtaining the pruned target detection network includes: By setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function; Combine the channel weight of the BN layer with the scaling factor of the BN layer, perform sparse regularization on the scaling factor of the BN layer, make the scaling factor of the BN layer gradually sparse, use the new training loss function to perform multiple iterative training, and obtain the scaling factor of the BN layer; The scaling factor of the BN layer is used to form the sparse factor of the BN layer; According to the sparse factor of the BN layer, the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are determined, and the channel weights of the convolutional layer and the BN layer whose scaling factor is less than the threshold are pruned; Set multiple pruning rates to perform pruning operations on the target detection network; Evaluate the impact of different pruning rates on the performance of the target detection network and obtain the pruning rate evaluation results; Fine-tuning is performed based on the pruning rate evaluation results to recover part of the accuracy lost due to pruning and obtain the pruned object detection network.
6. The lightweight network hardware acceleration method for target detection according to claim 5 is characterized in that: By setting the sparse penalty coefficient, the L1 regularization term is added to the original training loss function of the target detection network to construct a new training loss function, wherein the new training loss function is expressed as: ； In the formula, L is the loss value of the new training loss function, x and y are the input and target of the training respectively; is the trainable weight of the convolutional layer; It is the prediction function of the forward propagation of the trained target detection network; is the scaling factor of the BN layer in the trained target detection network; is the penalty coefficient; It is the original training loss function of the trained target detection network.
7. The lightweight network hardware acceleration method for target detection according to claim 1, characterized in that: Select the acceleration algorithm according to the Roofline model, and optimize the convolution accelerator module, depthwise separable convolution accelerator module, pooling accelerator module, and upsampling accelerator module, including: Select the Winograd algorithm to optimize the convolution accelerator module; Select the addition tree to optimize the depthwise separable convolution accelerator module; Select pipeline to optimize the pooling accelerator module and the upsampling accelerator module.
8. The lightweight network hardware acceleration method for target detection according to claim 7 is characterized in that: Methods for selecting the Winograd algorithm to optimize the convolution accelerator module include: Obtain an input feature map, divide each channel feature map of the input feature map into slices according to a predetermined size, and generate P slices for each channel of the input feature map; According to the Winograd algorithm, the P slices generated by each channel of the input feature map are accumulated and calculated, the results of the accumulated calculation are inversely transformed, and the convolution accelerator module is optimized.
9. The lightweight network hardware acceleration method for target detection according to claim 2, characterized in that: After the YOLOv4-tiny network is improved by using a lightweight module, the depthwise separable convolution module DSConv is accelerated by increasing the interface width and the number of interfaces, including: The DW convolution hardware accelerator is accelerated by output parallelism; The PW convolution hardware accelerator is accelerated by using input and output parallelism.
10. A target detection method, characterized in that: The method according to claim 1 is used for detection.
Citation Information
Cited By
Low-coupling Transform target detection method and system based on FPGA acceleration
CN120911516A