A method for accelerating target detection based on structural reparameterization

Abstract

The present invention discloses a target detection acceleration method based on structural re-parameterization. For a convolutional neural network target detection model, a re-parameterizable detection head structure is constructed. After training, the constructed multi-branch decoupled detection head uses channel segmentation to achieve feature alignment, that is, the channel segmentation layer divides the input feature into two parts, corresponding to classification and regression tasks respectively. Experiments show that the improved network training convergence speed and accuracy are improved; the constructed multi-branch decoupled detection head structure can be equivalently converted into a single-channel structure during actual deployment and inference, and the inference speed is improved in terms of parameter quantity, calculation amount and calculation density; finally, the target detection head realizes group convolution fusion by filling the convolution kernel, and there is only one 3×3 convolution layer after fusion, and the calculation density and parallelism have significant advantages over the existing design.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and specifically relates to a target detection acceleration method based on structural reparameterization. Background Art

[0002] Currently, convolutional neural network algorithms are an important research direction in the field of computer vision. Object detection models based on convolutional neural networks have achieved excellent performance that far exceeds traditional methods. However, the improvement in the performance of object detection models has also greatly increased the amount of computing and storage required for the algorithm, which has placed higher requirements on end-side embedded devices in terms of low power consumption and real-time processing. In the application scenarios of emerging businesses such as autonomous driving and unmanned retail, the method of uploading data to the cloud to obtain results and then transmitting them back is too time-consuming and has poor real-time performance. In order to meet the real-time requirements, the method of lightweighting the network and deploying it in embedded end-side devices has significant advantages. With the continuous enhancement of the computing power of embedded devices and the development of model lightweighting technology, deep learning object detection models have begun to be used in embedded end-side devices, but their running speed after deployment is still difficult to meet actual needs, and there is much room for improvement. The embedded convolutional neural network acceleration method based on structural reparameterization proposed in this invention is studied in this context.

[0003] In actual scenarios, the computing power of embedded devices is relatively poor. Generally, the model is trained on a server with sufficient computing resources first, and then deployed on the embedded device. The computing resources consumed during server-side training have no effect on the overhead and performance of the embedded device during inference. Based on structural reparameterization, the present invention uses more computing resources during training to construct a branched convolutional neural network structure with high precision but slow inference speed. In actual deployment applications, the network branches are merged and equivalently converted into a network structure with the same precision and fast inference speed, thereby achieving acceleration of the convolutional neural network on embedded devices.

[0004] Other existing deep learning model lightweighting methods, such as model pruning, model quantization, and direct design of lightweight networks, have problems such as deployment difficulties and weak actual acceleration effects in practical applications. For example, the NVIDIA Jetson TX2 embedded GPU platform used in this invention does not support INT8 computing, so it is impossible to use INT8 quantization to accelerate the model. Although some lightweight convolutional neural networks have a small amount of computation and parameters, their memory access costs are increased, and the acceleration effect on embedded GPU devices is weak. The acceleration method proposed in this invention can be applied to various end-side embedded devices. At the same time, it can be used in combination with other existing deep learning model acceleration methods when they are feasible to achieve further acceleration, which has significant advantages. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a target detection acceleration method based on structure reparameterization, which can improve detection accuracy and speed.

[0006] A target detection acceleration method based on structure reparameterization, comprising:

[0007] After extracting feature data including the target location and target category from the image, it is input into the detection head to train the detection head;

[0008] The detection head includes a feature scale conversion unit, a channel switching layer and two detection channels;

[0009] The feature scale transformation unit includes a first 1×1 convolution layer, a first batch of normalization layers and an activation function layer arranged in sequence;

[0010] The channel switching layer divides the feature data output by the feature scale transformation unit into two parts, and the feature sizes of the two parts become Among them, H×W×C is the size of the feature data output by the feature scale transformation unit, H and W are the height and width of the feature image respectively, and C is the number of feature map channels;

[0011] The two detection channels correspond to the target position data and the target category data in the received feature data, respectively; each detection channel includes a second 1×1 convolutional layer, a second batch normalization layer, a first 3×3 convolutional layer, a third batch normalization layer, a third 1×1 convolutional layer, and a fourth batch normalization layer, which are arranged in sequence;

[0012] Use the trained detection head for object detection.

[0013] Furthermore, before using the trained detection head for target detection, it is first subjected to equivalent transformation, specifically:

[0014] First, the first 1×1 convolutional layer and the first batch of normalization layers in the feature scale transformation unit are converted into a fourth 1×1 convolutional layer;

[0015] Then, a first conversion is performed on both detection channels:

[0016] Convert the second 1×1 convolutional layer and the second batch normalization layer into a fifth 1×1 convolutional layer;

[0017] Convert the first 3×3 convolutional layer and the third batch normalization layer into a second 3×3 convolutional layer;

[0018] Convert the third 1×1 convolutional layer and the fourth batch normalization layer to the sixth 1×1 convolutional layer;

[0019] Finally, the two detection channels are converted for the second time, that is, the two detection channels after the first conversion are converted into a third 3×3 convolution layer respectively;

[0020] Utilize the equivalently transformed detection head for target detection.

[0021] Furthermore, after the second conversion is performed on the two detection channels, a third conversion is performed on them, and then the detection head is used to perform target detection; wherein the third conversion is:

[0022] Convert the two third 3×3 convolutional layers into one fourth 3×3 convolutional layer.

[0023] Preferably, the specific method of the third conversion is: after splicing the two third 3×3 convolutional layers with a convolutional layer of the same scale and weight 0 as their respective ones, the two obtained convolutional layers are spliced ​​again to obtain a fourth 3×3 convolutional layer.

[0024] The present invention has the following beneficial effects:

[0025] The present invention constructs a reparameterizable detection head structure for the convolutional neural network target detection model. After training, the constructed multi-branch decoupling detection head uses channel splitting to achieve feature alignment. That is, the channel splitting layer splits the input features into two parts, corresponding to the classification and regression tasks respectively. Experiments have shown that the improvement has improved the convergence speed and accuracy of network training.

[0026] The constructed multi-branch decoupled detection head structure can be equivalently converted into a single-path structure during actual deployment and inference, which improves the inference speed in terms of parameter quantity, computational complexity, and computational density.

[0027] Finally, the target detection head implements group convolution fusion by padding the convolution kernel. After fusion, there is only one 3×3 convolution layer, which has significant advantages in computational density and parallelism compared with existing designs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The detection head structure based on structural reparameterization of the present invention;

[0029] Figure 2 The detection head structure after batch normalization fusion of the present invention;

[0030] Figure 3 The detection head structure after linear over-parameterization fusion of the present invention;

[0031] Figure 4 Schematic diagram of the detection head actually deployed after conversion of the present invention;

[0032] Figure 5 Schematic diagram of the fusion of grouped convolution of the present invention;

[0033] Figure 6 This is a flow chart of the deployment of the terminal-side embedded device of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0035] This paper provides a convolutional neural network acceleration method based on structural reparameterization. By combining the advantages of existing advanced technologies and considering the impact of multiple factors, such as model computational complexity, parameter count, and computational density, on actual inference speed, a new detection head is proposed. By decoupling the training and inference architectures, a multi-branch detection head is trained during training and converted to a single-branch detection head during deployment. Experimental results on the PASCAL VOC dataset show that, using the proposed method on the NVIDIA Jetson TX2 embedded GPU platform to accelerate the currently advanced lightweight object detection model YOLOX, a 29.1% improvement in inference speed can be achieved with a mAP loss of less than 1%.

[0036] The present invention specifically comprises the following steps:

[0037] one, Figure 1 This paper describes a detection head structure in a convolutional neural network for target detection. Generally speaking, the image to be detected is input into the target detection network. After the feature extraction and feature fusion parts of the target detection network are passed through, the features extracted from the image are obtained. The detection results are obtained only after calculations by the detection head. This paper mainly improves the detection head part. Figure 1 The convolutional neural network shown in the figure is used as the detection head of the target detection network. Assuming that there is a vehicle target in the input image, the feature extraction and feature fusion parts calculate the features that represent the position and category of the vehicle in the image, and use them as the input of the detection head. The specific calculation process of the detection head constructed by the present invention is that the input feature Feature is calculated through the 1×1 convolution layer, batch normalization layer, and activation function layer, and its size becomes H×W×C, where H and W are the height and width of the feature map, and C is the number of channels in the feature map. The calculated features pass through the channel segmentation layer, and the channel segmentation layer divides the features into two parts from the center position of the channel dimension of the feature map, and the feature sizes of the two parts become The features corresponding to predicting vehicle category and vehicle bounding box are halved in number. Although the number of channels is reduced, the features extracted by the convolutional neural network are redundant, so the required features can still be represented after channel segmentation. After the channel segmentation layer, the two sets of features are used to predict vehicle category and vehicle bounding box, respectively, achieving feature alignment for classification and regression bounding box tasks. The aligned features are passed through a 1×1 convolutional layer, a batch normalization layer, a 3×3 convolutional layer, a batch normalization layer, and a 1×1 convolutional layer, a batch normalization layer, to obtain the corresponding classification and regression prediction results. The vehicle is detected from the image and its location is annotated with a bounding box. This convolutional layer design achieves complete decoupling of the classification and regression tasks from feature usage to gradient backpropagation, achieving higher accuracy than other designs.

[0038] 2. Replace the detection head of the YOLOX target detection model with the detection head based on structural reparameterization proposed in this invention, then train it based on the target detection training dataset and save the trained weights.

[0039] Third, we perform equivalent transformations on the trained detection head based on structural reparameterization. For a unit consisting of a convolutional layer and a batch normalization layer in series, the convolutional layer and the batch normalization layer can be regarded as linear operations during the object detection model inference. The batch normalization layer calculation can then be integrated into the convolutional layer calculation. This fusion not only eliminates the batch normalization layer calculation operation but also reduces memory access operations, thereby improving the inference speed without affecting the accuracy. The specific method is as follows:

[0040] Based on the structural reparameterization, the trained detection head is converted equivalently. For a convolution layer, it can be regarded as a linear operation, and its calculation can be expressed as formula (1):

[0041] Conv(x)=W(x)+b (1)

[0042] Where Conv(x) represents the result of convolution of input x, W(x) represents the convolution calculation of x and the convolution kernel, and b is the deviation in the convolution calculation. The calculation formula of the batch normalization layer is shown in formula (2):

[0043]

[0044] Where BN(x) represents the result of batch normalization of input x, γ and β are the expansion parameters and translation parameters learned during training, mean and var are the mean and variance of the training batch data respectively. Substituting the convolutional layer result into the batch normalization layer formula, we can get formula (3):

[0045]

[0046] After further simplification, we can get (4):

[0047]

[0048] As shown in formula (4), the output of the batch normalization layer can be simplified into the same format as the convolution layer. According to the formula, the convolution kernel is linearly transformed, and the calculation of the batch normalization layer can be integrated into the convolution layer calculation. The fused detection head is completely equivalent to the detection head before fusion in a mathematical sense, and the calculation of the batch normalization layer is no longer required, thereby speeding up the deployment inference speed.

[0049] 4. The detection head structure after batch normalization fusion is as follows Figure 2 As shown in the figure, for the two 1×1 convolutional layers corresponding to the classification and regression tasks respectively; the 3×3 convolutional layer; the 1×1 convolutional layer connected in series can be regarded as three linear operations. Using the linear over-parameterization technology (ExpandNets: Linear Over-parameterization to Train CompactConvolutional Networks), the convolutional layer weights regarded as three linear operations are represented by a sparse matrix. After transposition and multiplication, they can be equivalent to a sparse matrix representation of the 3×3 convolutional layer weights. Then, the original series branch can be equivalent to a 3×3 convolutional layer. The detection head structure after linear over-parameterization fusion is as follows: Figure 3 The specific method is as follows:

[0050] The linear over-parameterization technique is used on the two convolution branches after channel segmentation of the detection head after batch normalization fusion. The three convolution layers corresponding to each branch can be expressed as The corresponding input channels are 0.5C, C, C, and the output channels are C, C, cls, or 5. Cls represents the number of detection categories, 5 represents the 5 parameters required to describe the bounding box, and the convolution kernel sizes are 1×1, 3×3, and 1×1. For input with width and height of W and H, and channel of 0.5C, The result of the convolution operation can be expressed as formula (5):

[0051]

[0052] Where * and × represent convolution operations and matrix multiplication respectively. is the structured sparse matrix of the convolution kernel, is the matrix representation of the input, Y C×W×H is the convolution output result, and reshape means the dimension conversion of the result. From formula (5), we can see that the convolution operation can be equivalent to the product of the structured sparse matrix and the input matrix, and then for Three convolutional layers, multiplying their convolution kernel structure sparse matrices:

[0053]

[0054] You can use F 1 ,F 2 ,F 3 The three convolutional layers are equivalent to one convolutional layer F, and the equivalent diagram is shown in formula (7).

[0055]

[0056] Among them, * represents the convolution operation, X and Y represent the convolution input and convolution output respectively.

[0057] 5. If Figure 3 As shown in the figure, the two parts of features after channel segmentation are respectively passed through a 3×3 convolution layer and then the prediction results are output. These two 3×3 convolution layers can be regarded as a grouped convolution layer with a group number of 2. For the grouped convolution layer, the convolution kernels of the same size with a weight of 0 can be connected in parallel so that the number of convolution kernels corresponding to each output feature map is equal to the number of channels of the input feature map, thereby converting the original grouped convolution layer into a 3×3 convolution layer. Figure 4 As shown in the figure, after the above steps, the trained detection head based on structural reparameterization can be converted into a structure with a 1×1 convolution layer, an activation function layer, and a 3×3 convolution layer in series during actual deployment and inference. This improves the number of parameters, computational complexity, computational density, and storage requirements, thereby speeding up the inference speed of actual deployment. The specific method is as follows:

[0058] For the convolutional layer Input feature X C×W×H The two features after channel segmentation are The result after corresponding convolution and channel splicing can be expressed by formula (8):

[0059]

[0060] Among them, CONCAT1 means that in the Y 1 cls×W×H ,Y 2 5×W×H Perform channel concatenation in the first dimension.

[0061] Group convolution is equivalent to grouping the regular convolution, which is equivalent to performing two-group convolution on the regular convolution. The input feature map and convolution kernel are divided into two groups and the regular convolution is performed on each group to obtain the corresponding output feature map Y. 1 cls×W×H ,Y 2 5×W×H , and then concatenate to get the output Y of the grouped convolution (5+cls)×W×H .

[0062] like Figure 5As shown in Figure 1, the convolution kernels of these two 3×3 convolution layers are connected in parallel with the convolution kernels of the same scale with a weight of 0, so that the number of convolution kernels corresponding to each output feature map is equal to the number of channels of the input feature map. Then, the original grouped convolution layer can be converted into a 3×3 convolution layer, as shown in Formula (9). The result after splicing is exactly the same as before splicing.

[0063]

[0064]

[0065]

[0066] in, For Convolution kernels of the same scale and weight 0; For Convolution kernels of the same scale and weight 0 Express The splicing operation is performed in the second dimension. After the above steps, the trained detection head based on structural reparameterization can be converted into a structure with a 1×1 convolution layer, an activation function layer, and a 3×3 convolution layer in series during actual deployment and inference. This improves the number of parameters, computational complexity, computational density, and storage requirements, thereby accelerating the inference speed of actual deployment.

[0067] 6. Generate the target detection model weights converted in the above steps into ONNX (Open Neural Network Exchange) format, deploy them on the NVIDIA Jetson TX2 end-side embedded device described in the present invention in FP16 data format through the deep learning inference framework TensorRT, read the data through the onboard camera, build a GStreamer pipeline to stream the video into the target detection model deployed on the TensorRT framework, and use the OpenCV open source image processing library to visualize the results. The deployment flow chart is as follows: Figure 6 shown.

[0068] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target detection acceleration method based on structural reparameterization, characterized in that: include: After the feature data including the target position and target category are extracted from the image, they are input into the detection head to train the detection head; The detection head includes a feature scale transformation unit, a channel switching layer and two detection channels; The feature scale transformation unit includes a first 1x1 convolution layer, a first batch of normalization layers and an activation function layer arranged in sequence; The channel switching layer divides the feature data output by the feature scale transformation unit into two parts, and the feature sizes of the two parts become Among them, H×W×C is the size of the feature data output by the feature scale transformation unit, H and W are the height and width of the feature image, respectively, and C is the number of feature map channels; The two detection channels correspond to the target position data and the target category data in the received feature data respectively; each detection channel includes a second 1x1 convolution layer, a second batch normalization layer, a first 3x3 convolution layer, a third batch normalization layer, a third 1x1 convolution layer and a fourth batch normalization layer arranged in sequence; Use the trained detection head for object detection; Before using the trained detection head for target detection, perform an equivalent transformation on it, specifically: First, convert the first 1x1 convolutional layer and the first batch of normalization layers in the feature scale transformation unit into a fourth 1x1 convolutional layer; Then, a first conversion is performed on both detection channels: Convert the second 1x1 convolutional layer and the second batch normalization layer into a fifth 1x1 convolutional layer; Convert the first 3x3 convolutional layer and the third batch normalization layer into a second 3x3 convolutional layer; Convert the third 1x1 convolutional layer and the fourth batch normalization layer to the sixth 1x1 convolutional layer; Finally, the two detection channels are converted for the second time, that is, the two detection channels after the first conversion are converted into a third 3x3 convolution layer respectively; The equivalently transformed detection head is used for target detection.

2. A target detection acceleration method based on structural reparameterization as claimed in claim 1, characterized in that ,After the second conversion of the two detection channels, they are converted for the third time, and the detection head is used to perform target detection; ,the third conversion is: Convert the two third 3x3 convolutional layers into one fourth 3x3 convolutional layer.

3. The target detection acceleration method based on structural reparameterization as claimed in claim 2, characterized in that ,The specific method of the third conversion is as follows: after splicing two third 3x3 convolutional layers with a convolutional layer with the same scale and weight of 0 respectively, the two convolutional layers obtained are spliced ​​again to obtain a fourth 3x3 convolutional layer.

Images