Implementation method of lightweight convolutional neural network target detection

Abstract

The application discloses an implementation method of a lightweight convolutional neural network target detection, which respectively performs lightweight on anchor boxes, backbone networks and detection heads of a target detection model, improves a maximum intersection over union training sample matching strategy through lightweight anchor boxes, alleviates the imbalance problem of the number of positive samples among different targets, greatly reduces the density of anchor boxes while improving detection accuracy, adopts a lightweight convolutional neural network ShuffleNetV2 as a backbone network, increases a channel attention module, re-calibrates channel features, learns the importance of different channel features and the relationship between channel features, and proposes a serial lightweight detection head structure for lightweight of the detection head, connects a convolution of a regression branch of the detection head after a convolution of a classification branch, makes the regression branch fully utilize the classification result, improves positioning accuracy of the model, and uses a grouping convolution to lightweight the detection head. The application performs lightweight on the target detection network, greatly improves network reasoning speed while ensuring accuracy.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and computer vision technology, and relates to object detection and the application of lightweight convolutional neural networks in vehicle detection. It is a method for implementing object detection using lightweight convolutional neural networks. Background Technology

[0002] Deep learning is a type of machine learning that extracts and combines specific low-level features into abstract high-level features by stacking multiple hidden layers. Through backpropagation, it learns the distribution of data, fits a functional relationship from input to output, and further achieves functions such as recognition. With the development of deep learning, a new structure called the convolutional neural network (CNN) has emerged within deep neural networks. CNNs transform the computation between layers of fully connected networks from fully connected computation to convolutional computation. The characteristics of convolutional computation are sparse connections and weight sharing, which greatly reduces the number of parameters and computational cost, making it suitable for image applications.

[0003] With the widespread application of deep convolutional neural networks (CNNs) and their great success in image recognition and object detection, practical deployments require limitations on the number of parameters and computational load to meet real-time requirements while minimizing accuracy degradation. Lightweight CNNs address this by redesigning models to find more efficient computational methods. This is primarily achieved by changing the convolution and its organization to reduce parameters and computational load without compromising accuracy. Common lightweight convolutions include grouped convolutions and depthwise separable convolutions, and common lightweight networks include MobileNet, SqueezeNet, and ShuffleNet.

[0004] Object detection is an important branch of image processing and computer vision, and it is the core and foundation of intelligent video application systems. The task of object detection is to find foreground objects in an image and output their location and category. Traditional object detection algorithms generally consist of three steps: region selection, feature extraction, and classification. Their drawbacks include poor robustness to factors such as lighting and angle, non-end-to-end processing, and slow speed. Deep learning-based object detection algorithms are mainly divided into two-stage and one-stage algorithms. Two-stage algorithms first perform binary classification and regression on anchor boxes to extract region proposals, and then further classify and regress these proposals. They are characterized by high accuracy but slow speed. One-stage algorithms directly perform fine classification and regression on the anchor boxes, characterized by lower accuracy but faster speed. Vehicle detection is a more sub-field of object detection research. Vehicle detection has high requirements for both accuracy and efficiency, and current deep learning detection methods cannot yet meet these needs and require improvement. Summary of the Invention

[0005] The problem this invention aims to solve is that vehicle detection is the core and foundation of intelligent transportation systems, significantly impacting the system's accuracy and real-time performance. In practical applications, it is desirable for detection methods to better balance accuracy, real-time performance, and cost-effectiveness. The purpose of this invention is to lightweight existing target detection models without sacrificing accuracy, reducing the number of parameters and computational load, thereby improving the speed of the detection model.

[0006] The technical solution of this invention is: a method for implementing lightweight convolutional neural network object detection, which involves lightweighting the object detection convolutional neural network, including lightweight anchor boxes, lightweight backbone network, and lightweight detection head. Two or all three lightweighting methods can be selected. Specifically, the three lightweighting methods are as follows:

[0007] S1 Lightweight Anchor Box: Includes setting lightweight anchor boxes and improving sample matching; only one anchor box is set for each feature point, let the number of Ground Truths be N, sample labels -1 represent ignored samples, 0 represent negative samples, [1,n] represent positive samples, n∈N, and the improved sample matching is as follows:

[0008] S1.1) Set all sample labels to -1 initially, which means that the samples are ignored during training.

[0009] S1.2) Set negative samples. If the intersection-union ratio (IUU) of a sample with any Ground Truth is less than the negative sample threshold T... neg If the sample is negative, then the label is set to 0.

[0010] S1.3) Set positive samples. If the maximum value of the intersection-union ratio (IU) of a sample with all Ground Truths is greater than the positive sample threshold T. pos If the sample is positive, the label is set to the corresponding Ground Truth number, and [1,n] represents the sample as positive and the corresponding Ground Truth.

[0011] S1.4) Balance the number of positive samples in Ground Truth. If the number of positive samples matched by a certain Ground Truth is less than k, match the k samples with the largest intersection-union ratio with that Ground Truth to that Ground Truth. k is a hyperparameter.

[0012] S1.5) Suppress low-quality positive samples. If the center point of a positive sample does not fall within the matched Ground Truth, then cancel the match.

[0013] S2 Lightweight Backbone Network: The lightweight convolutional neural network ShuffleNetV2 is used as the backbone network. Lightweight settings include:

[0014] S2.1) Connect the backbone network ShuffleNetV2 with the feature pyramid network FPN, and use the output feature maps of Stage2, Stage3 and Stage4 of ShuffleNetV2 as input to FPN to construct the feature pyramid structure;

[0015] S2.2) Add a channel attention module to ShuffleNetV2 to recalibrate channel features and learn the importance of different channel features and the relationship between channel features;

[0016] S3 Lightweight Detector Head: Construct a serial lightweight detector head structure, connecting the regression branch convolution after the classification branch convolution, so that the regression branch can make full use of the classification results, and the output classification prediction and regression prediction convolution positions remain unchanged.

[0017] To accelerate the inference speed of the object detection network, this invention lightweights the network by reducing the size of the anchor boxes, backbone network, and detection head. This significantly improves the network's inference speed while maintaining accuracy.

[0018] This invention improves the maximum intersection-union ratio (MUC) training sample matching strategy to alleviate the imbalance of positive sample numbers among different targets. While improving detection accuracy, it can significantly reduce the density of anchor boxes, making the anchor boxes lightweight. The backbone network commonly used in target detection algorithms is replaced by ResNet50 with a lightweight convolutional neural network, ShuffleNetV2, and a channel attention module is added to recalibrate channel features and learn the importance of different channel features and the relationships between them. A serial lightweight detection head structure is proposed, where the regression branch convolution of the detection head is connected after the classification branch convolution. The classification and regression are changed from the original parallel structure to a serial structure, so that the regression branch can make full use of the classification results, improve the model's localization accuracy, and use grouped convolution to make the detection head lightweight. Attached Figure Description

[0019] Figure 1 This is a flowchart of the vehicle detection implementation method based on lightweight convolutional neural networks of the present invention.

[0020] Figure 2 This is a flowchart of the lightweight anchor point frame of the present invention.

[0021] Figure 3 (a) is a schematic diagram of the residual block structure of ShuffleNetV2, a lightweight backbone network based on channel attention, with a step size of 1.

[0022] Figure 3 (b) is a schematic diagram of the residual block structure of ShuffleNetV2, a lightweight backbone network based on channel attention, with a step size of 2 according to the present invention.

[0023] Figure 4 This is a schematic diagram of the structure of the serial lightweight detection head of the present invention. Detailed Implementation

[0024] This invention proposes a lightweight convolutional neural network object detection method that can significantly improve detection speed without sacrificing detection accuracy.

[0025] The present invention will be further described below with reference to the accompanying drawings and examples.

[0026] This invention relates to a lightweight object detection convolutional neural network, which includes anchor boxes for region selection, a backbone network for feature extraction, and a detection head for classification and recognition. Figure 1 As shown in the embodiment, the object detection convolutional neural network is the RetinaNet object detection model. The lightweighting of this invention comprises three parts: lightweight anchor boxes, a lightweight backbone network, and a lightweight detection head. When lightweighting the convolutional neural network, any two or all three lightweighting methods can be selected.

[0027] S1 Lightweight Anchor Box: Includes lightweight anchor box setup and improved sample matching. This invention improves the maximum intersection-union-ratio (MUR) training sample matching strategy within the lightweight anchor box, alleviating the problem of imbalanced positive sample numbers between different targets. Using the improved sample matching strategy, while improving detection accuracy, the density of anchor boxes can be significantly reduced. Lightweight anchor boxes reduce the number of parameters and computational load, improving running speed. The flowchart is shown below. Figure 2 As shown, the details are as follows:

[0028] For each feature point, only one anchor box is set. Let the number of ground truths be N, and the sample label be -1 to indicate an ignored sample, 0 to indicate a negative sample, and [1,n] to indicate a positive sample, where n∈N. The improved sample matching is as follows:

[0029] S1.1) Set all samples to -1 by default, which means that the samples are ignored during training;

[0030] S1.2) Set negative samples. If the intersection-union ratio (IUU) of a sample with any Ground Truth is less than the negative sample threshold T... neg If the sample is negative, then the label is set to 0.

[0031] S1.3) Set positive samples. If the maximum value of the intersection-union ratio (IU) of a sample with all Ground Truths is greater than the positive sample threshold T. pos If the sample is positive, then the label is set to the corresponding Ground Truth number.

[0032] The maximum intersection-union (MUC) matching strategy (S1.1)-S1.3) partitions the samples, but this mechanism has the following drawbacks:

[0033] (1) This leads to large-scale objects matching more positive samples and small-scale objects matching fewer positive samples. This mechanism naturally results in small objects containing less gradient information during training, making it difficult to achieve the same learning effect as large objects. Therefore, the accuracy is significantly lower than that of large objects.

[0034] (2) Dense anchor boxes are required. In order for small objects to match positive samples as much as possible, denser anchor boxes are needed. Otherwise, sparse anchor boxes may cause most small objects to fail to match positive samples. Existing detection models, such as the RetinaNet object detection model, set 9 anchor boxes at each feature point, with 3 scales and 3 aspect ratios. However, the drawback of this is that it increases the number of network parameters and computational cost, and reduces the inference speed of the network.

[0035] The improved maximum intersection-union ratio (MUC) sample matching strategy of this invention adds steps S1.4)-S1.5).

[0036] S1.4) Balance the number of positive samples in Ground Truth. If the number of positive samples matched by a certain Ground Truth is less than k, match the k samples with the largest intersection-union ratio with that Ground Truth to that Ground Truth. k is a hyperparameter.

[0037] S1.5) Suppress low-quality positive samples. If the center point of a positive sample does not fall within the matched Ground Truth, then cancel the match.

[0038] The method of this invention balances the number of positive samples in the Ground Truth, making up for the small number of positive samples in the Ground Truth, thus improving the accuracy of small objects. At the same time, it can significantly reduce the number of anchor boxes, setting only one anchor box at each feature point, reducing it to 1 / 9 of the original number without sacrificing accuracy. Meanwhile, the inference speed is greatly improved, thanks to the improved sample matching strategy.

[0039] S2 Lightweight Backbone Network: Taking the RetinaNet object detection algorithm as an example, the original backbone network ResNet50 is replaced with a lightweight convolutional neural network ShuffleNetV2 as the backbone network, and a channel attention module is added to recalibrate channel features and learn the importance of different channel features and the relationships between channel features.

[0040] S2.1) Selecting a suitable backbone network. The vehicle detection network adopts the RetinaNet algorithm, and the backbone network of the RetinaNet algorithm is ResNet50. The number of parameters of ResNet50 is 25.56M, and the computational FLOPs are 8.28G; the number of parameters of ShuffleNetV2 1.0x is 2.3M, and the computational FLOPs are 146M; the number of parameters of ShuffleNetV2 1.5x is 3.5M, and the computational FLOPs are 299M. It can be seen that the model size of ShuffleNetV2 is much smaller than that of ResNet50. The ResNet50 structure is formed by stacking residual blocks, which use traditional convolutions. There is redundant computation in the densely connected structure of 50 layers, so the lightweight ShuffleNetV2 1.5x model is used as the backbone network of RetinaNet. The number of output channels per stage of ShuffleNetV2 1.5x is 1.5 times that of 1.0x, which has stronger feature extraction capabilities and stronger model expressive power. While significantly reducing the number of parameters and computation, it also ensures the accuracy of the detection algorithm.

[0041] In the above analysis, the number of network parameters refers to the number of learnable parameters. The computational complexity is measured using floating-point operations (FLOPs), which calculate the number of multiplication and addition operations. Specifically, FLOPs are calculated as follows: Assume the feature map size of a certain layer in the network is (H, W, C1), where H is the length, W is the width, C1 is the number of channels, the convolutional kernel size is K*K*C1*C2, the stride is S, and the padding is P. Then, the output feature map size is (L1, L2, C2), where... This indicates rounding down. The number of parameters for this layer is K. 2 C1C2+C2, FLOPs is 2K 2 C1L1L2C2. The purpose of this invention is to reduce the network's weight and computational load; the metric for measuring computational load is FLOPs.

[0042] S2.2) After selecting the backbone network, connect the backbone network ShuffleNetV2 1.5x with the Feature Pyramid Network (FPN). In RetinaNet, the feature extraction network forms a feature pyramid from feature maps C3, C4, and C5, and then performs 1x1 convolutions to output M3, M4, and M5 (P5). P5 is upsampled and fused with M4 to output P4. P4 is upsampled and fused with M3 to output P3. P5 is then convolved with a stride of 2 to output P6, and P6 is convolved with a stride of 2 to output P7. This outputs five feature maps: P3, P4, P5, P6, and P7. The dimensions of the feature maps corresponding to C3, C4, and C5 are as follows: H and W represent the height and width of the input image. FPN is a structure in RetinaNet. After replacing the backbone network with ShuffleNetV2, the question arises about how to connect it with FPN. After research, this invention uses the output feature maps of Stage 2, Stage 3, and Stage 4 of ShuffleNetV2 as input to FPN to construct a feature pyramid structure.

[0043] S2.3) To learn the importance of different channel features and the relationships between channels, this invention adds a channel attention module to ShuffleNetV2. The ShuffleNetV2 residual block contains residual branches and skip-layer branches. The channel attention module is attached after the residual branch. After feature recalibration by the channel attention module, it is then concatenated with the skip-layer feature map. The channel attention module is used on Stage 3 and Stage 4 of ShuffleNetV2. For example... Figure 3 As shown, ShuffleNetV2 includes two types of residual blocks with a stride of 1 and a stride of 2, as detailed below:

[0044] (1) For the ShuffleNetV2 residual block with a stride of 1 based on the attention channel, the input feature map is divided into two parts according to the channel, namely the residual branch and the skip branch. The residual branch goes through 1x1 convolution, BN, ReLU, then through 3x3 depthwise separable convolution, BN, then through 1x1 convolution, BN, ReLU. Then feature recalibration is performed. Let the feature map size be HxWxC. After global average pooling, it becomes 1x1xC. After the first fully connected layer, in order to reduce the number of parameters, the channel is compressed to C / r, and the output size is 1x1xC / r. After the ReLU activation function, after the second fully connected layer, the channels are restored and the output size is 1x1xC. After the Sigmoid function, the channel weights are scaled to (0,1). Then it is multiplied element-wise with the original feature map. During the multiplication, the weight vector is broadcast to HxWxC. The weighted feature map is obtained after passing through the channel attention module. It is then concatenated with the feature map from the skip branch along the channel dimension. After a channel shuffling operation to prevent channel separation and information blockage between channels, the final feature map is output. The feature map size remains unchanged after passing through a residual block with a stride of 1.

[0045] (2) For the ShuffleNetV2 residual block with a stride of 2 based on the attention channel, the stride of the 3x3 depthwise separable convolution is set to 2 based on the stride of 1. At the same time, in order to ensure that the size of the concatenated feature maps is consistent in the later concatenation, the skip branch also needs to perform depthwise separable convolution and 1x1 convolution with a stride of 2. Channel separation is no longer performed when inputting the feature map. The rest of the operation is the same as that with a stride of 1. After the residual block with a stride of 2, the width and height of the feature map are halved, and the number of channels is multiplied by 2.

[0046] (3) The lightweight backbone network architecture is shown in Table 1.

[0047] Table 1

[0048]

[0049] The input image size is 768x1344. After passing through Stage 1 (Conv1 and MaxPool), the image size becomes 384x672 and 192x336 respectively. After Stage 2 (one ShuffleNetV2 residual block with a stride of 2 and three ShuffleNetV2 residual blocks with a stride of 1), the image size becomes 96x168. After Stage 3 (one ShuffleNetV2 residual block with a stride of 2 and seven ShuffleNetV2 residual blocks with a stride of 1), the image size becomes 48x84. After Stage 4 (one ShuffleNetV2 residual block with a stride of 2 and three ShuffleNetV2 residual blocks with a stride of 1), the image size becomes 24x42. Using ShuffleNetV2 at 1.5x scale, the number of output channels for Stages 1-4 are 24, 176, 352, and 704 respectively. Channel attention modules are used on Stage 3 and Stage 4. The output feature maps of Stage 2, Stage 3, and Stage 4 are used as inputs to FPN to further generate feature pyramids.

[0050] S3 Lightweight Detection Head: Constructing a serial lightweight detection head structure. This invention connects the regression branch convolution of the detection head after the classification branch convolution, adjusting the classification and regression from the parallel structure of the existing model to a serial structure. This allows the regression branch to fully utilize the classification results, improving the model's localization accuracy, and further utilizes a grouped convolution lightweight detection head.

[0051] S3.1) Adjust the structure of the RetinaNet header by connecting the four cascaded convolutions of the regression branch to the four cascaded convolutions of the classification branch, while keeping the positions of the output classification prediction and regression prediction convolutions unchanged. In other words, the original parallel feature extraction is changed to serial feature extraction.

[0052] S3.2) To further reduce the network's weight, the four convolutions in the regression branch use grouped convolutions, with g as the number of groups. g is a hyperparameter; experimentally, g=4 achieves the optimal balance between accuracy and speed. The serial lightweight detection head structure is as follows: Figure 4 As shown.

[0053] This invention promotes network lightweighting by including lightweight anchor boxes, lightweight backbone networks, and lightweight detection heads. Two or all three lightweighting methods can be selected. Lightweight anchor boxes accelerate the detection network and improve accuracy, lightweight backbone networks significantly accelerate the detection network but may slightly reduce accuracy, and lightweight detection heads can slightly accelerate and improve accuracy. Combining any two methods can achieve simultaneous improvement in accuracy and speed. Preferably, all three methods are used in combination, with lightweight anchor boxes and lightweight detection heads used to compensate for the accuracy loss of lightweight backbone networks, achieving a significant acceleration and accuracy improvement for the overall convolutional network.

[0054] Table 2 shows the performance of vehicle detection based on a lightweight convolutional neural network. The dataset is a tunnel scene vehicle detection dataset labeled by the laboratory. The target categories include cars, buses, minibuses, trucks, and vans. 2000 frames of images were randomly extracted from multiple cameras and at multiple time periods as the dataset. The training set includes 1500 images, and the test set includes 500 images. Baseline represents the original RetinaNet, ShuffleNetV2 1.5x SE represents the ShuffleNetV2 1.5x scale model based on the channel attention module, i.e., the S2 lightweight backbone network, BLA (Balanced LabelAssign) represents the balanced sample matching strategy, the hyperparameter k is set to 10, the anchor boxes are set to one scale and one aspect ratio, i.e., the S1 lightweight anchor boxes, SerialHead represents the serial lightweight detection head, and the number of grouped convolutions in the classification and regression branches is set to 4, i.e., the S3 lightweight detection head. Table 1 shows that the vehicle detection method based on lightweight convolutional neural networks can improve AP50 by 8.7%, significantly improving the detector's accuracy. AP decreases by 0.9%, slightly reducing the detector's localization performance. The number of parameters is reduced to nearly 1 / 5 of the original, the computational load is reduced to about 1 / 4.5 of the original, and the running speed is nearly twice as fast. Note that the following experimental results were obtained with an input image size of 1333×800.

[0055] Table 2

[0056]

Claims

1. A lightweight convolutional neural network object detection implementation method, characterized by: A convolutional neural network for object detection is used for vehicle detection. The dataset consists of road scene images randomly captured from multiple cameras at multiple time periods. Training and testing sets are used to train the object detection convolutional neural network. The network is then lightweighted, including lightweight anchor boxes, a lightweight backbone network, and a lightweight detection head. Two or all three lightweighting methods can be selected. The three lightweighting methods are as follows: S1 Lightweight Anchor Box: Includes setting lightweight anchor boxes and improving sample matching; only one anchor box is set for each feature point, let the number of Ground Truths be N, sample labels -1 represent ignored samples, 0 represent negative samples, [1,n] represent positive samples, n∈N, and the improved sample matching is as follows: S1.1) Set all sample labels to -1 initially, which means that the samples are ignored during training. S1.2) Set negative samples. If the intersection-union ratio (IUU) of a sample with any Ground Truth is less than the negative sample threshold T... neg If the sample is negative, then the label is set to 0. S1.3) Set positive samples. If the maximum value of the intersection-union ratio (IU) of a sample with all Ground Truths is greater than the positive sample threshold T. pos If the sample is positive, the label is set to the corresponding Ground Truth number, and [1,n] represents the sample as positive and the corresponding Ground Truth. S1.4) Balance the number of positive samples in Ground Truth. If the number of positive samples matched by a certain Ground Truth is less than k, match the k samples with the largest intersection-union ratio with that Ground Truth to that Ground Truth. k is a hyperparameter. S1.5) Suppress low-quality positive samples. If the center point of a positive sample does not fall within the matched Ground Truth, then cancel the match. S2 Lightweight Backbone Network: The lightweight convolutional neural network ShuffleNetV2 is used as the backbone network. Lightweight settings include: S2.1) Connect the backbone network ShuffleNetV2 with the feature pyramid network FPN, and use the output feature maps of Stage2, Stage3 and Stage4 of ShuffleNetV2 as input to FPN to construct the feature pyramid structure; S2.2) Add a channel attention module to ShuffleNetV2 to recalibrate channel features and learn the importance of different channel features and the relationships between channel features; S3 Lightweight Detection Head: Construct a serial lightweight detection head structure, connecting the regression branch convolution after the classification branch convolution, so that the regression branch can make full use of the classification results, and the output classification prediction and regression prediction convolution positions remain unchanged; By employing the aforementioned lightweighting method, the convolutional neural network for target detection is made lighter without sacrificing the accuracy of the detection model, thereby reducing the number of parameters and computational load and improving the speed of the detection model.

2. The method for implementing lightweight convolutional neural network object detection according to claim 1, characterized in that: In S2.2), the ShuffleNetV2 residual block contains residual branches and skip branches. The channel attention module is connected after the residual branch. After the channel features are recalibrated by the channel attention module, they are then concatenated with the skip feature map.

3. The method for implementing lightweight convolutional neural network object detection according to claim 1, characterized in that: When using a lightweight detection head, grouped convolution is applied to the regression branch, with the number of groups being g, where g is a hyperparameter.

4. The method for implementing lightweight convolutional neural network object detection according to claim 3, characterized in that: The number of groups, g, is 4.

5. The method for implementing lightweight convolutional neural network object detection according to claim 1, characterized in that: In S1.4), k takes the value 10.

6. The method for vehicle detection based on a lightweight convolutional neural network according to claim 1, characterized in that: The convolutional neural network for object detection is the RetinaNet model for object detection.

Images