A lightweight real-time object detection method

Abstract

The invention provides a lightweight real-time target detection method, which effectively ensures the accuracy while realizing the lightweight of the network model. YOLOv5s is selected as a benchmark model to propose an improved network model EGBNet. Firstly, in order to help accelerate reasoning and effectively extract features, an improved version of the lightweight network ShuffleNetv2, ESNet, is cleverly used to construct a lightweight feature extraction network in the backbone network part, and an SPPF module is used to increase the receptive field of the network; then, the idea of ​​BiFPN is combined in the neck part to help multi-scale feature fusion, and the lightweight network GhostNet is also used to process the feature transition to facilitate the prediction of the detection head, and the CBAM dual-channel attention mechanism is used in the process to help improve the performance of the model; finally, a lightweight upsampling operator CARAFE is used in the neck part to perform the upsampling operation, and upsampling is performed within a larger receptive field based on the input content combined with the semantic information of the feature map to reduce the loss of information, thereby realizing the lightweight of the model while improving the accuracy.

Description

Technical Field

[0001] The present invention relates to deep learning and target detection, and in particular to a lightweight real-time target detection method. Background Art

[0002] Object detection is widely used in many computer vision tasks, including autonomous driving, machine vision, intelligent translation, industrial quality monitoring and target tracking. The application of neural network-based object detection algorithms in natural scenes is limited due to their large number of parameters and complex calculations. In response to this problem, more and more researchers are paying more attention to how to design lightweight and efficient object detection algorithms suitable for edge mobile terminals.

[0003] The two-step algorithm first generates a large number of candidate boxes and then further fine-screens them for classification and regression, such as RCNN, Faster-RCNN, etc. In the first stage, the position of the target box is directly predicted and regressed, which can greatly reduce the computational complexity and accelerate the reasoning of the model. Representative methods of this type include the YOLO series and SSD. Compared with the two-step algorithm, the one-step algorithm like the YOLO series can achieve better performance in terms of both speed and accuracy. However, the processing method of the YOLO series also faces the following problems: 1) It is necessary to manually design anchorboxes to adapt to different data sets; 2) The idea of ​​anchors will lead to an imbalance of positive and negative samples. Recently, the idea of ​​anchorfree has made great progress in balancing speed and accuracy, such as FCOS. In the field of target detection, a lot of work is devoted to achieving better performance in efficiency and accuracy. YOLOv4, which was designed through compression and compilation, became popular for a while. YOLObile's clever design achieved real-time target detection on embedded platforms. PP-YOLO-Tiny uses MobileNetV3 as the backbone network and TinyFPN in PP-YOLO as the neck structure to achieve good performance. NanoDet uses ShuffleNetV2 as the backbone network to make the model lighter, and also uses ATSS and GFL to increase the accuracy of the model. These methods can effectively ensure accuracy while improving the speed of the model and reducing the number of parameters. In addition, some lightweight network variants proposed around YOLO and SSD can achieve significant results, such as YOLO-Nano, MobileNet-SSDLite, etc. In the present invention, YOLOv5s is selected as the benchmark model to propose an improved network model EGBNet. First, in order to help accelerate reasoning and effectively extract features, the improved version of the lightweight network ShuffleNetv2, ESNet, is cleverly used to build a lightweight feature extraction network in the backbone network part, and the SPPF module is used to increase the network's receptive field; then, the idea of ​​BiFPN is combined with the neck part to help multi-scale feature fusion, and the lightweight network GhostNet is also used to transition feature processing to facilitate the prediction of the detection head, and the CBAM dual-channel attention mechanism is used in the process to help improve the performance of the model; finally, the lightweight upsampling operator CARAFE is used in the neck part to perform upsampling operations, and upsampling is performed within a larger receptive field based on the input content combined with the semantic information of the feature map to reduce information loss, thereby achieving lightweight models while improving accuracy, and making great contributions to engineering practice. Summary of the invention

[0004] The technical problem to be solved by the present invention is: to provide a lightweight real-time target detection method to solve the problem of real-time target detection for embedded platforms and edge mobile terminals, aiming to achieve lightweight models while comprehensively balancing speed and accuracy, which has important engineering practical significance.

[0005] The technical solution adopted by the present invention is as follows: A lightweight real-time target detection method comprises the following steps:

[0006] Step 1, obtain the data set: obtain images through the acquisition device, manually annotate them through the annotation tool labelimg, construct training sample sets, verification sets and test sample sets, or obtain training sample sets, verification sets and test sample sets from public data sets, such as PascalVOC or MSCOCO data sets;

[0007] Step 2: Data preprocessing: Use random flipping, random cropping, geometric changes, scale scaling, illumination changes, Mosaic and MixUp to randomly splice or synthesize multiple images into one image so that the model can adapt to various data transformations;

[0008] Step 3: Use ESNet, an improved version of the lightweight network ShuffleNetv2, to stack on the backbone network to build a lightweight and efficient feature extraction network. Use the SPPF module in the last layer of the backbone network to increase the network's receptive field information.

[0009] Step 4: In the neck part, the idea of ​​BiFPN is combined to help the fusion of multi-scale feature information. The C3Ghost module built with the lightweight network GhostNet as the basic unit is used to perform transition processing on the features to help the prediction of the detection head. In the process, the CBAM dual-channel attention mechanism module is used to suppress unfavorable information and GhostConv is used to increase feature information. The upsampling operator CARAFE is used to perform upsampling operations, which effectively combines the semantic information of the input information and increases the receptive field range.

[0010] Step 5: Use the Adam optimizer to optimize the model and train the model using the training data set obtained in step 1;

[0011] Step 6: During the training process, the validation set obtained in step 1 is loaded for validation after each iteration. The model is trained until the model weight with the optimal validation result is obtained after the number of iterations. The number of iterations can be appropriately adjusted or early stopping can be set according to the accuracy results measured on the validation set to save training time and avoid overfitting. The model weight is loaded to test the model effect on the test data set obtained in step 1, and the test set is tested 5 times to obtain the average test result.

[0012] The principle of the present invention is:

[0013] The application of neural network-based target detection algorithms in natural scenes is limited due to their large number of parameters and complex calculations. This paper selects YOLOv5s as the benchmark model, and combines lightweight networks in the backbone network and the neck part to help multi-scale feature fusion, thereby achieving lightweight models while improving accuracy.

[0014] The EGBNet model in the present invention includes three parts: the ESNet backbone network part, the GA-BiFPN neck part and the detection head. The present invention mainly improves the Backbone part and the Neck part. First, in order to reduce model parameters and achieve lightweight, an improved version of the lightweight ShuffleNetv2, ESNet, is used to build a lightweight and efficient feature extraction network in the backbone network part, and the SPPF module is used to increase the network's receptive field; then, the idea of ​​BiFPN is combined in the neck part to help multi-scale feature fusion, and the lightweight network GhostNet is used to process the feature transition for the detection head prediction, and the CBAM dual-channel attention mechanism is used in the process to help improve the performance of the model; finally, the lightweight upsampling operator CARAFE is used in the neck part to perform upsampling operations, and upsampling is performed within a larger receptive field range based on the input content combined with the semantic information of the feature map to reduce information loss, while improving the accuracy and achieving the lightweight of the model.

[0015] In summary, compared with some existing methods, the beneficial effects of the present invention are:

[0016] (1) Compared with some other models, without loading the pre-trained model weights, the model EGBNet proposed in the present invention achieved an accuracy of 55.32% mAP with 4.94M parameters, which is 2.61% mAP higher than the original YOLOv5s, and the real-time speed can reach 76.92FPS.

[0017] (2) The GA-BiFPN proposed in the present invention can effectively assist in feature transition processing without increasing the number of parameters, and can be simply used in other algorithm frameworks to effectively help improve the detection performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific details of the present invention, some drawings involved in the present invention are briefly introduced. The following drawings only show some embodiments of the present invention to facilitate detailed understanding, and do not limit the scope of the present invention in any way.

[0019] Figure 1 It is a schematic diagram of the EGBNet network structure in the present invention;

[0020] Figure 2 It is a schematic diagram of the structure of two step sizes of ESNet in the present invention, where: Figure 2 (a) and Figure 2 (b) corresponds to the two structures of ESNet with a step size of 1 and a step size of 2;

[0021] Figure 3 Schematic diagram of the Ghost Bottleneck unit module in the present invention, wherein: Figure 3 (a) and Figure 3 (b) Two structures with step length of 1 and step length of 2 respectively. Figure 3 (c) is a Ghost Block module. The C3Ghost module can be obtained by repeatedly stacking Ghost Block modules.

[0022] Figure 4 Schematic diagram of the CBAM_GhostConv module in the present invention;

[0023] Figure 5 It is a schematic diagram of the CBAM module in the present invention. DETAILED DESCRIPTION

[0024] The following is a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings. However, the following embodiments are limited to explaining the present invention, and the protection scope of the present invention should include all the contents defined in the claims; and through the following embodiments, those skilled in the art can realize all the contents of the claims of the present invention.

[0025] like Figure 1 As shown in the figure, the overall network structure diagram of the present invention mainly includes three parts: ESNet backbone network, GA-BiFPN Neck and detection head. The ESNet backbone network uses the ESNet basic unit module with a step size of 1 and adopts the stacking method of [3,9,9,3] to extract features. In the GA-BiFPN Neck part, the idea of ​​BiFPN is combined and the C3Ghost and CBAM modules are used to process the feature transition to help the detection head prediction. There are mainly the following steps:

[0026] Step 1, obtain the data set: obtain images through the acquisition device, manually annotate them through the annotation tool, construct the training sample set, the validation set and the test sample set, or obtain the training sample set, the validation set and the test set from the public data set. For example, in the present invention, VOC2007 and VOC2012 data are used together, and 21,503 pictures including VOC2007 and VOC2012 are divided into training set and test set at a ratio of about 0.75:0.25, that is, 16,551 pictures are used as the training set, and 4,952 pictures including VOC2007 and VOC2012 are used as the test set;

[0027] Step 2, data preprocessing: Use random flipping, random cropping, geometric changes, scale scaling, illumination changes, Mosaic and MixUp and other data enhancement methods to increase data diversity, so that the model can adapt to various data transformations as much as possible. Specifically, such as random flipping, median filtering, multi-scale transformation, HSV enhancement, etc., the data after data enhancement is richer, the model can learn more features, and it is more robust in actual test performance;

[0028] Step 3. Use the improved version of the lightweight network ShuffleNetv2, ESNet, to stack on the backbone network to build a lightweight and efficient feature extraction network. Use the SPPF module in the last layer of the backbone network to increase the network's receptive field information. Specifically, the backbone network uses the improved version of the lightweight network ShuffleNetv2, ESNet, as the basic unit module, which can be divided into two types with a step size of 1 and a step size of 2. The basic unit module of ESNet with a step size of 1 is stacked by stacking the number of times [3,9,9,3]. Before the first stacking, a convolution operation with a step size of 2 is used for downsampling. The other layers are downsampled by adding a basic unit module of ESNet with a step size of 2 in the middle. A total of 5 downsamplings are performed. The SPPF module is used in the last layer of the backbone network to increase the receptive field and enrich the expressive power of the feature map. SPPF is an accelerated and improved version of SPP, which can reduce the number of floating-point operations and help accelerate reasoning with the same results.

[0029] Step 4. In the neck part, the idea of ​​BiFPN is combined to help the fusion of multi-scale feature information. The C3Ghost module constructed with the lightweight network GhostNet Bottleneck as the basic unit is used to perform transition processing on the features to help the prediction of the detection head. In the process, the CBAM dual-channel attention mechanism module is used to suppress unfavorable information and GhostConv is used to increase feature information. It should be noted that the upsampling operator CARAFE is used here to perform upsampling operations, which effectively combines the semantic information of the input information and increases the receptive field range. Specifically, C3Ghost constructed by GhostBottleneck in GhostNet is used to process the feature transition to help the prediction of the detection head. Each time the C3Ghost module is used, it is stacked 3 times. During the process, the CBAM_GhostConv unit is used to assist in feature processing. It is obvious that the CBAM attention mechanism module and the GhostConv module are included, wherein GhostConv is used to change the number of feature channels and perform downsampling operations. In the upsampling process of the neck part, GhostConv with a convolution kernel size of 1 and a step size of 1 is used to change the number of feature channels. In the downsampling process of the neck part, GhostConv with a convolution kernel size of 3 and a step size of 2 is used for downsampling. In order to further improve the performance of the model, the CBAM module is used before each GhostConv to focus on the area of ​​interest in the scene. In summary, the neck part proposed in the present invention is called GA-BiFPNNeck. According to statistics, GA-BiFPNNeck contains a total of 4 CBAM_GhostConv unit modules, 2 CARAFE upsampling operators, 4 Concat splicing modules and 12 C3Ghost modules.

[0030] Figure 4 This is a schematic diagram of the CBAM_GhostConv module in the present invention: some unfavorable information is first suppressed by the CBAM module, and then the output is obtained through the processing of GhostConv. GhostConv mainly combines the idea of ​​using GhostNet, first generates a part of the feature map through k×k convolution, and the convolution kernel size k=1 and step size s=1 during upsampling, and the convolution kernel size k=3 and step size s=2 during downsampling, and then generates another part of the feature map through a 5×5 deep convolution, and then the two parts of the feature map are spliced ​​to get the final output.

[0031] Figure 5 This is a schematic diagram of the CBAM module in the present invention: the CBAM module successively obtains a weighted feature map through the channel attention response path and the spatial dimension attention corresponding path branches, and then multiplies the weighted feature map with the input feature map to obtain an output refined feature map after adaptive selection.

[0032] Step 5: Set the number of training iterations to 300, the batch normalization size to 16, the initial learning rate to 1e-3, use the Adam optimizer to optimize the model, and train the model using the training data set obtained in step 1;

[0033] Step 6: During the training process, the validation set obtained in step 1 is loaded for validation after each iteration of training. The current validation accuracy result is recorded and compared each time. When the accuracy is higher, the current model weight is saved until the iteration cycle ends. Finally, the model weight with the best accuracy of the validation result is saved. This model weight is loaded and the test set obtained in step 1 is used for inference testing.

[0034] In order to verify the effectiveness of the present invention, the number of parameters, mAP and detection speed FPS are used as evaluation indicators of model performance, and the mAP here mainly refers to AP@0.5:0.95. Among them, the number of parameters is used to measure the lightweight of the model, mAP is used to measure the detection accuracy of the model, and the number of images processed per second (FPS) is used as a real-time evaluation indicator. Try to combine the backbone network part with GhostNet to generate YOLOv5s-Ghost, try to combine the backbone network part with MobileNetv3 to generate YOLOv5s-MobileNetV3, and use YOLOv5s for comparison. The comparison results are shown in Table I.

[0035] Table I Performance comparison of each model

[0036]

[0037] As shown in Table I, among the algorithm models involved in the comparison, the accuracy of the two models generated by combining the lightweight network in the backbone network part has decreased accordingly, and the speed of the two models has been improved to a certain extent. Among them, the accuracy result obtained by YOLOv5s-MobileNetV3 has almost decreased by 9.51% mAP. In contrast, the result obtained by combining MobileNetV3 is not as good as that obtained by combining GhostNet in terms of accuracy and speed, indicating that GhostNet can effectively generate high-quality feature maps here. The method EGBNet proposed in this invention has achieved the highest accuracy of 55.32% mAP with a parameter amount of 4.94MB, which is relatively Compared with the original YOLOv5s' 52.71% mAP, it is 2.61% higher, and the real-time speed is 76.92FPS. Although it is not as fast as the original YOLOv5s' 113.63FPS, it can still meet the real-time requirements of most scenarios, and its accuracy improvement is comparable. In addition, its model is lightweight to a large extent, with only 4.94MB of model parameters. This result is in line with our initial design point, and can better balance the speed and accuracy. It also provides the possibility for model deployment on embedded platforms and edge devices, which is of great significance to engineering practice.

[0038] The parts not elaborated in detail in the present invention belong to the common knowledge in the art. The above description is only a preferred example of the present invention and is not intended to limit the present invention. Those skilled in the art can make appropriate adjustments and effectiveness improvements according to specific application scenarios.

Claims

1. A lightweight real-time target detection method, characterized in that: The steps include: Step 1. Obtain the data set: obtain images through the acquisition device, manually annotate them through the labeling tool labelimg, construct training sample sets, verification sets and test sample sets, or obtain training sample sets, verification sets and test sample sets from public data sets, or from Pascal VOC or MS COCO data sets; Step 2: Data preprocessing: Use random flipping, random cropping, geometric changes, scale scaling, illumination changes, and use Mosaic and MixUp to randomly splice or synthesize multiple images into one image, so that the model can adapt to various data transformations; Step 3: Use the improved version of the lightweight network ShuffleNetv2 ESNet to stack on the backbone network to build a lightweight and efficient feature extraction network, and use the SPPF module in the last layer of the backbone network to increase the network's receptive field information; specifically, the backbone network uses the improved version of the lightweight network ShuffleNetv2 ESNet as the basic unit module, which can be divided into two types with a step size of 1 and a step size of 2. The basic unit module of ESNet with a step size of 1 is stacked by using the stacking times of [3, 9, 9, 3]. Before the first stacking, a convolution operation with a step size of 2 is used for downsampling. The other layers are all downsampled by adding a basic unit module of ESNet with a step size of 2 in the middle. A total of 5 downsamplings are performed. In the last layer of the backbone network, the SPPF module is used to increase the receptive field and enrich the expression ability of the feature map. SPPF is an accelerated and improved version of SPP, which can reduce the number of floating-point operations and help accelerate reasoning under the same results. Step 4. In the neck part, the idea of ​​BiFPN is combined to help the fusion of multi-scale feature information. The C3Ghost module constructed by the lightweight network GhostNet as the basic unit is used to perform transition processing on the features to help the prediction of the detection head. In the process, the CBAM dual-channel attention mechanism module is used to suppress unfavorable information and GhostConv is used to increase feature information. The upsampling operator CARAFE is used to perform upsampling operations, which effectively combines the semantic information of the input information and increases the receptive field range. Specifically, C3Ghost constructed by Ghost Bottleneck in GhostNet is used to process the feature transition to help the prediction of the detection head. Each time the C3Ghost module is used, it is stacked 3 times. In the process, the CBAM_GhostConv unit is used to help feature processing. Obviously, it includes the CBAM attention mechanism module and the GhostConv module. GhostConv is used to change the number of feature channels and perform downsampling operations. In the upsampling process of the neck part, GhostConv with a convolution kernel size of 1 and a step size of 1 is used. To achieve the change of the number of feature channels, GhostConv with a convolution kernel size of 3 and a step size of 2 is used for downsampling in the neck part. In order to further improve the performance of the model, the CBAM module is used before each GhostConv to focus on the area of ​​interest in the scene. In summary, the proposed neck part is called GA-BiFPN Neck. According to statistics, GA-BiFPN Neck contains a total of 4 CBAM_GhostConv unit modules, 2 CARAFE upsampling operators, 4 Concat splicing modules and 12 C3Ghost modules; Step 5: Use the Adam optimizer to optimize the model and train the model using the training data set obtained in step 1; Step 6: During the training process, the validation set obtained in step 1 is loaded for validation after each iteration. The model is trained until the model weight with the optimal validation result is obtained after the number of iterations. The number of iterations is appropriately adjusted or early stopping is set according to the accuracy results measured on the validation set to save training time and avoid overfitting. The model weight is loaded to test the model effect on the test data set obtained in step 1. The test set is tested 5 times to obtain the average test result.

Images