An object detection model based on multi-scale receptive field feature fusion

Abstract

The present invention relates to the field of target detection technology, and in particular to a target detection model based on multi-scale receptive field feature fusion. The present invention designs a new module CspRepSPP, uses depthwise convolution to replace the maximum pooling in the SPPF module, performs convolution operations on each channel of the feature map through depth convolution, effectively solves the information loss problem existing in the original module, and further reduces the computational complexity of the module during the reasoning process through structural reparameterization and CSP (CrossStagePartial). Compared with SPPF, the new module CspRepSPP has a faster computing speed and greater accuracy improvement in the reasoning stage. At the same time, inspired by the multi-scale receptive field feature fusion idea of ​​SPP, the present invention designs a new feature extraction module ImprovedBottleNeck, which replaces the traditional BottleNeck in YOLOv5, so that the target detection model fuses features of receptive fields of different sizes at each stage, further improving the detection accuracy of the model without additional computational complexity.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection algorithms, and in particular to a target detection model based on multi-scale receptive field feature fusion. Background Art

[0002] Object detection is a crucial technology in computer vision, widely used in scenarios such as public transportation, medical imaging, and remote sensing. YOLO (You Only Look Once) is a series of single-stage regression-based object detection algorithms that balances high detection speed with high accuracy. SPP (Spatial Pyramid Pooling) is a multi-scale receptive field module in YOLOv3. It primarily achieves feature fusion at three different receptive field scales through max pooling with three kernel sizes (k=13, 9, 5) (adjusting stride and padding to ensure the output feature map has the same size as the input) and channel-wise concatenation. YOLOv5's SPPF (SPP Fast) is an upgrade to SPP. It achieves the same functionality as SPP by concatenating three max pooling operations with smaller kernels (k=5) and channel-wise concatenation, but with much faster computation speed.

[0003] Although both SPP and SPPF extract features through maximum pooling, which has a small computational cost, the maximum pooling method only extracts the pixel with the largest pixel value in the receptive field each time, resulting in a large amount of information loss (see Figure 1 While YOLOv5's SPPF module can quickly achieve multi-scale receptive field feature fusion, it underutilizes pixel information. Furthermore, YOLOv5 only extracts features using the fixed-size receptive field of the feature extraction module BottleNeck, without considering multi-scale receptive fields. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is that in the prior art, YOLOv5 only extracts features through the fixed-size receptive field of the feature extraction module BottleNeck, without considering the technical problem of multi-scale receptive fields.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions:

[0006] A target detection model based on multi-scale receptive field feature fusion, characterized by using CspRepSPP based on deep convolution to extract features. The CspRepSPP uses deep convolution to solve the problem of maximum pooling information loss, and uses structural reparameterization and CSP to solve the problem of large deep convolution computational complexity.

[0007] Preferably, the construction method of the CspRepSPP is as follows:

[0008] Based on the SPPF of YOLOv5, we use deep convolution to replace maximum pooling and change the channel splicing to element-wise addition to design a new module RepSPP.

[0009] Structural reparameterization technology is used to optimize branches with N structures connected in parallel and the results added. The principle of structural reparameterization is as follows:

[0010]

[0011] Among them, M (2) 、M (1) are the output and input feature maps of each branch respectively, bn is batch normalization (as shown in Formula 2), W (i) is the convolution kernel of the i-th convolution, * represents the depth convolution operation, μ (i) , σ (i) is the mean and variance obtained after the i-th convolution, γ (i) , β (i) are two learnable variables in bn. In the present invention, since RepSPP performs 4 depthwise convolution operations, N=4.

[0012]

[0013]

[0014]

[0015] Where x(i) is the i-th feature of feature map M, and m is the total number of features. Batch normalization calculates the mean and standard deviation of the feature map using formulas (3) and (4). According to formula (2), the mean μ is subtracted from the feature map M, divided by the standard deviation σ, multiplied by the scaling factor γ, and added to the bias β, where γ and β are learnable variables. According to formula (2), formula (1) can be further simplified:

[0016]

[0017] Formula (5) first combines the convolution operation with batch normalization according to formula (2), and then combines multiple convolutions into one convolution. The convolution kernel weight of the model in the inference phase is W′ and the bias is β′;

[0018] The optimized model is CSPified in two stages to obtain the model CspRepSPP.

[0019] Preferably, the structural reparameterization is combined with the deep convolution to design the feature extraction module ImprovedBottleNeck, wherein the Improved BottleNeck uses a deep convolution with a kernel size of 7*7 and 3*3 during the training phase and adds the results.

[0020] Preferably, the Improved BottleNeck also sets a residual connection, and to prevent overfitting, a regularization method Drop Path is added before the residual connection.

[0021] Preferably, during the inference phase, the two depthwise convolutions of Improved BottleNeck can be fused into one using structural reparameterization.

[0022] Preferably, it comprises a Backbone part, a Neck part and a Head part, wherein the Backbone part comprises a downsampling module and a feature extraction operation module, the Neck part performs a deeper fusion of the information extracted from the Backbone part, and the Head part is used to output the prediction result.

[0023] Preferably, the downsampling module is implemented by a convolution with a stride of 2, and the feature extraction module is a C3 module. The C3 module first reduces the number of channels of the input feature map by half through two convolutions with a kernel size of 1*1, and uses N BottleNeck modules on one of the branches, splicing the result with the output of the other branch in the channel dimension, and finally outputs it through a convolution with a kernel size of 1*1.

[0024] Preferably, the Neck part performs a deeper fusion of the information extracted from the backbone, and its structure is PANet (Path Aggregation Network), that is, it gradually splices smaller feature maps from top to bottom, and then gradually splices larger feature maps from bottom to top. The fusion of feature maps of different sizes can improve the detection accuracy of the model.

[0025] Compared with the prior art, the present invention designs a new module CspRepSPP, uses depthwise convolution to replace the maximum pooling in the SPPF module, and performs convolution operations on each channel of the feature map through depth convolution, which effectively solves the information loss problem existing in the original module, and further reduces the computational complexity and computing speed of the module in the reasoning process through structural reparameterization and CSP (Cross Stage Partial). Compared with SPPF, the new module CspRepSPP has a faster computing speed and greater accuracy improvement in the reasoning stage. At the same time, the present invention uses the multi-scale receptive field feature fusion idea of ​​SPP to design a new feature extraction module Improved BottleNeck, and replaces the traditional BottleNeck in YOLOv5 with this module, so that the target detection model can fuse features of receptive fields of different sizes at each stage, further improving the detection accuracy of the model without bringing additional computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This figure illustrates the information loss problem of maximum pooling in this invention. This figure shows a schematic diagram of maximum pooling for a channel in a feature map. The surrounding gray grid is filled. Darker grids represent larger pixel values. Maximum pooling only selects the pixel with the largest pixel value within the receptive field at a time.

[0027] Figure 2 This is a schematic diagram of the network structure of the YOLOv5 target detection model. The blank arrows in the figure point to the improvements of this application.

[0028] Figure 3 Figure 1. Convolution, depthwise convolution, and maximum pooling. (a) Convolution: Each pixel value on the output feature map is equal to the weighted sum of the pixel values ​​on all channels of the input feature map and the convolution kernel. (b) Depthwise convolution: The output feature map of each channel is obtained by convolution of the input feature map of the corresponding channel. (c) Maximum pooling: Similar to depthwise convolution, but without convolution, it extracts features by filtering the maximum pixel value within the receptive field.

[0029] Figure 4 This is a comparison diagram of the network structures of SPP, SPPF, RepSPP, and CspRepSPP;

[0030] Figure 5 A comparison diagram of BottlNeck and Improved BottleNeck;

[0031] Figure 6The mAP curve and loss function curve of the model change with the iteration round. The solid line represents the original model, the dashed line represents the model after replacing SPPF with CspRepSPP, and the dotted line represents the model after replacing BottleNeck with Improved BottleNeck on this basis.

[0032] Figure 7 To improve the positioning visualization results of the previous and next models, the black borders are the true labels and the white borders are the positioning results of the original model. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] This paper proposes a target detection model based on multi-scale receptive field feature fusion (such as Figure 2 (As shown in the figure). Based on the YOLOv5 object detection model, this model proposes a new multi-scale receptive field feature fusion module, CspRepSPP. This module uses deep convolution instead of maximum pooling to effectively address information loss. It also addresses the high computational complexity of deep convolution through structural reparameterization and CSP (Cross Stage Partial). Furthermore, based on the concept of multi-scale receptive field feature fusion, a new feature extraction module, Improved BottleNeck, is designed. Unlike traditional fixed receptive field feature extraction methods, this feature extraction module uses two deep convolutions with different kernel sizes to extract and fuse image features from receptive fields of two different scales.

[0035] Specifically, a target detection model based on multi-scale receptive field feature fusion includes a backbone, a neck, and a head. The backbone includes a downsampling module and a feature extraction module, and the neck performs a deeper fusion of information extracted from the backbone. The head is used to output prediction results. In one embodiment, the head performs target prediction on feature maps of three different scales using three convolutions with a kernel size of 1*1.

[0036] In one embodiment, the downsampling module is implemented by a convolution with a stride of 2, and the feature extraction module is a C3 module. The C3 module first reduces the number of channels of the input feature map by half through two convolutions with a kernel size of 1*1, and uses N BottleNeck modules on one branch to splice the result with the output of the other branch in the channel dimension, and finally outputs it through a convolution with a kernel size of 1*1. In one embodiment, CspRepSPP based on deep convolution feature extraction is used. The CspRepSPP uses deep convolution to solve the problem of maximum pooling information loss, and uses structural reparameterization and CSP to solve the problem of large deep convolution computational complexity.

[0037] In one embodiment, the CspRepSPP is constructed as follows:

[0038] 1. Please refer to Figure 3 Based on the SPPF of YOLOv5, we first use deep convolution to replace the maximum pooling, and change the channel splicing to element-wise addition, and design the following Figure 4 (c) The new module RepSPP is shown.

[0039] 2. Due to the large amount of computation required for depthwise convolution, the computation speed of RepSPP using depthwise convolution is slower than that of SPPF and SPP. Therefore, the present invention uses structural reparameterization technology to optimize convolution with multiple parallel structures and summed results, thereby improving the model's inference speed while maintaining complete model equivalence. The principle of structural reparameterization is as follows:

[0040]

[0041] Among them, M (2) 、M (1) are the output and input feature maps of each branch respectively, bn is batch normalization (as shown in Formula 2), W (i) is the convolution kernel of the i-th convolution, * represents the depth convolution operation, μ (i) , σ (i) is the mean and variance obtained after the i-th convolution, γ (i) , β (i) are two learnable variables in bn. In the present invention, since RepSPP performs 4 depthwise convolution operations, N=4.

[0042]

[0043]

[0044]

[0045] Where x(i) is the i-th feature of feature map M, and m is the total number of features. Batch normalization calculates the mean and standard deviation of the feature map using formulas (3) and (4). According to formula (2), the mean μ is subtracted from the feature map M, divided by the standard deviation σ, multiplied by the scaling factor γ, and added to the bias β, where γ and β are learnable variables. According to formula (2), formula (1) can be further simplified:

[0046]

[0047] Formula (5) first combines the convolution operation with batch normalization according to formula (2), and then combines multiple convolutions into one convolution. The convolution kernel weight of the model in the inference stage is W′ and the bias is β′.

[0048] After using structural reparameterization in the model inference phase, Figure 4 (c) Model optimization is Figure 4 (d) Model (the dotted box contains the object with structural reparameterization).

[0049] 3. In order to further reduce the computational cost of the model, the present invention Figure 4 (d) The model is CSP-ized and processed in two stages to obtain Figure 4 (e) Model CspRepSPP. CSP performs two-branch processing on the feature map, both using 1*1 convolutions to halve the number of channels. Deeper operations are performed on one branch, and the results of the two branches are finally concatenated along the channel dimension. Because the 1*1 convolution reduces the number of channels in the feature map, subsequent operations require less computation. Table 1 shows the single-image processing speed of the YOLOv5 model with the five SPP modules. Ultimately, CspRepSPP achieves slightly faster computational speed than SPPF during inference.

[0050] Table 1 Computational speed of the YOLOv5 model with five SPP modules

[0051]

[0052] The Neck part performs a deeper fusion of the information extracted from the backbone, and its structure is PANet (Path Aggregation Network), that is, it gradually splices smaller feature maps from top to bottom, and then gradually splices larger feature maps from bottom to top. The fusion of feature maps of different sizes can improve the detection accuracy of the model. In order to further improve the ability of the target detection model to fuse multi-scale receptive field features, the present invention proposes a feature extraction module Improved BottleNeck that combines structural reparameterization with deep convolution, and replaces the traditional feature extraction module BottleNeck of YOLOv5 with this module, thereby improving the detection accuracy of the model without increasing the amount of model calculation. The BottleNeck structure is as follows Figure 5 As shown in (a), it consists of a convolution with a kernel size of 1*1 and 3*3 and a residual connection. To achieve the fusion of multi-scale receptive field features, Improved BottleNeck uses a depthwise convolution with a kernel size of 7*7 and 3*3 during the training phase and adds the results. Since the depthwise convolution is only performed on each channel and there is no interaction between the information of different channels, two convolutions with a kernel size of 1*1 are used before and after the depthwise convolution to enhance the information fusion between the channels of the feature map. To ensure the robustness of the model, Improved BottleNeck also sets a residual connection. At the same time, to prevent overfitting, a regularization method Drop Path is added before the residual connection. In the inference phase, the two depthwise convolutions of Improved BottleNeck can be fused into one using structural reparameterization, further reducing the amount of computation after the model is deployed.

[0053] Table 2 Model improvements by CspRepSPP and Improved BottleNeck

[0054]

[0055] As shown in Table 2, YOLOv5+SPP and YOLOv5+SPPF have the same accuracy, but SPPF is slightly faster. YOLOv5+CspRepSPP improves both accuracy metrics by 0.64 and 1.91 percentage points compared to YOLOv5+SPP and SPPF, respectively. During inference, the model processes a single image 0.2ms faster than SPPF and 0.7ms faster than SPP, with GFLOPs (billion floating-point operations per second) comparable to those of SPP and SPPF. YOLOv5+CspRepSPP+ImprovedBottleNeck improves both accuracy metrics by 2.12 and 2.92 percentage points compared to YOLOv5+SPP and SPPF, respectively. During inference, the model processes a single image 0.5ms faster than SPPF and 1ms faster than SPP, with fewer GFLOPs. The improvement in model inference speed and the reduction in computational complexity are mainly due to the application of deep convolution and the optimization of modules through structural reparameterization.

[0056] See also Figure 6 , which plots the mAP curve of the model as the number of epochs changes, as well as the loss function curve. The mAP curve of the model on the validation set is as follows Figure 6 (a) and (b) show the classification loss curve and bounding box loss curve on the validation set. Figure 6 As shown in (c) and (d), the solid line represents the original model, the dashed line represents the model after replacing SPPF with CspRepSPP, and the dotted line represents the model after replacing BottleNeck with Improved BottleNeck. Figure 6 (a) and (b) show that both CspRepSPP and Improved BottleNeck improve the mAP of the model to a certain extent, and the mAP of the improved model improves significantly faster than that of the original model. Figure 6 (c) shows that the classification loss of the improved model is reduced to a certain extent, so the improvement improves the classification ability of the model. Figure 6 (d) It can be seen that the bounding box loss of the model after CspRepSPP replaces SPPF is greatly reduced. Therefore, the CspRepSPP module significantly improves the target positioning ability of the target detection model, and the introduction of ImprovedBottleNeck further reduces the bounding box loss of the model, thereby further improving the target positioning ability of the model.

[0057] Figure 7 Visualization of target positioning for images in the dataset validation set, with black borders representing true labels. Figure 7 (a) The white border is the positioning result of the original model. Figure 7(b) The white border is the positioning result of the improved model. Figure 7 As we know, the original model's positioning results only include the main area of ​​the target, ignoring the edge details of the target. The improved model's positioning results are closer to the true label. At the same time, the model does not ignore the detailed information of the image edge, so the positioning is more accurate. This experiment proves that multi-scale feature fusion can effectively improve the model's target positioning ability, thereby improving the model's detection accuracy.

[0058] The present invention replaces maximum pooling with deep convolution, effectively solving the information loss problem caused by maximum pooling in SPP and SPPF, and proposes a new multi-scale receptive field feature fusion module RepSPP. Since the computational complexity of deep convolution is greater than that of maximum pooling, in order to reduce the computational complexity of RepSPP, the present invention further optimizes the RepSPP module by using structural reparameterization and CSP. The new module CspRepSPP has a lower computational complexity and a faster computing speed in the inference stage. The present invention also combines structural reparameterization with deep convolution to design the feature extraction module Improved BottleNeck, and replaces the traditional feature extraction module BottleNeck of YOLOv5, thereby improving the detection accuracy of the model without increasing the computational complexity of the model.

Claims

1. An object detection model based on multi-scale receptive field feature fusion, characterized by: Using CspRepSPP based on deep convolution feature extraction, the CspRepSPP uses deep convolution to solve the problem of maximum pooling information loss, and uses structural reparameterization and CSP to solve the problem of large amount of deep convolution calculation; The construction method of the CspRepSPP is as follows: Based on the SPPF of YOLOv5, we use deep convolution to replace maximum pooling and change the channel splicing to element-wise addition to design a new module RepSPP. Structural reparameterization technology is used to optimize branches with N structures connected in parallel and the results added. The principle of structural reparameterization is as follows: Among them, M (2) 、M (1) are the output and input feature maps of each branch respectively, bn is batch normalization, W (i) is the convolution kernel of the i-th convolution, * represents the depth convolution operation, μ (i) , σ (i) is the mean and variance obtained after the i-th convolution, γ (i) , β (i) are two learnable variables in bn. Since RepSPP performs 4 depth convolution operations, N=4. Where x(i) is the i-th feature of feature map M, and m is the total number of features. Batch normalization calculates the mean and standard deviation of the feature map using formulas (3) and (4). According to formula (2), the mean μ is subtracted from the feature map M, divided by the standard deviation σ, and multiplied by the scaling factor γ, and the bias β is added, where γ and β are learnable variables. According to formula (2), formula (1) can be further simplified: Formula (5) first combines the convolution operation with batch normalization according to formula (2), and then combines multiple convolutions into one convolution. The convolution kernel weight of the model in the inference phase is W′ and the bias is β′; The optimized model was CSPified in two stages to obtain the model CspRepSPP; The Improved BottleNeck feature extraction module is designed by combining structural reparameterization with deep convolution. During the training phase, the Improved BottleNeck uses a 7*7 kernel and a 3*3 deep convolution, and adds the results to extract and fuse image features from two receptive fields of different scales.

2. The object detection model based on multi-scale receptive field feature fusion according to claim 1, characterized in that: The Improved BottleNeck also sets a residual connection, and to prevent overfitting, a regularization method Drop Path is added before the residual connection.

3. The target detection model based on multi-scale receptive field feature fusion according to claim 2, characterized in that: During inference, the two depthwise convolutions of Improved BottleNeck are fused into one using structural reparameterization.

4. The object detection model based on multi-scale receptive field feature fusion according to claim 1, characterized in that: It includes Backbone, Neck and Head parts. The Backbone part includes a downsampling module and a feature extraction operation module. The Neck part performs a deeper fusion of the information extracted from the backbone. The Head part is used to output the prediction results.

5. The object detection model based on multi-scale receptive field feature fusion according to claim 4, characterized in that: The downsampling module is implemented by convolution with a stride of 2. The feature extraction module is a C3 module. The C3 module first reduces the number of channels of the input feature map by half through two convolutions with a kernel size of 1*1, and uses N BottleNeck modules on one of the branches. The result is spliced ​​with the output of the other branch in the channel dimension, and finally output through convolution with a kernel size of 1*1.

6. The object detection model based on multi-scale receptive field feature fusion according to claim 5, characterized in that: The Neck part performs a deeper fusion of the information extracted from the backbone. Its structure is PANet (Path Aggregation Network), which gradually splices smaller feature maps from top to bottom, and then gradually splices larger feature maps from bottom to top. The fusion of feature maps of different sizes can improve the detection accuracy of the model.

Images