Multi-scale high-precision lightweight object detection method based on large receptive field and attention mechanism

Abstract

The present invention provides a multi-scale, high-precision, lightweight object detection method based on a large receptive field and an attention mechanism. The method comprises: Step 1: constructing an improved YOLOv7-Tiny network, including adding a SimAM attention mechanism to the backbone network of the YOLOv7-Tiny network; Step 2: training the improved network using a remote sensing image dataset to obtain a lightweight object detection model; Step 3: inputting the remote sensing image to be tested into the lightweight object detection model to obtain a detection result.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image target detection, and in particular to a multi-scale, high-precision, lightweight target detection method based on a large receptive field and an attention mechanism. Background Art

[0002] Object detection in remote sensing images is a key task in remote sensing image processing. It automatically extracts objects of interest, such as buildings, roads, and vehicles, from remote sensing images. In the past, manual visual interpretation was the primary method for acquiring geographic information. However, with the continuous advancement of remote sensing technology and innovations in image processing algorithms, object detection in remote sensing images has become a crucial means for efficiently acquiring large-scale geographic information. However, most object detection methods have a large number of parameters and low detection efficiency, failing to strike a good balance between detection accuracy and parameter requirements.

[0003] Many detection models based on convolutional neural networks have achieved good results. However, most target detection methods have large parameters, and the detection efficiency is low due to the large number of targets in remote sensing images. The scale of targets in remote sensing images varies greatly. Target detection in remote sensing images usually faces the following challenges: (1) Most target detectors have large parameters, which increases hardware costs. (2) Target scales vary greatly in remote sensing images, with small, medium, and large-scale targets coexisting. Existing methods mainly focus on in-depth research on small target detection in remote sensing images, but have poor detection capabilities for medium and large targets in the same remote sensing image. (3) Many targets in remote sensing images are either hidden in shadows or have a color close to the ground, making it difficult for commonly used target detection methods to detect these targets. Summary of the Invention

[0004] In order to solve at least part of the above problems, the present invention provides a multi-scale, high-precision and lightweight target detection method based on a large receptive field and an attention mechanism.

[0005] The present invention provides a multi-scale, high-precision, lightweight target detection method based on a large receptive field and an attention mechanism, comprising:

[0006] Step 1: Build an improved YOLOv7-Tiny network, including: adding the SimAM attention mechanism to the backbone network in the YOLOv7-Tiny network;

[0007] Step 2: Using a remote sensing image dataset to train the improved network to obtain a lightweight target detection model;

[0008] Step 3: Input the remote sensing image to be tested into the lightweight target detection model to obtain the detection result.

[0009] Furthermore, in step 1, the newly added SimAM attention mechanism is located between the first MaxPool layer and the second ELAN module.

[0010] Furthermore, the last two ELAN modules in the backbone network are replaced by an efficient and concise feature extraction module; the structure of the efficient and concise feature extraction module is expressed by formula (1) to formula (3):

[0011] Y1=P 3,1 (P 3,1 (F 1,1 (X))) (1)

[0012] Y2=P 3,1 ((F 1,1 (X))+σ(SimAM(X))) (2)

[0013]

[0014] Among them, X represents the input of the efficient and concise feature extraction module, Y1 and Y2 represent the outputs of the first branch and the second branch of the efficient and concise feature extraction module respectively, Y3 represents the final output of the efficient and concise feature extraction module, and F 1,1 represents a CBL module with a filter size of 1×1 and a step size of 1, P 3,1 represents a PBL module with a filter size of 3×3 and a stride of 1, SimAM represents the SimAM attention mechanism, σ is the Sigmoid activation function, This is the Concat operation.

[0015] Furthermore, the SPPCSPC module in the neck network is replaced with a spatial pyramid pooling enhancement module; the structure of the spatial pyramid pooling enhancement module includes: a first branch and a second branch;

[0016] In the first branch of the spatial pyramid pooling enhancement module, the input feature map first passes through a CBL module, then passes through three MaxPool layers in sequence, then concats the output of the CBL module and the output of the three MaxPool layers, and finally passes through another CBL module to obtain the output of the first branch;

[0017] In the second branch of the spatial pyramid pooling enhancement module, the input feature map passes through a CBL module and a PBL module respectively, and then the outputs of the two are added element by element to obtain the output of the second branch;

[0018] The outputs of the first and second branches of the spatial pyramid pooling enhancement module are concatenated in the channel dimension and then passed through a CBL module to obtain the final output of the spatial pyramid pooling enhancement module.

[0019] Furthermore, the convolution kernel sizes of the three MaxPool layers are 5, 9, and 13 respectively; the receptive field calculation formula is as follows:

[0020]

[0021] Among them, n is the nth convolution, s is its step size, r is its corresponding receptive field, and k is the convolution kernel size.

[0022] Furthermore, the last two CBL modules in the neck network are replaced by convolutional aggregation cross layers; the structure of the convolutional aggregation cross layer is expressed by formula (5):

[0023] O=SimAM(F 1,1 (AvgPool(X)))+F 3,2 (X) (5)

[0024] Among them, O and X represent the output and input of the convolutional aggregation cross layer respectively, SimAM represents the SimAM attention mechanism, and F 3,2 Represents a CBL module with a filter size of 3×3 and a stride of 2, and AvgPool represents the average pooling operation.

[0025] Beneficial effects of the present invention:

[0026] (1) The SimAM attention mechanism is embedded in the backbone network and a new feature extraction module, the efficient and concise feature extraction module, is constructed. It is embedded in the backbone network to obtain more robust features, thereby improving the feature extraction capability of the model and effectively reducing the number of model parameters.

[0027] (2) An improved spatial pyramid pooling model, the spatial pyramid pooling enhancement module, is proposed, which can better capture and fuse multi-scale feature information and enhance the detection capability of medium and large targets in remote sensing images at different scales.

[0028] (3) A convolutional aggregation cross layer is proposed, which can retain important feature information while reducing the size of the feature map and enhance the fusion of contextual information to obtain a feature map with more semantic information.

[0029] (4) Experiments on SIMD datasets show that the method of the present invention has better performance in remote sensing image target detection than other algorithms. From the ablation experiment, it can be seen that each improved module proposed in the present invention can effectively improve the detection accuracy. Therefore, the method proposed in the present invention achieves a better balance between detection accuracy and model parameter quantity for existing remote sensing target detection algorithms, and has important practical significance for the research on target detection in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a multi-scale, high-precision, lightweight object detection method based on a large receptive field and an attention mechanism provided by an embodiment of the present invention;

[0031] Figure 2 The network structure of the lightweight target detection model provided by the embodiment of the present invention;

[0032] Figure 3 A structural diagram of a VNFE provided in an embodiment of the present invention;

[0033] Figure 4 A structural diagram of the SPPE module provided in an embodiment of the present invention;

[0034] Figure 5 A structural diagram of the CACL provided in an embodiment of the present invention;

[0035] Figure 6 (a1) to Figure 6 (c6) is a detection effect diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0037] Example 1

[0038] like Figure 1 As shown, the embodiment of the present invention provides a multi-scale, high-precision, lightweight target detection method based on a large receptive field and an attention mechanism, comprising the following steps:

[0039] S101: Build an improved YOLOv7-Tiny network, including: adding the SimAM attention mechanism to the backbone network of the YOLOv7-Tiny network;

[0040] Specifically, this embodiment uses the YOLOv7-Tiny network as the basic framework, and makes improvements to the basic framework to obtain an improved YOLOv7-Tiny network. Figure 2As shown in the figure, the improved YOLOv7-Tiny network mainly consists of three parts: the backbone network (Backbone), the neck network (Neck), and the prediction head (Head). The detection process of the improved YOLOv7-Tiny network is as follows: First, the backbone network is used to extract the feature information of the input remote sensing image. In order to detect targets of different sizes, the backbone network outputs three feature maps of different sizes (128×128, 64×64, and 32×32). Then, the extracted feature maps of different sizes are sent to the neck network for feature fusion to obtain three feature maps of different scales with rich contextual information. Finally, these feature maps are sent to the prediction head for detection to obtain the final detection results.

[0041] In this embodiment, the newly added SimAM attention mechanism is located between the first MaxPool layer and the second ELAN module.

[0042] S102: training the improved network using a remote sensing image dataset to obtain a lightweight target detection model;

[0043] S103: Inputting the remote sensing image to be tested into the lightweight target detection model to obtain a detection result.

[0044] The lightweight target detection method provided by the embodiment of the present invention can accurately detect remote sensing images with a large number of targets to be detected by adding a SimAM attention mechanism to the backbone network.

[0045] Example 2

[0046] In order to further improve the detection performance of the lightweight detection model, based on the above embodiment 1, the embodiment of the present invention further improves the YOLOv7-Tiny network, including replacing the last two ELAN modules in the backbone network with the Validity-Neat Feature Extraction (VNFE) module, such as Figure 2 As shown;

[0047] Specifically, for the original YOLOv7-Tiny network model, there are too many convolutional layers and pooling layers in the backbone network, which not only leads to feature redundancy but also computational redundancy. This embodiment uses the VNFE module to replace some ELAN modules in the backbone network. In this way, more discriminative feature information can be captured in the backbone network, while also reducing the number of parameters of the entire network model.

[0048] The structure of the VNFE module is as follows Figure 3The VNFE module consists of two main branches: the first branch uses the PBL module to extract spatial features from input channels, while reducing computational redundancy and memory access to ensure the lightweightness of the network model; the second branch uses the SimAM attention mechanism to improve the feature extraction capability of the neural network without introducing additional parameters.

[0049] The structure of the VNFE module is expressed as follows:

[0050] Y1=(P 3,1 (P 3,1 (F 1,1 (X))) (1)

[0051] Y2=P 3,1 ((F 1,1 (X))+σ(SimAM(X))) (2)

[0052]

[0053] Among them, X represents the input of the efficient and concise feature extraction module, Y1 and Y2 represent the outputs of the first branch and the second branch of the efficient and concise feature extraction module respectively, Y3 represents the final output of the efficient and concise feature extraction module, and F 1,1 represents a CBL module with a filter size of 1×1 and a step size of 1, P 3,1 represents a PBL module with a filter size of 3×3 and a stride of 1, SimAM represents the SimAM attention mechanism, σ is the Sigmoid activation function, This is the Concat operation.

[0054] Example 3

[0055] In order to further improve the detection performance of the lightweight detection model, based on the above embodiments, the embodiment of the present invention further improves the YOLOv7-Tiny network, including replacing the SPPCSPC module in the neck network with a Spatial Pyramid Pooling Enforce (SPPE) module.

[0056] Specifically, the existing spatial pyramid pooling method still has room for improvement in detecting medium and large targets. In order to better enhance the model's detection ability for medium and large targets, this embodiment proposes a new SPPE module. The structure of the SPPE module is as follows: Figure 4 As shown, it includes a first branch and a second branch.

[0057] In the first branch of the spatial pyramid pooling enhancement module, the input feature map first passes through a CBL module, then passes through three MaxPool layers in sequence, then concats the output of the CBL module and the output of the three MaxPool layers, and finally passes through another CBL module to obtain the output of the first branch;

[0058] Specifically, in the first branch, pyramid layers of varying scales are constructed to capture multi-scale feature information. Pooling operations at varying scales enable the network to perceive and process objects of varying scales and sizes within the input image, thereby improving the model's adaptability and generalization capabilities. The use of pyramid layers and pooling operations not only extracts multi-scale features but also reduces the loss of spatial information.

[0059] At the same time, a larger pooling operation can preserve the overall features and spatial layout of the target object. By increasing the convolution kernel size, the model can cover more details and contextual information of the target area, which helps to more accurately detect medium and large targets. Therefore, this embodiment changes the convolution kernel size of the three-layer maximum pooling layer to 5, 9, and 13, so that its receptive field increases to 5, 13, and 25 respectively, thereby enhancing the detection ability of medium and large targets. The receptive field calculation formula is as follows:

[0060]

[0061] Among them, n is the nth convolution, s is its step size, r is its corresponding receptive field, and k is the convolution kernel size.

[0062] In the second branch of the spatial pyramid pooling enhancement module, the input feature map passes through a CBL module and a PBL module respectively, and then the outputs of the two are added element by element to obtain the output of the second branch;

[0063] Specifically, in the second branch, the input passes through CBL and PBL respectively, and the number of output channels is halved without changing the size of the feature map. The outputs of the two are then added element by element to increase the amount of information in each dimension of the feature map to supplement the feature details, thereby obtaining a richer and more comprehensive feature representation and compensating for the attenuation of small target detection ability due to the increase in receptive field.

[0064] The outputs of the first and second branches of the spatial pyramid pooling enhancement module are concatenated in the channel dimension and then passed through a CBL module to obtain the final output of the spatial pyramid pooling enhancement module.

[0065] Specifically, the results of the two branches are concatenated in the channel dimension, and then a CBL with a convolution kernel size of 1×1 is used to adjust the number of output channels of SPPE to be consistent with the input.

[0066] Example 4

[0067] To further enhance the detection performance of the lightweight detection model, based on the above embodiments, the present invention further improves the YOLOv7-Tiny network by replacing the last two CBL modules in the neck network with a Convolution Aggregation Criss Layer (CACL).

[0068] Specifically, in order to reduce the information loss caused by downsampling in Neck, this embodiment designs a new CACL to be embedded in the network, so that the feature map can retain more feature information when downsampling. The structure of CACL is as follows: Figure 5 shown.

[0069] In the first branch of CACL, using an average pooling layer, compared to a max pooling layer, better preserves spatial information in the feature map and more fully utilizes global information during feature extraction. Subsequently, 1×1 convolutions are used to deepen the network, introducing more nonlinearity without increasing the receptive field, enhancing the neural network's expressive power. The number of channels is also doubled. Next, the SimAM attention mechanism is used to focus the network model on more discriminative feature regions, improving the model's discriminability.

[0070] In the second branch of CACL, a CBL module with a convolution kernel size of 3×3 and a stride of 2 is used to downsample spatially and reduce the size of the feature map while retaining important feature information, which helps to enhance the robustness of the network model.

[0071] By summing the two branches mentioned above, CACL can integrate different feature expressions, improve the diversity and expression ability of features, and enhance the representation ability and detection performance of the network model.

[0072] The structure of CACL is expressed as follows:

[0073] O=SimAM(F 1,1 (AvgPool(X)))+F 3,2 (X) (5)

[0074] Among them, O and X represent the output and input of the convolutional aggregation cross layer respectively, SimAM represents the SimAM attention mechanism, and F 3,2 Represents a CBL module with a filter size of 3×3 and a stride of 2, and AvgPool represents the average pooling operation.

[0075] Example 5

[0076] In this embodiment, experiments are conducted using a challenging SIMD dataset to verify the effectiveness of the model proposed in the present invention.

[0077] (1) Experimental environment and training parameters:

[0078] The experimental platform is equipped with an NVIDIA GeForce RTX 3070 graphics card (8GB), an Intel Core i7-10700K CPU, and an Ubuntu 20.04.4LTS operating system. The model is developed and trained using the PyTorch deep learning framework (version 1.13.1) in Python 3.8 and CUDA 11.4.

[0079] During model training, the input image size is 1024×1024, and the Stochastic Gradient Descent (SGD) optimizer with momentum is used for training. The momentum parameter is set to 0.937, the initial learning rate is set to 0.01, the weight decay coefficient is set to 0.0005, the batch size is set to 4, and the total number of training iterations is 300.

[0080] (2) Experimental results:

[0081] The experiments on the SIMD dataset used the K-Means++ algorithm to generate new prior boxes. The experimental data after using the new prior boxes are shown in Table 1. In addition, in order to display the results more directly, a corresponding name is assigned to each category in the dataset, as shown in Table 2.

[0082] Table 1 Results of YOLOv7-Tiny with and without k-means++ algorithm

[0083] Model mAP <![CDATA[mAP 0.5:0.95 ]]> Params(m) FLOPs(G) YOLOv7-Tiny 82.3 64.2 6.05 13.3 YOLOv7-Tiny+K-Means++ 82.7 64.8 6.05 13.3

[0084] The experimental results compared with other detection methods are shown in Table 2. The method proposed in the present invention achieves the best detection results with a smaller number of parameters.

[0085] Table 2 Comparison of experimental results of the method of the present invention and other methods on SIMD data sets

[0086] Method mAP <![CDATA[mAP 0.75 ]]> <![CDATA[mAP 0.5:0.95 ]]> <![CDATA[AP S ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> Params(m) FLOPs(G) YOLOX-S 80.3 74.8 62.7 12.2 59.5 67.8 8.94 68.5 YOLOv6-N 74.4 69.0 58.5 8.2 49.3 63.1 4.7 11.4 YOLOv6-S 78.9 73.1 62.7 8.5 57.8 69.2 18.5 45.3 YOLOv8-N 81.6 - 65.9 - - - 3.01 8.2 YOLO-HR-N 83.0 - 64.0 - - - 3.34 4.4 YOLOv5-N 75.7 69.3 57.9 15.1 49.6 62.0 1.78 4.3 YOLOv5-S 82.8 74.9 64.4 11.2 61.9 68.0 7.06 16.1 YOLOv7-Tiny 82.3 75.4 64.2 19.3 59.5 70.8 6.05 13.3 Ours 84.7 78.3 66.8 20.1 61.9 72.1 5.28 12.2

[0087] From Table 2 above, we can see that:

[0088] (1) Compared with YOLOv7-Tiny, the model parameters proposed in this paper are reduced by 12.7%, mAP is improved by 2.4%, and mAP 0.75 and mAP 0.5:0.95There are also 2.9% and 2.6% improvements respectively; on medium and large targets, AP m and AP l They are 2.4% and 1.3% better than YOLOv7-Tiny respectively.

[0089] (2) Compared with YOLOv5-S, the method proposed in this paper reduces the number of parameters by 25.2%, mAP and mAP 0.5:0.95 The AP s Shanghe AP l The two aspects increased by 8.9% and 4.1% respectively.

[0090] (3) Although YOLOv5-N has an advantage in terms of the number of parameters over the proposed model, it has a significant difference in mAP and mAP. 0.5:0.95 , the present invention is 9% and 8.9% higher than them respectively; in terms of medium and large target detection, the AP of the present invention model is m 12.3% higher than that of AP l It is 10.1% higher than that.

[0091] (4) Compared with YOLOX-S and YOLOv6-S, the number of parameters is significantly reduced by 40.9% and 71.4%, respectively, while mAP is improved by 4.4% and 5.8%, respectively. 0.5:0.95 An increase of 4.1%.

[0092] (5) Although the number of parameters of YOLOv6-N and YOLOv8-N is slightly lower than that of the method proposed in this paper, the mAP of the present invention is 10.3% and 3.1% higher than that of the two methods, respectively. 0.5:0.95 They were 8.3% and 0.9% higher respectively.

[0093] In order to more intuitively demonstrate the advantages of our proposed detection method, we provide several detection results of different scene images. Figure 6 As shown, (a1)-(a6) are input images, (b1)-(b6) are the detection results of YOLOv7-Tiny, and (c1)-(c6) are the detection results of the method proposed in this invention.

[0094] Depend on Figure 6 As can be seen from the yellow boxes in (b1) and (c1), YOLOv7-Tiny does not correctly identify the vehicle target surrounded by the background, while the method proposed in this invention distinguishes it from the background and correctly detects the target. Figure 6 As can be seen from (b2)-(b4) and (c2)-(c4), the method proposed in this invention can well detect objects hidden in shadows or whose target color is similar to the ground. Figure 6As shown in the yellow boxes in (b5) and (c5), YOLOv7-Tiny did not detect the object with only partial features, while the method of the present invention accurately marked it. Figure 6 As shown in (b6) and (c6), the method proposed in the present invention also has high detection accuracy for dense small targets.

[0095] To validate the effectiveness of the proposed improvements, we conducted ablation experiments on the SIMD dataset. As shown in Table 3, we used YOLOv7-Tiny as the baseline and achieved a mAP of 84.7% on the SIMD dataset. Each of our proposed improvements significantly improved detection performance. The proposed complete detection method, integrating all improvements, achieved a 2.4% improvement in mAP compared to YOLOv7-Tiny while using 1.1M fewer parameters.

[0096] Table 3 Ablation experiments on SIMD dataset.

[0097]

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-scale, high-precision, lightweight target detection method based on a large receptive field and attention mechanism, characterized by: include: Step 1: Build an improved YOLOv7-Tiny network, including: adding a SimAM attention mechanism to the backbone network of the YOLOv7-Tiny network; the added SimAM attention mechanism is located between the first MaxPool layer and the second ELAN module; replacing the last two ELAN modules in the backbone network with efficient and concise feature extraction modules; replacing the SPPCSPC module in the neck network with a spatial pyramid pooling enhancement module; The structure of the efficient and concise feature extraction module is expressed by formula (1) to formula (3): Y1=P 3,1 (P 3,1 (F 1,1 (X))) (1) Y2=P 3,1 ((F 1,1 (X))+σ(SimAM(X))) (2) Among them, X represents the input of the efficient and concise feature extraction module, Y1 and Y2 represent the outputs of the first branch and the second branch of the efficient and concise feature extraction module respectively, Y3 represents the final output of the efficient and concise feature extraction module, and F 1,1 represents a CBL module with a filter size of 1×1 and a step size of 1, P 3,1 represents a PBL module with a filter size of 3×3 and a stride of 1, SimAM represents the SimAM attention mechanism, σ is the Sigmoid activation function, Concat operation; The structure of the spatial pyramid pooling enhancement module includes: a first branch and a second branch; In the first branch of the spatial pyramid pooling enhancement module, the input feature map first passes through a CBL module, then passes through three MaxPool layers in sequence, then concats the output of the CBL module and the output of the three MaxPool layers, and finally passes through another CBL module to obtain the output of the first branch; In the second branch of the spatial pyramid pooling enhancement module, the input feature map passes through a CBL module and a PBL module respectively, and then the outputs of the two are added element by element to obtain the output of the second branch; The outputs of the first and second branches of the spatial pyramid pooling enhancement module are concatenated in the channel dimension and then passed through a CBL module to obtain the final output of the spatial pyramid pooling enhancement module. Step 2: Using a remote sensing image dataset to train the improved network to obtain a lightweight target detection model; Step 3: Input the remote sensing image to be tested into the lightweight target detection model to obtain the detection result.

2. The multi-scale, high-precision, lightweight target detection method based on large receptive field and attention mechanism according to claim 1 is characterized in that: The convolution kernel sizes of the three MaxPool layers are 5, 9, and 13 respectively; the receptive field calculation formula is as follows: Among them, n is the nth convolution, s is its step size, r is its corresponding receptive field, and k is the convolution kernel size.

3. The multi-scale, high-precision, lightweight target detection method based on large receptive field and attention mechanism according to claim 1 is characterized in that: The last two CBL modules in the neck network are replaced by convolutional aggregation cross layers; the structure of the convolutional aggregation cross layer is expressed by formula (5): O=SimAM(F 1,1 (AvgPool(X)))+F 3,2 (X) (5) Among them, O and X represent the output and input of the convolutional aggregation cross layer, respectively. SimAM represents the SimAM attention mechanism. 3,2 Represents a CBL module with a filter size of 3×3 and a stride of 2, and AvgPool represents the average pooling operation.

Images