A remote sensing image target detection method based on YOLOv7-RS

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a remote sensing image target detection method based on YOLOv7-RS. In order to improve the accuracy of target detection in remote sensing images, the present application designs a remote sensing image target detection network based on YOLOv7-RS, redesigns the D-ELAN module in the network, fuses the SimAM attention mechanism in the backbone network, uses the SIOU loss function to replace the CIOU loss function, and optimizes the positive and negative sample allocation strategy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a remote sensing image target detection method based on YOLOv7-RS. BACKGROUND

[0002] With the continuous development of remote sensing technology, remote sensing image target detection has become an important research direction in the field of remote sensing image interpretation. Remote sensing image-based target detection is of great significance in military operations, national defense security and other fields. Improving the accuracy of target detection helps to quickly detect target information of interest in a large amount of image data and improve intelligence reconnaissance capabilities.

[0003] In recent years, the rapid development of deep learning has provided favorable technical support for remote sensing image feature extraction. Most deep learning-based target detection methods use convolutional neural networks as the backbone network. Compared with traditional manual feature extraction, convolutional neural networks have stronger feature representation capabilities because they can automatically extract high-level semantic features. At the same time, convolutional neural networks have a stronger advantage in the era of big data due to their ability to actively learn features. The rapid development of convolutional neural networks has solved many problems in the field of computer vision and achieved great success in the field of image target detection. However, due to the characteristics of multi-scale, multiple rotation angles, complex scenes and other characteristics of remote sensing image targets, deep learning still faces great challenges in the application of remote sensing image target detection under the condition of limited high-quality labeled samples.

[0004] Current deep learning-based target detection algorithms are mainly divided into two-stage target detection and one-stage target detection. The YOLO series algorithm is a typical one-stage target detection algorithm. YOLOv1 was first proposed in 2015, effectively solving the problem of slow inference speed of two-stage detection networks. YOLOv2 improved in terms of faster, more and more accurate, and the recognized objects were expanded to 9000, so it was also called YOLO9000. YOLOv3 introduced the feature pyramid FPN and the residual module Darknet-53, supporting the detection of three different scales of objects, realizing the fusion of multi-scale. YOLOv4 and YOLOv5 combined the popular technologies at the time such as weighted residual connection (WRC), cross-stage partial connection (CSP), Mosaic data augmentation, further improving the detection accuracy and speed. YOLOX combined Anchor-Free network to replace the decoupled detection head of YOLOv5, improving the convergence speed of the network, in addition to this, it also proposed a positive and negative sample matching strategy SimOTA based on OTA. YOLOv6 is a target detection framework developed and optimized by the visual intelligence department of Meituan, and has been widely applied in the industry. YOLOv7 was born in July 2022, aiming at network performance, proposing E-ELAN architecture and auxiliary training module, further improving the speed and accuracy of the algorithm.

[0005] Existing research work has achieved many results and progress, but there are still problems that need further research and solution: the target detection algorithm based on YOLO performs well in natural images, but due to the different imaging methods of remote sensing images and natural images, the remote sensing image background is complex and diverse, the remote sensing target has the characteristics of rotation invariance and large scale difference, which makes the remote sensing image target detection based on YOLO poor. SUMMARY

[0006] The purpose of the present application is to solve the problem of poor target detection accuracy of remote sensing images, and a remote sensing image target detection method based on YOLOv7-RS is proposed, and a remote sensing image target detection network based on YOLOv7-RS is designed, which can efficiently solve the problems existing in remote sensing image target detection.

[0007] In order to achieve the above purpose, the following technical scheme is adopted in the present application:

[0008] A remote sensing image target detection method based on YOLOv7-RS, comprising the following steps:

[0009] Step 1, obtaining a remote sensing image and pre-processing the remote sensing image;

[0010] Step 2, based on the YOLOv7-RS network structure, constructing a remote sensing image target detection model;

[0011] Step 3, inputting the pre-processed remote sensing image together with the weight file into the constructed model to detect the target of the remote sensing image.

[0012] Further, the remote sensing image is pre-processed in step 1, specifically: the obtained remote sensing image is scaled to 640X640 size, and the insufficient part is supplemented by pixel padding.

[0013] Further, the YOLOv7-RS network structure in step 1 includes a D-ELAN module, an SIOU loss function part, and an input (Input) stage, a backbone network (Backbone) stage, a neck network (Neck) stage, and a head network (Head) stage.

[0014] The D-ELAN module is redesigned according to the segmentation gradient flow idea of CSPNet: that is, the first branch directly passes through a 1x1 convolution; the second branch passes through three groups of two 3x3 convolutions on this basis, and finally the 1x1 convolution and the three groups of 3x3 convolution results are spliced, the feature extraction capability is improved by improving the block utilization and increasing the network depth.

[0015] Further, in the backbone network stage, a three-dimensional attention module SimAM is introduced to fuse channel attention and spatial attention, and the calculation formula is as follows:

[0016]

[0017] wherein, is the output feature, X is the input feature, E represents the minimum energy function of all neurons in the channel and space Energy tensor of, single neuron minimum energy function As shown in formula (2):

[0018]

[0019] wherein, t is the target neuron, λ is a hyperparameter, is the average value of all neurons in a single channel, is the variance of all neurons in a single channel, as shown in formulas (3) and (4):

[0020]

[0021]

[0022] wherein, M represents the number of neurons in each channel, X i represents the i-th neuron of the input feature map in a single channel.

[0023] Further, in the SIOU loss function, the angle deviation between the real box and the predicted box is defined as the angle loss, and the distance loss is added to the calculation, that is, the SIOU loss function is composed of four parts of angle loss (Angle cost), distance loss (Distance cost), shape loss (Shape cost) and IOU loss, and the calculation formula is as follows:

[0024]

[0025] wherein, IOU is the IOU loss, Δ is the distance loss, Ω is the shape loss, and the calculation formulas of the three are as follows:

[0026]

[0027]

[0028]

[0029]

[0030] wherein, Λ is the angle loss, C h and Cw respectively, the height and width of the minimum bounding rectangle of the real and predicted boxes, γ is assigned as a time-limited distance value, ρ x is the proportion of the difference between the width of the real box and the width of the predicted box in C w , ρ y is the proportion of the difference between the height of the real box and the height of the predicted box in C h , x gt and y gt are the horizontal and vertical coordinates of the center point of the real box, x and y are the horizontal and vertical coordinates of the center point of the predicted box, σ is the distance between the center points of the real and predicted boxes, w gt and h gt are the width and height of the real box, w and h are the width and height of the predicted box, ω w is the proportion of the difference between the width of the real box and the width of the predicted box in the maximum value of the two, ω h is the proportion of the difference between the height of the real box and the height of the predicted box in the maximum value of the two, θ is the control of the attention degree to the shape loss.

[0031] Further, only positive samples participate in the calculation of the loss function in the SIOU loss function.

[0032] Further, the SIOU loss function optimizes the positive and negative sample allocation strategy, that is, on the basis of the positive and negative sample allocation strategy of YOLOv7, the rotation invariance of remote sensing image targets is considered, and three positive sample candidate boxes are increased to four positive sample candidate boxes.

[0033] Compared with the prior art, the present application has the following advantages:

[0034] (1) In order to improve the deficiency of YOLOv7 network in extracting remote sensing image features, the D-ELAN module is redesigned.

[0035] (2) In order to reduce the interference of background noise in remote sensing images, the SimAM attention mechanism is fused in the YOLOv7 network, so that the network can pay attention to more valuable information in remote sensing images.

[0036] (3) In order to improve the convergence speed of the network, the SIOU loss function is used to replace the CIOU loss function.

[0037] (4) In order to improve the problem of missed detection when small targets are densely arranged in remote sensing images, the positive and negative sample allocation strategy is optimized.

[0038] (5) The YOLOv7-RS proposed in the present application is superior to most existing methods, and has competitive detection ability on NWPU VHR-10 dataset and DOTA dataset, which can better adapt to the complexity and diversity of remote sensing images, indicating the effectiveness of the method. Attached Figure Description

[0039] Figure 1 This is a flowchart of the remote sensing image target detection method based on YOLOv7-RS of the present invention;

[0040] Figure 2 This is a network structure diagram of the remote sensing image target detection method based on YOLOv7-RS of this invention;

[0041] Figure 3 This is a comparison of the structural diagrams of the D-ELAN module of this invention and the ELAN module in YOLOv7;

[0042] Figure 4 This is a schematic diagram of the parameters of the ground truth bounding box and the predicted bounding box in SIOU of the present invention;

[0043] Figure 5 This is a schematic diagram illustrating the positive and negative sample strategy optimization;

[0044] Figure 6 This is a comparison chart of the visualization results of the NWPU VHR-10 dataset;

[0045] Figure 7 This is a comparison chart of the visualization results of the DOTA dataset. Detailed Implementation

[0046] To facilitate understanding of the present invention, a more comprehensive description will be provided below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the present invention.

[0047] Example 1

[0048] like Figure 1 As shown, a remote sensing image target detection method based on YOLOv7-RS according to the present invention includes the following steps:

[0049] Step 1: Acquire remote sensing images and preprocess them, i.e., scale the images to 640x640.

[0050] Step 2, based on the YOLOv7-RS network structure (e.g.) Figure 2 As shown), a remote sensing image target detection model is constructed. The YOLOv7-RS network structure includes a D-ELAN module, a SIOU loss function part and an input stage, a backbone network stage, a neck network stage, and a head network stage.

[0051] Wherein the D-ELAN (Deeply-ELAN) module is re-designed according to the segmentation gradient flow idea of CSPNet: the first branch directly passes through a 1x1 convolution; the second branch passes through three groups of two 3x3 convolutions on this basis, and finally the 1x1 convolution and the three groups of 3x3 convolution results are spliced, the feature extraction capability is improved by improving the block utilization and increasing the network depth. The structure diagram comparison of D-ELAN module and ELAN module is shown as Figure 3

[0052] In the backbone network stage, a three-dimensional attention module SimAM that fuses channel attention and spatial attention is introduced, and the calculation formula is as follows:

[0053]

[0054] Wherein, is the output feature, X is the input feature, E represents the energy tensor of all neurons in the channel and space The minimum energy function of a single neuron is As shown in formula (2):

[0055]

[0056] Wherein, t is the target neuron, λ is a hyperparameter, is the average value of all neurons in a single channel, is the variance of all neurons in a single channel, as shown in formulas (3) and (4):

[0057]

[0058]

[0059] Wherein, M represents the number of neurons in each channel, X i represents the i-th neuron in a single channel of the input feature map.

[0060] In the SIOU loss function, the angle deviation between the real box and the predicted box is defined as the angle loss, and the distance loss is added to the calculation, that is, the SIOU loss function is composed of angle loss, distance loss, shape loss and IOU loss Four parts, the calculation formula is as follows:

[0061]

[0062] Wherein, IOU is the IOU loss, Δ is the distance loss, Ω is the shape loss, and the calculation formulas of the three are as follows:

[0063] ​

[0064]

[0065]

[0066]

[0067] Where Λ represents the angle loss, and C h and C w These are the height and width of the minimum bounding rectangles of the ground truth bounding box and the predicted bounding box, respectively. γ is assigned a time-finite distance value, and ρ... x The difference between the width of the ground truth bounding box and the width of the predicted bounding box in C w The specific gravity in, ρ y The difference between the height of the ground truth bounding box and the height of the predicted bounding box in C h The proportion of x gt and y gt , where x and y are the x and y coordinates of the center point of the ground truth bounding box, respectively; σ is the distance between the center points of the ground truth bounding box and the center point of the predicted bounding box, respectively; w gt and h gt Let w and h be the width and height of the ground truth bounding box, and ω be the width and height of the predicted bounding box. w ω represents the proportion of the difference between the width of the ground truth bounding box and the width of the predicted bounding box in the maximum value of the two. h θ represents the proportion of the difference between the height of the ground truth bounding box and the height of the predicted bounding box in the maximum value of the two. θ controls the degree of attention paid to shape loss, and the parameter range is [2,6]. A schematic diagram of each parameter is shown below. Figure 4 As shown, the lower left corner is the predicted bounding box, and the upper right corner is the ground truth bounding box.

[0068] In the SIOU loss function, only positive samples participate in the loss function calculation. The problem of missed detection when small targets are densely packed in remote sensing images is improved by optimizing the positive and negative sample allocation strategy. The optimized positive and negative sample allocation strategy is based on the positive and negative sample allocation strategy of YOLOv7, taking into account the rotation invariance of remote sensing targets, increasing the number of positive candidate boxes from three to four. Figure 5 As shown, calculations show that when the remote sensing image is rotated by 45°, the positive sample loss rate is reduced from 46% to 28%.

[0069] Step 3: Input the preprocessed remote sensing image along with the weight file into the constructed model to perform target detection in the remote sensing image. In this embodiment, the weight file is the best weight file obtained by iteratively training the NWPU VHR-10 dataset and the DOTA dataset for 300 epochs.

[0070] Example 2

[0071] The remote sensing image target detection network model proposed in the application is applied to the NWPU VHR-10 dataset, and experiments show the effectiveness of the network model.

[0072] The NWPU VHR-10 dataset is published by Northwest Polytechnic University in 2014, and the images are extracted from Google Earth and Vaihingen, including 10 categories of airplanes (PL), ships (SH), storage tanks (ST), baseball fields (BD), tennis courts (TC), basketball courts (BC), ground runways (GTF), ports (HA), bridges (BR) and vehicles (VE), 800 remote sensing images (including 150 background images). The data annotation adopts the HBB (Horizontal Bounding Boxes) annotation format, and there are a total of 3651 instances. 90% of the data set is randomly divided as the training set, and 10% is randomly divided as the test set.

[0073] In the target detection task, mAP (Mean Average Precision) is usually used to measure the overall performance of the model; mAP is the average value of the average precision AP (Avanrage Precision) of multiple categories in the data set; each category can draw a curve in the coordinates of 0 to 1 according to Precision and Recall, and the area surrounded by the curve and the coordinate axis is the average precision, as shown in formula (10):

[0074]

[0075] Wherein, the accuracy rate Precision represents the proportion of TP in the predicted positive samples in the detector, as shown in formula (11); the recall rate Recall represents the proportion of correctly predicted positive samples in the total sample quantity in the detector, as shown in formula (12):

[0076]

[0077]

[0078] In the formula, TP is a true example, FN is a false negative example, and FP is a false positive example.

[0079] The yolov7-RS proposed in the application is compared with SSD, Faster R-CNN, YOLOv3, YOLOv5s and YOLOv7 algorithms on the NWPU VHR-10 dataset, and the results are shown in Table 1.

[0080] Table 1 Experimental results of different algorithms on the NWPU VHR-10 dataset

[0081]

[0082] From Table 1, compared with SSD, Faster R-CNN, YOLOv3, YOLOv4, YOLOv5s, YOLOv7, the mAP of YOLOv7-RS is increased by 14.3%, 10.9%, 20.3%, 6.3%, 5.3%, and 2.6%, respectively. The detection accuracy of YOLOv7-RS in each category is above 89%, and the overall detection accuracy is good. In the detection of aircraft (PL) and storage tank (ST), the accuracy is optimal, reaching 99.6%, compared with other algorithms. Compared with the original YOLOv7, the detection accuracy of aircraft (PL), storage tank (ST), tennis court (TC), basketball court (BC), and vehicle (VE) is improved.

[0083] Through a large number of experiments, the detection results of YOLOv7 and YOLOv7-RS are compared, and two groups of visual results are selected for analysis. As shown in FIG. 6, the left side is the detection result of the YOLOv7 algorithm, and the right side is the detection result of the YOLOv7-RS algorithm. Figure 6

[0084] In FIG. 6(a), a yellow landmark is mistakenly detected as an aircraft, Figure 6 In FIG. 6(b), a bridge is missed. YOLOv7-RS can accurately detect the target, and it can be seen that YOLOv7-RS effectively improves the detection effect in a complex background. Figure 6

[0085] Example 3

[0086] The remote sensing image target detection network model proposed in the application is applied to the DOTA dataset, and experiments show that the network model is effective.

[0087] ​DOTAv1.0 dataset is from Google Earth, GF-2 and JL-1 satellite images provided by China Resource Satellite Data and Application Center, and aerial images provided by CycloMedia B.V, including 15 categories of airplanes (PL), ships (SH), small vehicles (SV), large vehicles (LV), oil tanks (ST), tennis courts (TC), playground tracks (GTF), bridges (BR), loops (RA), swimming pools (SP), baseball fields (BD), basketball courts (BC), ports (HA), helicopters (HC) and football fields (SBF), 2806 aerial images from different sensors and platforms, image sizes ranging from 800x800 to 4000x4000, a total of 188282 instances. In this embodiment, the DOTA_devkit is used to preprocess the dataset with HBB labeling method, and the original image is cropped into a sub-image with a size of 1024x1024 and an overlapping pixel of 200. The image with a resolution that does not meet the specified pixel is filled by pixel padding. The processed training set has 15749 images, and the test set has 5297 images.

[0088] YOLOv7-RS proposed in the present application is compared with SSD, Faster R-CNN, YOLOv3, YOLOv5s and YOLOv7 algorithms on NWPU VHR-10 dataset, and the results are shown in Table 2.

[0089] Table 2 Comparison of results of different models on DOTA dataset

[0090]

[0091] As shown in Table 2, compared with SSD, Faster R-CNN, YOLOv3, YOLOv4, YOLOv5s and YOLOv7, the mAP of YOLOv7-RS is increased by 21.7%, 32.1%, 9.6%, 5.7%, 4.6% and 2.4% respectively. The detection accuracy of YOLOv7-RS on baseball field (BD), bridge (BR), large vehicle (LV), football field (SBF) and loop (RA) is the best among the other algorithms; compared with the original YOLOv7, except that the detection accuracy of tennis court (TC), basketball court (BC) and oil tank (ST) is decreased by 0.1%-0.2%, the rest is obviously improved.

[0092] The detection results of YOLOv7 and YOLOv7-RS are compared through a large number of experiments, and two groups of visual results are selected for analysis, as shown in Figure 7 The left side is the detection result of YOLOv7 algorithm, and the right side is the detection result of YOLOv7-RS algorithm.

[0093] YOLOv7 in Figure 7 (a) detected 5 ports, Figure 7 (b) detected 153 small cars and 3 large cars.YOLOv7-RS in Figure 7 (a) detected 5 ports and 6 small cars, and in Figure 7 (b) detected 272 small cars and 4 large cars, which shows that YOLOv7-RS effectively improves the missed detection problem under the condition of complex background and small target dense arrangement.

[0094] In summary, the YOLOv7-RS proposed in the application is superior to most existing methods, and the mAP on NWPU VHR-10 and DOTA datasets reaches 95.4% and 74.1%, which can better adapt to the complexity and diversity of remote sensing images, indicating the effectiveness of the method.

Claims

1. A YOLOv7-RS-based remote sensing image target detection method, characterized in that, The method comprises the following steps: Step 1, obtaining a remote sensing image and pre-processing the remote sensing image; Step 2, constructing a remote sensing image target detection model based on a YOLOv7-RS network structure; the YOLOv7-RS network structure comprises a D-ELAN module, an SIOU loss function part and an input end stage, a backbone network stage, a neck network stage and a head network stage; a first branch of the D-ELAN module directly passes through a 1x1 convolution; a second branch passes through three groups of two 3x3 convolutions on this basis; finally, the results of the 1x1 convolution and the three groups of 3x3 convolutions are spliced, the feature extraction capability is improved by improving the block utilization rate and increasing the network depth; In the backbone network stage, a three-dimensional attention module SimAM that fuses channel attention and spatial attention is introduced, and the calculation formula is as follows: （1） where, is the output feature, is the input feature, E denotes the energy tensor of all neurons in the channel and space, and the minimum energy function of a single neuron is the energy tensor of all neurons in the channel and space, and the minimum energy function of a single neuron as shown in formula (2): （2） where t is the target neuron, and λ is a hyperparameter, is the average of all neurons over a single channel, is the variance of all neurons over a single channel, as shown in equations (3) and (4): （3） （4） wherein M represents the number of neurons per channel, X i represents the i-th neuron of the input feature map on a single channel; Step 3, inputting the pre-processed remote sensing image together with a weight file into the constructed model to perform remote sensing image target detection.

2. The YOLOv7-RS-based remote sensing image target detection method according to claim 1, characterized in that, In step 1, the remote sensing image is pre-processed, specifically: the obtained remote sensing image is scaled to 640X640 size, and the insufficient part is supplemented by pixel padding.

3. The YOLOv7-RS-based remote sensing image target detection method according to claim 1, characterized in that, In the SIOU loss function, the angle deviation between the real box and the predicted box is defined as the angle loss, and the distance loss is added to the calculation, that is, the SIOU loss function is composed of four parts of angle loss, distance loss, shape loss and IOU loss, and the calculation formula is as follows: （5） Wherein, IOU is the IOU loss, ∆ is the distance loss, and Ω is the shape loss, and the calculation formulas of the three are as follows: （6） （7） （8） （9） where Λ is the angle loss, and are the height and width of the minimum bounding rectangle of the real and predicted boxes respectively, γ is assigned as a time-limited distance value, ρ x is the proportion of the difference between the width of the real box and the width of the predicted box in , ρ y is the proportion of the difference between the height of the real box and the height of the predicted box in , and are the horizontal and vertical coordinates of the center point of the real box, x, y are the horizontal and vertical coordinates of the center point of the predicted box, is the distance between the center points of the real and predicted boxes, is the distance between the center points of the real and predicted boxes, and are the width and height of the real box, w, h are the width and height of the predicted box, ω w is the proportion of the difference between the width of the real box and the width of the predicted box in the maximum value of the two, ω h is the proportion of the difference between the height of the real box and the height of the predicted box in the maximum value of the two, is the control of the degree of attention to shape loss.

4. The YOLOv7-RS-based remote sensing image target detection method according to claim 1, characterized in that, Only positive samples participate in the calculation of the loss function in the SIOU loss function.

5. The YOLOv7-RS-based remote sensing image target detection method according to claim 4, characterized in that, In the SIOU loss function, the positive and negative sample allocation strategy is optimized, that is, on the basis of the positive and negative sample allocation strategy of YOLOv7, the rotation invariance of the remote sensing image target is considered, and three positive sample candidate boxes are increased to four positive sample candidate boxes.

Images