Remote sensing image small target detection method and device, medium and product

Abstract

The invention discloses a remote sensing image small target detection method and device, a medium and a product, and relates to the field of remote sensing image application, an SCW-YOLO model used for remote sensing image small target detection is obtained through improvement on the basis of a YOLOv8 model, and the improvement comprises the steps of removing a large object detection head and adding a small object detection head on the basis of the YOLOv8 model, a feature fusion network of a neck structure of the YOLOv8 model is optimized by adopting a bidirectional weighted feature pyramid network structure so as to fuse feature information directly generated from a backbone network, a coordinate attention mechanism is added in a second C2f module and a third C2f module in the backbone network of the YOLOv8 model, and a loss function of the YOLOv8 model is optimized by adopting a Wise IOU method. The detection accuracy of small targets such as automobiles, airplanes, ships and oil storage tanks in the remote sensing images can be improved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing image applications, and in particular to a remote sensing image small target detection method, equipment, medium and product. Background Art

[0002] With the continuous advancement of remote sensing satellite technology, the amount of information in high-resolution remote sensing images (i.e. remote sensing images) is increasing, and the detailed information they contain is becoming more and more abundant. In recent years, with the development of deep learning, it has been widely used in remote sensing image target detection due to its ability to effectively extract information from massive data.

[0003] According to the different stages of target detection, target detection methods are divided into one-stage detection algorithms and two-stage detection algorithms. The two-stage detection method (i.e., the two-stage detection algorithm) is based on the deep convolutional network of candidate areas: first, a candidate area containing the detection target is generated, and then the deep convolutional network is used to extract features, and then classified and corrected to achieve target detection. The more classic algorithms include R-CNN, Fast R-CNN, Faster R-CNN, etc. Although the two-stage algorithm (i.e., the two-stage detection algorithm) has high detection accuracy, it has a slow detection speed and a large number of parameters, which is not conducive to application deployment. The one-stage detection method (i.e., the one-stage detection algorithm) is a target detection algorithm based on a regression deep convolutional neural network, using an end-to-end target detection method, the most typical of which are SDD, YOLO series, etc. These methods have fast detection speed, small model calculation amount, and can achieve fast detection.

[0004] At present, the YOLOv8 model is already one of the most advanced target detection algorithms, and has achieved good results in ordinary natural optical images. However, in comparison, in the detection of small targets such as cars, ships, airplanes, and oil tanks in remote sensing images, the target scale accounts for a small proportion of the image and is easily affected by complex environmental factors. In addition, the targets in remote sensing images are usually densely arranged, which makes it difficult to separate the target from the background, increases the difficulty of feature extraction, and leads to low detection accuracy. Therefore, it is very necessary to further improve the detection effect of deep learning technology in remote sensing images, especially to improve the detection accuracy of small targets such as cars, airplanes, ships, and oil tanks in remote sensing images. Summary of the invention

[0005] The purpose of this application is to provide a remote sensing image small target detection method, equipment, medium and product to improve the detection accuracy of small targets such as cars, airplanes, ships and oil tanks in remote sensing images.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a remote sensing image small target detection method, the remote sensing image small target detection method comprising:

[0008] Acquire a model training data set; the model training data set includes a plurality of training remote sensing images marked with positions and categories of small targets; the small targets include small cars, large cars, airplanes, ships, and oil storage tanks;

[0009] Take the YOLOv8 model as the baseline network. On the basis of the YOLOv8 model, remove the large object detection head of the YOLOv8 model, add the small object detection head, change the connection between the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure to the connection between the first C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure, change the connection between the third C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure to the connection between the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure, remove The fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network are optimized by using a bidirectional weighted feature pyramid network structure to optimize the feature fusion network of the YOLOv8 model neck structure to fuse the feature information directly generated from the backbone network. A coordinate attention mechanism is added to the second C2f module and the third C2f module in the YOLOv8 model backbone network. The WiseIOU method is used to optimize the loss function of the YOLOv8 model to obtain the SCW-YOLO model; wherein the large object detection head is a detection head of size 20×20; and the small object detection head is a detection head of size 160×160;

[0010] Using the model training data set to train and optimize the SCW-YOLO model to obtain an optimized SCW-YOLO model;

[0011] The remote sensing image to be detected is input into the optimized SCW-YOLO model, and the position and category of the small target in the remote sensing image to be detected are obtained by using the optimized SCW-YOLO model.

[0012] Optionally, before acquiring the model training data set, the remote sensing image small target detection method further includes:

[0013] Get the DOTA dataset;

[0014] Extracting remote sensing images containing small automobile targets, remote sensing images containing large automobile targets, remote sensing images containing aircraft targets, remote sensing images containing ship targets, and remote sensing images containing oil tank targets in the DOTA data set; the area of ​​small targets in the extracted remote sensing images is less than 1%;

[0015] The extracted remote sensing images are cropped to a uniform size of 1024×1024 to obtain a plurality of training remote sensing images;

[0016] Each of the training remote sensing images is labeled with examples to obtain a plurality of training remote sensing images labeled with positions and categories of small targets;

[0017] A model training data set is created using multiple training remote sensing images marked with the locations and categories of small targets.

[0018] Optionally, removing the large object detection head of the YOLOv8 model and adding a small object detection head specifically includes:

[0019] Remove the 20×20 detection head in the YOLOv8 model, add a 160×160 detection head in the YOLOv8 model, modify the connection relationship between the YOLOv8 model neck structure and the 80×80 detection head and the 40×40 detection head in the YOLOv8 model, connect the C2f module connected to the 80×80 detection head in the YOLOv8 model neck structure to the 160×160 detection head, connect the C2f module connected to the 40×40 detection head in the YOLOv8 model neck structure to the 80×80 detection head, and connect the C2f module connected to the 20×20 detection head in the YOLOv8 model neck structure to the 40×40 detection head.

[0020] Optionally, removing the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network specifically includes:

[0021] Remove the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network, and connect the third C2f module in the YOLOv8 model backbone network to the spatial pyramid pooling module in the YOLOv8 model backbone network.

[0022] Optionally, the adopting of a bidirectional weighted feature pyramid network structure to optimize the feature fusion network of the neck structure of the YOLOv8 model to fuse feature information directly generated from the backbone network specifically includes:

[0023] A bidirectional weighted feature pyramid network structure is used to optimize the feature fusion network of the neck structure of the YOLOv8 model. A new connection established by the bidirectional weighted feature pyramid network structure is used to enable the feature fusion network of the neck structure of the YOLOv8 model to fuse feature information directly generated from the backbone network. The new connection is a connection between the second C2f module in the backbone network of the YOLOv8 model and the third fusion module in the neck structure of the YOLOv8 model. The third fusion module in the neck structure of the YOLOv8 model is a fusion module connected to the first convolution module in the neck structure of the YOLOv8 model.

[0024] Optionally, adding a coordinate attention mechanism to the second C2f module and the third C2f module in the YOLOv8 model backbone network specifically includes:

[0025] After the feature information is fused through residual convolution in the second C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added. At the same time, after the feature information is fused through residual convolution in the third C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added.

[0026] Optionally, the Wise IOU method is used to optimize the loss function of the YOLOv8 model, specifically including:

[0027] The Wise IOU loss function is used to replace the CIOU loss function of the YOLOv8 model.

[0028] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the remote sensing image small target detection methods described above.

[0029] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the remote sensing image small target detection methods described above.

[0030] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the remote sensing image small target detection methods described above.

[0031] According to the specific embodiments provided in this application, this application has the following technical effects:

[0032] The present application provides a method, device, medium and product for detecting small targets in remote sensing images. The SCW-YOLO model with high detection accuracy is obtained by improving the YOLOv8 model. The large object detection head is removed and the small object detection head is added on the basis of the YOLOv8 model. The feature extraction with higher resolution is realized based on the small object detection head, and the local and detail features are not lost. The feature information can be effectively extracted from the shallow network to generate more discriminative features. The feature fusion network of the neck structure of the YOLOv8 model is optimized through the bidirectional weighted feature pyramid network structure, and the feature information (i.e., feature map) directly generated from the backbone network is effectively fused, so that the feature information generated in the middle of the backbone network is directly fused into the feature fusion network of the neck structure of the YOLOv8 model, and the loss of some information is prevented. The coordinate attention mechanism is added to the second C2f module and the third C2f module in the backbone network of the YOLOv8 model, so the local incremental feature extraction capability of the backbone network is enhanced. The IOU method optimizes the loss function of the YOLOv8 model, solves the problem of poor bounding box fitting ability of the CIOU loss function in the YOLOv8 model, can better detect occlusions and small objects, and uses the SCW-YOLO model with high detection accuracy to detect small targets in remote sensing images, thereby improving the detection accuracy of small targets such as cars, airplanes, ships, and oil tanks in remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a process for detecting small targets in remote sensing images provided in one embodiment of the present application;

[0035] Figure 2 A flowchart of the steps provided for this application;

[0036] Figure 3 Schematic diagram of the YOLOv8 model provided for this application;

[0037] Figure 4 Schematic diagram of the SCW-YOLO model provided for this application;

[0038] Figure 5 A schematic diagram for comparing the model structure improvements provided for this application;

[0039] Figure 6Schematic diagram of the C2f module structure of the model provided for this application;

[0040] Figure 7 The CA-C2f module structure diagram of the model provided for this application;

[0041] Figure 8 The coordinate attention module structure diagram provided for this application;

[0042] Fig. 9 Schematic diagram of position regression based on IOU provided for this application;

[0043] Fig.10 The accuracy comparison chart of YOLOv8 and SCW-YOLO provided for this application;

[0044] Fig.11 Comparison chart of YOLOv8 and SCW-YOLO prediction results provided for this application;

[0045] Fig.12 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The purpose of this application is to provide a remote sensing image small target detection method, equipment, medium and product to improve the detection accuracy of small targets such as cars, airplanes, ships and oil tanks in remote sensing images.

[0048] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0049] like Figure 1 As shown, the present application provides a remote sensing image small target detection method, comprising:

[0050] Step 101: Obtain a model training data set; the model training data set includes a plurality of training remote sensing images with the positions and categories of small targets marked; the small targets include small cars, large cars, airplanes, ships, and oil storage tanks.

[0051] Before step 101, the following steps are also included:

[0052] Get the DOTA dataset.

[0053] Remote sensing images containing small car targets, remote sensing images containing large car targets, remote sensing images containing aircraft targets, remote sensing images containing ship targets, and remote sensing images containing oil tank targets are extracted from the DOTA data set; the area of ​​small targets in the extracted remote sensing images is less than 1%.

[0054] Each extracted remote sensing image is cropped to a uniform size of 1024×1024 to obtain multiple training remote sensing images.

[0055] Each training remote sensing image is instance-labeled to obtain multiple training remote sensing images labeled with the location and category of small targets.

[0056] A model training data set is created using multiple training remote sensing images marked with the locations and categories of small targets.

[0057] Step 102: Take the YOLOv8 model as the baseline network. On the basis of the YOLOv8 model, remove the large object detection head of the YOLOv8 model, add the small object detection head, modify the connection between the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure to the first C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure, modify the connection between the third C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure to the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure. The fifth convolution module and the fourth C2f module in the backbone network of the YOLOv8 model are connected, and the feature fusion network of the neck structure of the YOLOv8 model is optimized using a bidirectional weighted feature pyramid network structure to fuse the feature information directly generated from the backbone network. The coordinate attention mechanism is added to the second C2f module and the third C2f module in the backbone network of the YOLOv8 model. The WiseIOU method is used to optimize the loss function of the YOLOv8 model to obtain the SCW-YOLO model; among them, the large object detection head is a detection head with a size of 20×20; the small object detection head is a detection head with a size of 160×160.

[0058] In step 102, the large object detection head of the YOLOv8 model is removed, and the small object detection head is added, which specifically includes: removing the detection head of size 20×20 in the YOLOv8 model, adding a detection head of size 160×160 in the YOLOv8 model, modifying the connection relationship between the YOLOv8 model neck structure and the detection head of size 80×80 and the detection head of size 40×40 in the YOLOv8 model, connecting the C2f module connected to the detection head of size 80×80 in the YOLOv8 model neck structure to the detection head of size 160×160, connecting the C2f module connected to the detection head of size 40×40 in the YOLOv8 model neck structure to the detection head of size 80×80, and connecting the C2f module connected to the detection head of size 20×20 in the YOLOv8 model neck structure to the detection head of size 40×40.

[0059] In step 102, the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network are removed, specifically including: removing the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network, and connecting the third C2f module in the YOLOv8 model backbone network to the spatial pyramid pooling module in the YOLOv8 model backbone network.

[0060] In step 102, a bidirectional weighted feature pyramid network structure is used to optimize the feature fusion network of the neck structure of the YOLOv8 model to fuse the feature information directly generated from the backbone network, specifically including: optimizing the feature fusion network of the neck structure of the YOLOv8 model by using a bidirectional weighted feature pyramid network structure, and using a new connection established by the bidirectional weighted feature pyramid network structure to enable the feature fusion network of the neck structure of the YOLOv8 model to fuse the feature information directly generated from the backbone network; the new connection is a connection between the second C2f module in the backbone network of the YOLOv8 model and the third fusion module in the neck structure of the YOLOv8 model; the third fusion module in the neck structure of the YOLOv8 model is a fusion module connected to the first convolution module in the neck structure of the YOLOv8 model.

[0061] In step 102, a coordinate attention mechanism is added to the second C2f module and the third C2f module in the YOLOv8 model backbone network, specifically including: after the feature information is fused through residual convolution in the second C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added; at the same time, after the feature information is fused through residual convolution in the third C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added.

[0062] In step 102, the Wise IOU method is used to optimize the loss function of the YOLOv8 model, specifically including: using the Wise IOU loss function to replace the CIOU loss function of the YOLOv8 model.

[0063] Step 103: Train and optimize the SCW-YOLO model using the model training data set to obtain an optimized SCW-YOLO model.

[0064] Step 104: input the remote sensing image to be detected into the optimized SCW-YOLO model, and use the optimized SCW-YOLO model to obtain the position and category of the small target in the remote sensing image to be detected.

[0065] The technical solution of this application is described below with a specific embodiment:

[0066] The remote sensing image small target detection method provided by the present application is a model method for target detection in remote sensing images. Based on the YOLOv8 model, the present application proposes a SCW-YOLO model for detecting small targets such as cars (including small cars and large cars), airplanes, ships and oil tanks in remote sensing images. The SCW-YOLO model provided by the present application is a remote sensing image small target detection model improved based on YOLOv8. The model is not only suitable for marine ship detection, but also for target detection of cars, airplanes and oil tanks on land. Just input a high-resolution remote sensing image into the model, and the model can mark the specific locations of small targets such as cars, airplanes, ships and oil tanks in the image.

[0067] like Figure 2 As shown, the remote sensing image small target detection method provided by this application includes:

[0068] 1. Preparation of dataset.

[0069] The DOTA remote sensing image dataset (i.e., DOTA dataset) was obtained, and five typical targets were selected from this dataset: small cars, large cars, airplanes, ships, and storage tanks (i.e., oil tanks), and the area of ​​the above five typical targets was less than 1%. By extracting five typical target instances of small cars, large cars, airplanes, ships, and storage tanks from the DOTA dataset, a total of 5243 remote sensing images were finally obtained, of which 4235 images (i.e., remote sensing images) were used for training and 1008 images were used for testing. The remote sensing image data were annotated with instances to produce a model training dataset.

[0070] This application obtains training remote sensing images from the DOTA dataset. Since the pixel size of the original images of the DOTA dataset is in the range of 800*800 to 4000*4000, it is not suitable for direct training and prediction. Therefore, this application uniformly crops them to a size of 1024*1024.

[0071] 2. Using the YOLOv8 model as the baseline network, a SCW-YOLO model is proposed for small target detection in remote sensing images.

[0072] Based on the YOLOv8 model, this application proposes a SCW-YOLO model for small object detection. Based on the existing YOLOv8, this application removes the large object detection head, adds a higher resolution feature extractor, ensures that local and detail features are not lost, and optimizes the network structure. This application introduces an efficient feature pyramid network to guide subsequent network information to perform feature fusion through a top-down path to reduce redundant calculations. In addition, in order to make the backbone network focus on the target and generate more feature information, this application integrates a coordinate attention mechanism, namely a CA (Coordinate Attention) attention mechanism, in the bottleneck structure of the shallow backbone network, focusing on generating the location information of the feature map, better capturing the spatial and channel information in the image, so as to accurately extract the target features in the image. In view of the poor fitting ability of the bounding box of the CIOU loss function in the YOLOv8 model, in order to improve the fitting ability of the bounding box, this application proposes a dynamic non-monotonic Wise-IOU Loss as a bounding box regression loss function, and solves the problem of annotation quality imbalance by introducing the ratio of the intersection and union of the predicted box and the measured box, and enhances the convergence and robustness of the model.

[0073] Compared with other mainstream algorithms, YOLOv8 has higher accuracy in small target detection, but there are still some problems in small target detection in remote sensing images, such as false positives and missed detections. To solve this problem, this application proposes an improved remote sensing small target detection algorithm SCW-YOLO based on YOLOv8. Figure 3 Compared with the original YOLOv8 network structure shown in Figure 4 The improvements made to the SCW-YOLO network structure shown in YOLOv8 include:

[0074] First, this application designs an efficient feature pyramid structure: the feature layer that generates a feature map of size 20×20 is removed, and the feature layer that generates a feature map of size 160×160 is added, which not only reduces the amount of model parameters, but also can effectively extract feature information from the shallow network and generate more discriminative features; inspired by the bidirectional weighted feature pyramid network (BiFPN) structure, its structure is used to optimize the feature fusion network of the neck structure (i.e., the neck network), and the feature information from the trunk (i.e., the trunk network) is effectively integrated. Secondly, after the feature information is fused through residual convolution in the C2f module of the P4 and P6 parts of the backbone network (there is already residual convolution in the C2f module), a coordinate attention mechanism is added to enhance the local incremental feature extraction capability of the backbone. Finally, this application uses the Wise IOU (WIOU) method to optimize the loss function. This method combines the attention mechanism and the dynamic non-monotonic mechanism to better detect occlusions and small objects.

[0075] In convolutional neural networks, deep features have low resolution, fewer fine-grained features, and more semantic information, while shallow features have high resolution, rich fine-grained features, and less semantic information. In the YOLOv8 model, the backbone network extracts features from the original image after 8, 16, and 32 times downsampling. Although continuously shrinking the feature map can enable the network to obtain a larger receptive field and rich semantic information, for target instances in remote sensing images, since they occupy fewer pixels in the image, the features of the position information in the image are lost after continuous downsampling, resulting in the model being unable to locate the target. In addition, the semantic loss caused by continuous downsampling will have the opposite effect on the feature learning of the detected target.

[0076] The YOLOv8 network has a large receptive field for mid-level and deep features, mainly focusing on representing abstract semantic information. In the feature extraction of small targets, since the pixel area occupied by small targets in the image is relatively small, the reasonable setting of the receptive field size and the use of a multi-scale receptive field strategy can capture more detail information in a limited area. In this application, the input image size is fixed to 640×640, and the prediction head of the model will generate three feature maps corresponding to sizes of 80×80, 40×40, and 20×20, respectively. The smaller the resolution, the larger the receptive field accepted by the neuron, the richer the semantic level, and the corresponding loss of local features and detail features. However, the size of the object detected in the remote sensing image is small, and the detection target may be completely lost in the feature map of size 20×20, affecting the subsequent feature fusion. Since the previously lost information cannot be fully restored during the upsampling process, the model loses key semantic information during the detection process, thereby reducing the detection accuracy. This application will remove the feature extractor of size 20×20 (including the C2f module and the fusion module connected to the C2f module) and the corresponding detection head, which not only simplifies the structure of the model, but also solves the problem of information loss. Among them, after the detection head of size 20×20 is removed, its corresponding feature extractor of size 20×20 is no longer needed, that is, the connection between the detection head of size 20×20 and the C2f module connected to the detection head is lost, and the C2f module establishes a new connection with the detection head of size 40×40, thereby obtaining a feature extractor of size 40×40, so it can be said that the feature extractor of size 20×20 is removed.

[0077] After the above optimization operations, only the 80×80 and 40×40 feature extractors in the model extract target features, and more comprehensive location information and semantic information cannot be extracted. In order to expand the perception range of the network and enhance the detection capability of small objects, this application extends a four-fold down-sampling feature extractor in the P2 part of the backbone network, that is, adding a feature extractor of size 160×160 (after adding a detection head of size 160×160, the C2f module connected to the detection head of size 80×80 is connected to the detection head of size 160×160, thereby obtaining a feature extractor of size 160×160, so it can be said that a feature extractor of size 160×160 is added). This improvement helps to capture a wider range of contextual information, thereby improving the model's detection performance for small objects. The new feature extractor focuses on low-level feature information and enhances the feature extraction capability for small targets. An upsampling module is added to the feature extractor that generates a feature map of size 80×80 (the C2f module connected to a detection head of size 40×40 is connected to a detection head of size 80×80, thereby obtaining a feature extractor of size 80×80. The upsampling module connected to the feature extractor changes the number of input channels due to the change in position, so it can be called an added upsampling module) to perform feature fusion with the new feature extractor, deliver richer semantic information and position information, and thus improve the overall detection performance.

[0078] The neck structure of YOLOv8 adopts the path aggregation (PANet) network structure, which upsamples the features obtained from the backbone network and then performs downsampling fusion, upsamples each feature, and then downsamples and fuses it, so that each feature contains more information. In order to ensure that the model has both comprehensive location information and complete semantic information, this application uses the BiFPN structure to improve the feature fusion network of the neck structure.

[0079] BiFPN is an improvement on the PANet network structure, allowing features to be fused in both top-down and bottom-up directions, thereby more effectively combining features of different scales. BiFPN simplifies the bidirectional network by optimizing the structure, removing nodes with only one input edge, which contributes little to the feature network that fuses different features; adding additional edges between input and output nodes at the same level, and treating each bidirectional path as a feature network layer, to fuse more features without increasing costs too much; BiFPN optimizes the feature fusion process by adding weights to each input feature, so that the network can pay more attention to features with more information, thereby achieving differentiated fusion of information at different scales and improving model performance. However, it will also increase the amount of computation and time consumption of the model, and increase the difficulty of training the model. Therefore, the method of removing weights in this application retains the network structure of BiFPN, that is, this application removes the calculation weight operation of the BiFPN structure and retains the BiFPN structure part introduced above. Figure 5 Shown is a comparison of the original neck structure and the improved neck structure. Figure 5 Part (a) shows the original bottleneck structure. Figure 5 Part (b) shows the improved neck structure. Figure 5 The connection method of BiFPN is explained.

[0080] In the YOLOv8 network structure, the P2 part is downsampled 4 times, which will introduce unnecessary noise and affect the backbone network's extraction of feature information. For the target in this application, it occupies fewer pixels in the image, resulting in the model being unable to accurately notice the target. In order to further improve the detection accuracy of the model under complex backgrounds and enable the model to accurately locate the position of the detected target, an attention mechanism is integrated in the bottleneck in the C2f module of the P2 and P4 parts, so that the module can capture more critical feature information to enhance local features. However, in order to avoid generating a larger amount of computation, this application chooses to introduce a lightweight coordinate attention mechanism. This application refers to the enhanced module as the CA-C2f module, and the enhanced bottleneck structure as the CA-Bottleneck structure. The CA-C2f module, which combines the coordinate attention mechanism with the C2f module in the YOLOv8 backbone network, brings huge advantages to small target detection tasks. This integrated module improves the accuracy of the model's positioning and target recognition by capturing cross-channel, spatial and position information. Among them, the C2f module structure is as follows Figure 6 As shown, the CA-C2f module structure is as follows Figure 7 shown.

[0081] Coordinate attention is an efficient attention mechanism that divides the attention process into two one-dimensional feature encoding processes, namely, coordinate information embedding and coordinate attention generation. Figure 8 As shown in Figure 1. By inputting position information into channel attention, the network can obtain information from a larger area and avoid introducing large overhead. Coordinate attention encodes channel relationships and long-term dependencies through precise position information. It can convert any intermediate tensor as input and output an output of the same size Where X represents the input tensor, x C represents the Cth tensor in the input tensor, Y represents the output tensor, y C represents the Cth tensor in the output tensor, Represents a three-dimensional tensor, C represents the feature dimension, H represents the feature height, and W represents the feature width.

[0082] First, the global pooling method is used for global encoding of channel attention encoding spatial information. Since it compresses the global spatial information into the channel descriptor, although the position information of the original data is lost to a certain extent, it also enables the attention module to capture long-range spatial interactions with position information. The global pooling decomposition formula is as follows:

[0083]

[0084] Z c represents the output result of the cth channel, x c represents the input data of the cth channel, H represents the feature height, and W represents the feature width. Specifically, a pooling convolution kernel of size (H, 1) or (1, W) is used to encode each channel in the horizontal and vertical directions respectively. This application will obtain the output of the cth channel with a height of h. and the output of the cth channel with width w The formula is as follows:

[0085]

[0086] x c (h,i) represents the input data of the cth channel with height h, x c (j,w) represents the input data with the width of the cth channel being w. The above two transformations generate features in the vertical and horizontal directions respectively, thus obtaining a pair of feature maps with orientation information. This transformation enables the attention module to capture the long-term dependency along one spatial direction and preserve the precise position information along another spatial direction, which helps the network locate the detection target more accurately.

[0087] In order to better utilize the representations generated by the above modules with global receptive fields and precise location information, we propose the operation of transforming coordinate attention generation. First, the two feature maps generated by the previous modules are concatenated using a shared 1×1 convolution function F1 to generate It is an intermediate feature map with horizontal and vertical information, r represents the downsampling ratio, and σ represents the Sigmoid function. It is expressed as follows:

[0088] f=σ(F1([Z h ,Z w ]));

[0089] Among them, Z h is the output of all channels with height h, Z w is the output of all channels with width w. Next, f is split into two tensors along the spatial dimension: the tensor along the height direction and the tensor along the width direction Then use two 1×1 convolutions to transform F along the height direction h and the transformation F along the width direction w , so that the feature map f h and f w Transform to the same number of channels as the input x, the formula is as follows:

[0090] g h =σ(F h (f h ));

[0091] g w =σ(F w (f w ));

[0092] σ represents the sigmoid function. Finally, along the width direction g w and along the height direction g h As the attention weight, we get the final output y of the coordinate attention module c The expression of (i, j) is:

[0093]

[0094] in, represents the attention weight of the c-th channel along the height direction, represents the attention weight of the c-th channel along the width direction.

[0095] In computer vision, the intersection over union (IOU) is often used to measure the overlap between the measured box and the model's predicted box. Fig. 9 As shown in the figure, the application represents the predicted position, width and height of the bounding box as x, y, w, and h, respectively, and the measured position, width and height of the bounding box as x. gt ,y gt , w gt ,h gt .w i ,hi Represents the width and height of the overlap between bounding boxes, from which the loss function L is derived IoU is defined as:

[0096] L IoU = 1-IoU;

[0097]

[0098] Among them, L IoU Represents the IOU loss function. The larger the IOU value (<1), the greater the overlap between the model's predicted box and the measured box. However, blindly pursuing the low-quality target regression of strengthening the bounding box will endanger the performance improvement between models. Unlike the previous YOLO series, the Loss calculation of YOLOv8 includes two parts: classification and regression branches. The classification branch uses the binary cross entropy loss function (BCE Loss); the regression branch uses the complete intersection-over-union loss function (CIOU Loss), and it also needs to be bound to the integral form representation proposed in Distribution Focal Loss. CIOU Loss takes into account the distance, size, overlap, and aspect ratio between the centers of the bounding boxes to represent the change in the aspect ratio between the predicted box and the measured box. CIOU loss function L CIoU The calculation formula is as follows:

[0099]

[0100]

[0101] In the formula, α represents the weight function, and v represents the consistency of the aspect ratio. In addition to CIOU Loss, DIOU Loss, SIOU Loss, EIOU Loss, etc., the aspect ratio is also defined as a relative value. When the predicted box and the measured box have the same aspect ratio but different widths and heights, the value of the loss function of the above bounding box regression is the same, which limits the convergence of the model. Therefore, this application proposes WIOU as the bounding box regression loss function to enhance the pertinence of the model and accelerate network convergence. This application constructs distance attention based on the residual metric and obtains the loss function L′ with a two-layer attention mechanism WIoU The loss gradient R WIoU The calculation formula is as follows:

[0102] L' WIoU =R WIoU L IoU ;

[0103]

[0104] Among them, R WIoU ∈[0,1],LIoU ∈[0,1]. In order to ensure R WIoU Convergence of the proposed method separates W and H from the computational graph. Inspired by Focal E-Loss, a monotone focusing mechanism is designed. This application defines a non-monotonic focusing coefficient outlier to describe the quality of the anchor box. The smaller the outlier, the higher the quality of the anchor box. A small gradient gain is assigned to it, so that the bounding box is focused on the anchor box of normal quality. L WIoU The calculation formula is as follows:

[0105] L WIoU =γL' WIoU ;

[0106]

[0107] Among them, L WIoU is the WIOU loss function L′ WIoU It is improved on the basis of , β is the intermediate parameter, δ represents the abnormality coefficient, γ represents the non-monotonic coefficient, represents the monotone focal coefficient, represents the sliding average. It is dynamic, and the quality division criteria of the anchor boxes are also dynamic, which enables WIOU to make the most appropriate gradient gain allocation strategy at every moment.

[0108] This application builds a training environment and adjusts model parameters, including:

[0109] This application builds a deep learning environment in Pytorch. Test environment: NVIDIA RTX A5000, 24G graphics card memory, CUDA version 11.2, cudnn version 8.0.5, Python3.7 as compiler. This application is improved based on the YOLOv8 algorithm. In order to compare the YOLOv8 algorithm with the SCW-YOLO algorithm and ensure the reliability of this application, the training set and test set of this application are both carried out in the same training environment. In addition, the hyperparameters in the training process are set as follows: the initial learning rate is set to 0.01, the momentum is set to 0.937, the weight_decay is set to 0.0005, the training epoch is 200, and the batchsize is set to 6. The model will automatically select the best optimizer during the training process. The learning rate decay method is used to gradually reduce the learning rate during the training process, where the initial learning rate controls the model parameter update speed, and the final learning rate is obtained by multiplying the initial learning rate by the coefficient. This setting enables the model to converge smoothly to the optimal solution. After the training is completed, the optimal model weight is used for test comparison.

[0110] 3. Performance evaluation of the SCW-YOLO model.

[0111] In order to evaluate the performance of the SCW-YOLO model, accuracy (P), recall (R), average precision (mAP@0.5, mAP@0.5:0.95), detection speed (speed), F1 score and model parameter quantity (Params) are used as indicators for evaluating the model. Accuracy is used to describe the ratio of predicted positive examples to all positive examples, and the calculation formula is as follows:

[0112]

[0113] In the formula, P represents the accuracy, TP represents the number of correctly predicted positive samples, and FP represents the number of samples in which negative samples are predicted as positive samples. The recall rate is used to describe the ratio of predicted positive samples to actual positive samples, and the calculation formula is as follows:

[0114]

[0115] In the formula, R represents the recall rate, and FN represents the number of samples in which positive examples are predicted as negative examples. Later, this application introduced a parameter to integrate the above two parameters, namely mAP, which is defined as the average AP of all objects and is suitable for multi-label image classification and detection. The calculation formula is as follows:

[0116]

[0117] Where N represents the number of classification categories. In addition, this application uses the F1 score to represent the accuracy of the precision and recall rate of the binary classification model. The closer the score is to 1, the more accurate the model is, and the calculation formula is as follows:

[0118]

[0119] After testing the improved algorithm in this application on the DOTA small target dataset, the following results were obtained:

[0120] Compared with the original YOLOv8 model, the SCW-YOLO model mAP@0.5 is increased to 84.9%, the number of parameters is reduced by 53%, and the detection speed reaches 37.6ms. The SCW-YOLO model enhances the network's feature extraction capability in small target areas, separates useful target information from a large amount of feature data, has strong anti-interference ability, and can accurately identify objects.

[0121] The improved remote sensing small target detection model based on YOLOv8 provided in this application not only reduces the number of model parameters but also improves the detection accuracy of the model by optimizing the model structure and designing an efficient feature extraction network, making it easier to deploy it in terminal devices for use. The SCW-YOLO remote sensing image target detection model proposed in this application can be applied to other remote sensing image detection tasks. Fig.10 As shown, compared with the baseline model, the network model proposed in this application is superior to the original model in all aspects, and its performance improves rapidly during the entire training process and gradually stabilizes. Fig.10 The blue curve in the middle represents the detection accuracy of the YOLOv8 model. Fig.10 The orange curve in the middle represents the detection accuracy of the SCW-YOLO model. Fig.11 It can be seen that compared with the baseline detection model, the target detection model proposed in this application has stronger anti-interference ability in remote sensing target detection and can better adapt to complex background environments. Fig.11 Part (a) shows the YOLOv8 model detection results. Fig.11 Part (b) shows the detection results of the SCW-YOLO model.

[0122] Compared with the original YOLOv8 model, this application has the following advantages:

[0123] First of all, since the targets in remote sensing images occupy a very small area of ​​the entire image, the large object detection head of the original YOLOv8 model cannot detect small targets, which will only increase the number of parameters and computational complexity of the model. In addition, as the model is downsampled, the semantic information becomes less and less, resulting in the inability to subsequently fuse more information, thereby reducing the detection accuracy of the model. Therefore, this application introduces an efficient feature pyramid network based on the existing YOLOv8: removing the large object detection head and adding a higher resolution feature extractor. This network structure is not only used to guide subsequent network information to perform feature fusion through a top-down path, but also to ensure that local and detail features are not lost. In addition, in the YOLOv8 model, the neck structure adopts a path aggregation network structure (PANet), that is, a bottom-up channel and a top-down channel are established for feature fusion, which is conducive to combining high-level semantic information with low-level semantic information, thereby improving detection accuracy. In order to reduce the amount of calculation of the neck structure, this application is inspired by the bidirectional feature pyramid structure (BiFPN) and designs an efficient feature pyramid structure: delete nodes with only one input edge, and add an additional edge between the original input node and the output node. This structure can fuse more feature information without increasing the cost. This application optimizes the model structure, removes redundant structures, improves the efficiency of the network in feature fusion, adds a new detection layer, uses an efficient feature fusion network to fuse features, enhances the network's ability to learn features from different scales, and improves the model's training efficiency and detection performance.

[0124] Secondly, this application uses the CA-C2f module in the backbone network to embed coordinate attention into the bottleneck structure. This module enables the network to pay more attention to small target features, improves the model's feature extraction ability for small targets, and improves the overall detection performance of the model.

[0125] Finally, this application proposes a new position regression loss function WIOU to accelerate the convergence of the network.

[0126] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store remote sensing image small target detection data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a remote sensing image small target detection method is implemented.

[0127] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0128] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0129] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0130] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0131] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0132] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A remote sensing image small target detection method, characterized in that: The remote sensing image small target detection method comprises: Acquire a model training data set; the model training data set includes a plurality of training remote sensing images marked with positions and categories of small targets; the small targets include small cars, large cars, airplanes, ships, and oil storage tanks; The YOLOv8 model is used as the baseline network. On the basis of the YOLOv8 model, the large object detection head of the YOLOv8 model is removed, and a small object detection head is added. The connection between the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure is modified to be the connection between the first C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure. The connection between the third C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure is modified to be the connection between the second C2f module in the YOLOv8 model backbone network and the YOLOv8 model neck structure. The fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network are removed. The bidirectional weighted feature pyramid network structure is used to optimize the feature fusion network of the YOLOv8 model neck structure to fuse the feature information directly generated from the backbone network. The coordinate attention mechanism is added to the second C2f module and the third C2f module in the YOLOv8 model backbone network. The Wise The IOU method optimizes the loss function of the YOLOv8 model to obtain the SCW-YOLO model; wherein the large object detection head is a detection head with a size of 20×20; the small object detection head is a detection head with a size of 160×160; Using the model training data set to train and optimize the SCW-YOLO model to obtain an optimized SCW-YOLO model; The remote sensing image to be detected is input into the optimized SCW-YOLO model, and the position and category of the small target in the remote sensing image to be detected are obtained by using the optimized SCW-YOLO model.

2. The remote sensing image small target detection method according to claim 1, characterized in that: Before obtaining the model training data set, the remote sensing image small target detection method further includes: Get the DOTA dataset; Extracting remote sensing images containing small automobile targets, remote sensing images containing large automobile targets, remote sensing images containing aircraft targets, remote sensing images containing ship targets, and remote sensing images containing oil tank targets in the DOTA data set; the area of ​​small targets in the extracted remote sensing images is less than 1%; The extracted remote sensing images are cropped to a uniform size of 1024×1024 to obtain a plurality of training remote sensing images; Each of the training remote sensing images is labeled with examples to obtain a plurality of training remote sensing images labeled with positions and categories of small targets; A model training data set is created using multiple training remote sensing images marked with the locations and categories of small targets.

3. The remote sensing image small target detection method according to claim 1, characterized in that: The large object detection head of the YOLOv8 model is removed and the small object detection head is added, specifically including: Remove the 20×20 detection head in the YOLOv8 model, add a 160×160 detection head in the YOLOv8 model, modify the connection relationship between the YOLOv8 model neck structure and the 80×80 detection head and the 40×40 detection head in the YOLOv8 model, connect the C2f module connected to the 80×80 detection head in the YOLOv8 model neck structure to the 160×160 detection head, connect the C2f module connected to the 40×40 detection head in the YOLOv8 model neck structure to the 80×80 detection head, and connect the C2f module connected to the 20×20 detection head in the YOLOv8 model neck structure to the 40×40 detection head.

4. The remote sensing image small target detection method according to claim 1, characterized in that: The removal of the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network specifically includes: Remove the fifth convolution module and the fourth C2f module in the YOLOv8 model backbone network, and connect the third C2f module in the YOLOv8 model backbone network to the spatial pyramid pooling module in the YOLOv8 model backbone network.

5. The remote sensing image small target detection method according to claim 1, characterized in that: The bidirectional weighted feature pyramid network structure is used to optimize the feature fusion network of the neck structure of the YOLOv8 model to fuse the feature information directly generated from the backbone network, specifically including: A bidirectional weighted feature pyramid network structure is used to optimize the feature fusion network of the neck structure of the YOLOv8 model. A new connection established by the bidirectional weighted feature pyramid network structure is used to enable the feature fusion network of the neck structure of the YOLOv8 model to fuse feature information directly generated from the backbone network. The new connection is a connection between the second C2f module in the backbone network of the YOLOv8 model and the third fusion module in the neck structure of the YOLOv8 model. The third fusion module in the neck structure of the YOLOv8 model is a fusion module connected to the first convolution module in the neck structure of the YOLOv8 model.

6. The remote sensing image small target detection method according to claim 1, characterized in that: The coordinate attention mechanism is added to the second C2f module and the third C2f module in the YOLOv8 model backbone network, specifically including: After the feature information is fused through residual convolution in the second C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added. At the same time, after the feature information is fused through residual convolution in the third C2f module in the YOLOv8 model backbone network, a coordinate attention mechanism is added.

7. The remote sensing image small target detection method according to claim 1, characterized in that: The Wise IOU method is used to optimize the loss function of the YOLOv8 model, specifically including: The Wise IOU loss function is used to replace the CIOU loss function of the YOLOv8 model.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the remote sensing image small target detection method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the remote sensing image small target detection method described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the remote sensing image small target detection method described in any one of claims 1 to 7 is implemented.

Images