Remote sensing image target detection method based on improved YOLOv8 model

Abstract

The invention discloses a remote sensing image target detection method based on an improved YOLOv8 model, and the method comprises the steps: replacing a C2f layer of a Backbone network in the YOLOv8 model with an RFN composite convolution module, and additionally arranging an ASCPA adaptive spatial correlation pyramid attention module at a connection part of an SPPF module and a Neck feature enhancement network of the Backbone network, thereby obtaining the improved YOLOv8 model. And performing target detection on the remote sensing image by using the improved YOLOv8 model. By adopting the remote sensing image target detection method, a network can efficiently fuse local and global features, the diversity and integrity of the features are kept, meanwhile, multi-scale spatial information on a feature map can be effectively extracted, effective features are selectively enhanced through spatial correlation feature re-calibration, and the target detection accuracy is improved. And the attention capability to the small target is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular relates to a remote sensing image target detection method based on an improved YOLOv8 model. Background Art

[0002] With the continuous development of aerospace technology, high-resolution remote sensing images can clearly reflect the geometric structure and rich spatial information of objects. Remote sensing image target detection technology plays an important role in urban planning, environmental monitoring, disaster management and daily life. However, remote sensing images have unique characteristics. Targets usually show arbitrary directions, dense arrangements and complex backgrounds. Applying target detection models for simple backgrounds to remote sensing target detection often fails to meet actual requirements. These factors make remote sensing target detection in complex backgrounds more challenging. At the same time, remote sensing target detection also faces a series of unique challenges, including scale changes, perspective differences, changes in lighting conditions, background interference, noise interference, low imaging resolution, and significant changes in target morphology and distribution.

[0003] Remote sensing images are difficult to distinguish objects due to the phenomenon of "same object, different spectrum" and "same spectrum, different object". The difference between different types of objects caused by shadows and mutual occlusion between objects is small. In addition, remote sensing targets themselves have the characteristics of small size, multi-scale, dense distribution, etc. Pooling and other operations in the feature extraction process will reduce the image resolution when expanding the receptive field, and it is easy to lose the target feature information. Therefore, the target detection effect is not ideal. Although YOLOv8 can effectively extract the differentiated features of objects to a certain extent, the depth and complexity of its Bottleneck module are relatively low, which is not enough to extract deep feature information. The C2f layer is easy to ignore or confuse the features of small targets. There are problems such as insufficient learning of some target features and low accuracy of small target feature extraction. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a remote sensing image target detection method based on an improved YOLOv8 model, which solves the technical problems that the existing YOLOv8 has insufficient learning degree of some target features and low accuracy of small target feature extraction when applied to remote sensing image target detection.

[0005] The present invention can be achieved through the following technical solutions:

[0006] A remote sensing image target detection method based on an improved YOLOv8 model is proposed. The C2f layer of the Backbone network in the YOLOv8 model is replaced by the RFN composite convolution module, and an ASCPA adaptive spatial correlation pyramid attention module is added at the connection between the SPPF module of the Backbone network and the Neck feature enhancement network to obtain an improved YOLOv8 model. Then, the improved YOLOv8 model is used to detect targets in remote sensing images.

[0007] The RFN composite convolution module includes a first branch and a second branch in parallel, wherein the first branch uses the FocalNext module to extract deep features of the feature map X, and uses ordinary convolution to extract shallow features of the feature map X, and then concatenates and fuses the deep features and the shallow features to obtain a first fused feature;

[0008] The second branch uses the RepBottleneck module to extract the multi-level features of the feature map X, and uses ordinary convolution to extract the shallow features of the feature map X, and then splices and fuses the multi-level features and the shallow features to obtain the second fused features;

[0009] Finally, the first fusion feature and the second fusion feature are concatenated and fused for output.

[0010] Furthermore, the feature map X enters the first branch and the second branch respectively via the first common convolutional layer.

[0011] In the first branch, the first common convolution layer enters the third common convolution layer and the FocalNext module respectively, and then the processing results of the third common convolution and the FocalNext module are sequentially spliced, fused and subjected to common convolution operations to obtain the first fusion feature;

[0012] In the second branch, the fourth common convolution layer is first passed through the fifth common convolution layer and the RepBottleneck module respectively, and then the processing results of the fifth common convolution layer and the RepBottleneck module are sequentially spliced, fused and subjected to common convolution operations to obtain a second fusion feature;

[0013] The first fusion feature and the second fusion feature are sequentially concatenated and fused and subjected to ordinary convolution operations to output a final feature map.

[0014] Furthermore, the FocalNext module includes a plurality of FocalNextBlock modules which are arranged in parallel and stacked, and a split random partitioning method is used to control the feature maps entering each FocalNextBlock module.

[0015] Furthermore, the RepBottleneck module includes a RepConv re-parameterized convolution layer, a normal convolution layer and a residual connection path. In the training stage, the RepConv re-parameterized convolution layer is divided into 3×3 convolution and 1×1 convolution, and is normalized by a BN layer respectively, and then the SiLU activation function is used for feature adjustment; in the prediction stage, the RepConv re-parameterized convolution layer is re-parameterized into an equivalent 3×3 convolution, and then the SiLU activation function is used for feature adjustment.

[0016] Furthermore, the input of the ASCPA adaptive spatial correlation pyramid attention module is connected to the SPPF module, and its output is upsampled and concatenated with the C2F feature map from the upper layer of the Neck feature enhancement network.

[0017] Furthermore, the loss function of the improved YOLOv8 model is set as follows

[0018]

[0019] Among them, L Dt Represents the distance loss, L Sp represents shape loss, ω is the difficulty coefficient, ω=(1-IoU inner ) γ , γ is the focusing parameter, I inter Represents the overlapping area between the scaled inner target box and the inner anchor box, U union Represents the union area of ​​the scaled inner target box and the inner anchor box.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1) In the backbone network, the RepFocalNet composite convolution module is designed to replace the original C2f layer. Two parallel branches are used to extract deep features and multi-scale features respectively, and then fuse them with shallow local features. Through the concatenation and convolution operations of features at different levels, the effective fusion of local and global features is achieved to maintain the diversity and integrity of features. At the same time, the calculation of the prediction stage is simplified to improve the reasoning efficiency of the model.

[0022] 2) The Adaptive Spatial Correlation Pyramid Attention (ASCPA) module can be used to extract multi-scale spatial information from feature maps. This module extracts global features and generates weighted coefficients through autocorrelation calculations at different scales, performs weighted fusion on features of each scale, and realizes the recalibration of spatial correlation features. This method can selectively enhance effective features, especially improve the attention to insignificant areas and small targets, thereby improving the detection ability of the model;

[0023] 3) Design the Focal Inner SIoU (FIS) loss function. By introducing the Focal Loss mechanism, the loss weight of easy-to-classify samples is adjusted, so that the model pays more attention to difficult-to-classify samples during training. In addition, this loss function combines the characteristics of Inner SIoU and accelerates the bounding box regression process by introducing auxiliary bounding boxes. This design further enhances the model's ability to recognize small targets and other difficult-to-detect targets, thereby improving the overall detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall network structure diagram of the present invention;

[0025] Figure 2a Schematic diagram of the structure of the RepFocalNet composite convolution module of the present invention;

[0026] Figure 2b Schematic diagram of the coordination structure of the FocalNext module and other convolutional layers of the present invention

[0027] Figure 2c It is a structural schematic diagram of the FocalNextBlock module of the present invention;

[0028] Figure 2d Schematic diagram of the processing process of the RepConv heavy parameter convolution layer in the RepBottleneck module of the present invention in the training stage and the prediction stage;

[0029] Figure 3 It is a structural schematic diagram of the ASCPA module of the present invention;

[0030] Figure 4 It is a schematic diagram of the method for calculating the FIS main loss function of the present invention;

[0031] Figure 5 The figure is a schematic diagram of the visualization result of target detection using the remote sensing image target detection method of the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following examples and drawings specifically illustrate a remote sensing image target detection method based on an improved YOLOv8 model of the present invention. It should be noted that the description of these implementation methods is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0033] The traditional YOLOv8 network model mainly consists of three parts, namely the Backbone network, the Neck feature enhancement network and the Head detection network. The Backbone network is the basis of the model and is responsible for extracting features from the input image. These features are the basis for subsequent network layers to detect targets, and usually adopts a structure similar to CSPDarknet; the Neck feature enhancement network is located between the backbone network and the Head detection network, and its function is to perform feature fusion and enhancement; the Head detection network is the decision-making part of the target detection model and is responsible for generating the final detection results. However, due to the complexity and particularity of remote sensing images themselves, the use of the traditional YOLOv8 network model for target detection is not ideal. Therefore, Figure 1 As shown, the present invention provides a remote sensing image target detection method based on an improved YOLOv8 model, replacing the C2f layer of the Backbone network in the YOLOv8 model with an RFN composite convolution module, and adding an ASCPA adaptive spatial correlation pyramid attention module at the connection between the SPPF module of the Backbone network and the Neck feature enhancement network, thereby obtaining an improved YOLOv8 model, and then using the improved YOLOv8 model to perform target detection on remote sensing images.

[0034] The details are as follows:

[0035] S1: Design the RepFocalNet (RFN) composite convolutional module to replace the original C2f layer in the backbone network to improve the multi-scale modeling and feature extraction capabilities under complex backgrounds.

[0036] like Figure 2a-2dAs shown in the figure, the RepFocalNet composite convolution module includes a first branch and a second branch. The first branch uses the FocalNext module to extract the deep features of the feature map X, and uses ordinary convolution to extract the shallow features of the feature map X, and then the deep features and the shallow features are sequentially spliced ​​and fused and ordinary convolution operations are performed to obtain the first fused features; the second branch uses the RepBottleneck module to extract the multi-level features of the feature map X, and uses ordinary convolution to extract the shallow features of the feature map X, and then the multi-level features and the shallow features are sequentially spliced ​​and fused and ordinary convolution operations are performed to obtain the second fused features; finally, the first fused features and the second fused features are sequentially spliced ​​and fused and ordinary convolution operations are performed to output the final feature map.

[0037] The feature map X is passed through the first common convolution layer to extract basic features, and then is divided into two branches for further processing; the first branch passes the basic features to the FocalNext module through the second common convolution layer, which is composed of multiple parallel FocalNextBlocks continuously superimposed to extract deep-level features. At the same time, the basic features are passed to the third common convolution layer through the second common convolution layer for convolution processing to extract shallow-level features, that is, the original features after the preliminary convolution are retained, and then the shallow-level features after the preliminary convolution are spliced ​​and fused with the deep-level features output by the FocalNext module. The fused features are processed by common convolution to obtain the first fused features;

[0038] In the FocalNextBlock module, the input features are first randomly discarded through the Drop Path to enhance the regularization ability of the model, and then the dimensions of the feature map are rearranged through the Permute operation, and then normalized through the LayerNorms layer and activated through the GELU function; then convolution and Permute operations are performed, and finally the features are scaled and output according to the gamma parameter;

[0039] In addition, considering that the input features of the existing FocalNextBlock module are equally divided, this may result in a relatively large computational overhead. The present invention uses a split random partitioning method to selectively pass the input features to multiple FocalNextBlock modules, which can reduce computational expenses.

[0040] The second branch mainly uses the RepBottleneck module to capture multi-scale features. It uses a multi-branch structure to extract information from different paths, different receptive fields and feature spaces to capture multi-level features. That is, it first enters the fifth ordinary convolution layer and the RepBottleneck module through the fourth ordinary convolution layer, and then performs splicing, fusion and ordinary convolution operations on the processing results of the fifth ordinary convolution layer and the RepBottleneck module in turn to obtain the second fusion feature.

[0041] The RepBottleneck module consists of a RepConv heavy-parameter convolution layer, a normal convolution layer, and a residual connection path. It retains the original feature information by adding the convolution output to the input to enhance the training effect of the network.

[0042] In addition, the RepBottleneck module adopts different structures in the training and inference stages, such as Figure 2d As shown in the figure, in the training stage, the RepConv re-parameterized convolution layer is divided into 3×3 convolution and 1×1 convolution, and normalized by the BN layer respectively, and then combined with the SiLU activation function for feature adjustment; in the prediction stage, the RepConv re-parameterized convolution layer is re-parameterized into an equivalent 3×3 convolution, and then combined with the SiLU activation function for feature adjustment, which can reduce the amount of calculation and improve the reasoning speed.

[0043] Specifically, the input image with a resolution of 640×640×3 is first convolved twice to generate a feature map of 160×160×128; then the feature map of 160×160×128 is convolved once, once, and once with RepFocalNet to generate a feature map of 80×80×256; then the feature map of 80×80×256 is convolved once and once with RepFocalNet to generate a feature map of 40×40×512; finally, the feature map of 40×40×512 is convolved once and once with RepFocalNet to generate a feature map of 20×20×1024;

[0044] S2: An ASCPA adaptive spatial correlation pyramid attention module is added at the connection between the SPPF module of the Backbone network and the Neck feature enhancement network, and effective features are selectively enhanced by recalibrating spatial correlation features.

[0045] like Figure 3As shown in the figure, the ASCPA adaptive spatial correlation pyramid attention module takes the feature map x of size (B, H, W, C) processed by the SPPF module as input, and performs autocorrelation calculation on the feature map x at three scales. The autocorrelation matrix is ​​calculated only once at the first scale to obtain the autocorrelation matrix f1; at the second and third scales, the input feature map x is subjected to 3×3 and 5×5 average pooling respectively, and the obtained feature maps x2 and x3 are subjected to autocorrelation matrix calculation once to obtain f2 and f3 respectively; secondly, an average pooling operation is performed on each autocorrelation matrix f1, f2, f3 to obtain three global feature vectors v1, v2, v3; finally, the matrix V = stack (v1, v2, v3) is linearly transformed and normalized through the fully connected layer and the softmax function to generate the weighting coefficient V′, and the weighting coefficient is used to perform weighted summation on the autocorrelation matrices f1, f2, f3 of different scales to finally obtain the feature map f;

[0046] Subsequently, the feature map f output by the ASCPA adaptive spatial correlation pyramid attention module undergoes two operations: one is an upsampling operation, and the other is directly concatenated and fused with the C2F feature map from the upper layer of the Neck feature enhancement network through the Concat module, further enhancing the model's attention to insignificant features and improving the detection capability of small targets.

[0047] S3: Design the FIS main loss function and dynamically adjust the loss weights of different samples

[0048] We refer to the loss function L Innet-SIoU And the loss function L F (p t ) to create the main loss function for the improved YOLOv8 model of the present invention.

[0049] Using Inner SIoU, by introducing auxiliary bounding boxes and adjusting the scale factor to generate auxiliary bounding boxes of different scales, the bounding box regression process is accelerated. The expression of the loss function is:

[0050]

[0051] Among them, L Dt Represents the distance loss, L Sp represents the shape loss, I inter Represents the overlapping area between the scaled inner target box (InnerTarget Box) and the inner anchor box (InnerAnchor Box), U union Represents the union area of ​​the scaled inner target box and the inner anchor box.

[0052] Focal Loss is used to solve the problem of class imbalance in target detection. It adjusts the loss weight of easy-to-classify samples so that the model pays more attention to difficult-to-classify samples. The expression of the loss function is:

[0053] L F (p t )=-α t (1-p t ) γ log(p t )

[0054] Among them, p t is the probability of the predicted true category, α t is the balance factor, γ is the focusing parameter;

[0055] The main loss function expression of the present invention is:

[0056]

[0057] Where ω is the difficulty coefficient, ω=(1-IoU inner ) γ , γ is the focusing parameter,

[0058] Next, the remote sensing image target detection method of the present invention is used to perform target detection on the remote sensing image.

[0059] The experimental materials and equipment, parameter settings, and evaluation indicators of this embodiment are as follows:

[0060] 1) Experimental materials and equipment

[0061] The DIOR dataset was used to train and verify the algorithm during the experiment. The dataset contains 23,463 high-resolution remote sensing images and 192,472 instances, covering 20 object classes.

[0062] The experiment is implemented based on the deep learning framework PyTorch. The experimental platform is configured as follows: using the computer processor Intel Xeon Max 6426Y with 128GB of running memory; using the independent graphics card Nvidia A800 with 80GB of video memory; using Cuda12.1 to accelerate GPU computing; using Python packages such as Numpy for data processing; and using the Matplotlib library to complete operations such as visualization of detection results.

[0063] 2) Parameter settings

[0064] The model training parameters are set as follows: batch size is set to 16; the number of training epochs is set to 300; the initial learning rate is set to 0.001, and the learning rate is reduced to 0.0001 after 100 epochs; and the Adam optimizer is used to update the network training weights.

[0065] 3) Evaluation indicators

[0066] When YOLOv8 is applied to remote sensing target detection, it focuses on both the accuracy of the algorithm and the efficiency of the algorithm. The indicators for measuring target detection algorithms include recall rate (Recall), average precision (Average Precision, AP) and mean Average Precision (mAP) and other standards. Among them, recall rate is a performance indicator that measures the ratio of objects correctly identified by the algorithm to the total number of objects, while AP and mAP can better reflect the detection performance of various categories in the remote sensing image target detection task, and truly represent the actual detection situation of the algorithm. Therefore, mAP is selected as a performance indicator for evaluating the accuracy of the algorithm. Its calculation formula is shown as follows:

[0067]

[0068] Among them, Ωu represents the true result, p uj represents the position of object j, p uj <p ui Indicates that object j ranks before object i in the candidate list.

[0069] 4) Result evaluation

[0070] Table 1 shows the target detection performance comparison results of YOLOv8 and the improved method. It can be seen from Table 1 that the improved method has better performance. Compared with the original YOLOv8 algorithm, the classification accuracy of difficult-to-detect targets such as bridge and vehicle has increased by 4.2% and 8.9% respectively; the Recall and mAP@50 indicators have increased by 3% and 1.5% respectively. From the classification accuracy of 20 classes, the overall results of the improved algorithm are better than YOLOv8. Figure 5 The results of remote sensing target detection using YOLOv8 and improved YOLOv8 algorithms, where the first line is the manually annotated image, the second line is the detection result of the YOLOv8 algorithm, and the third line is the detection result of the same image corresponding to the algorithm of the present invention; from the results, it can be seen that the image detected by the YOLOv8 algorithm has target loss and is easily blocked by the target. In comparison, the algorithm proposed in the present invention can effectively identify tiny targets in remote sensing images, has a higher object recognition accuracy, and has a lower misjudgment and missed judgment rate. At the same time, it also shows good results in the detection of targets of different scales.

[0071] Table 1 Performance comparison between YOLOv8 algorithm and the algorithm of the present invention

[0072]

[0073]

[0074] The above-mentioned implementation modes are preferred cases of the present invention and are not used to limit the protection scope of the present invention. Various deformations or modifications that can be made by ordinary technicians in this field without creative work within the scope of the attached claims are still within the protection scope of this patent.

Claims

1. A remote sensing image target detection method based on an improved YOLOv8 model, characterized in that: The C2f layer of the Backbone network in the YOLOv8 model is replaced by the RFN composite convolution module, and an ASCPA adaptive spatial correlation pyramid attention module is added at the connection between the SPPF module of the Backbone network and the Neck feature enhancement network to obtain an improved YOLOv8 model, which is then used to perform target detection on remote sensing images. The RFN composite convolution module includes a first branch and a second branch in parallel, wherein the first branch uses the FocalNext module to extract deep features of the feature map X, and uses ordinary convolution to extract shallow features of the feature map X, and then concatenates and fuses the deep features and the shallow features to obtain a first fused feature; The second branch uses the RepBottleneck module to extract the multi-level features of the feature map X, and uses ordinary convolution to extract the shallow features of the feature map X, and then splices and fuses the multi-level features and the shallow features to obtain the second fused features; Finally, the first fusion feature and the second fusion feature are concatenated and fused for output.

2. The remote sensing image target detection method based on the improved YOLOv8 model according to claim 1, characterized in that: The feature map X enters the first branch and the second branch respectively through the first common convolution layer. In the first branch, it first enters the third common convolution layer and the FocalNext module respectively through the second common convolution layer, and then the processing results of the third common convolution and the FocalNext module are sequentially spliced, fused and subjected to common convolution operations to obtain a first fusion feature; In the second branch, the fourth common convolution layer is first passed through the fifth common convolution layer and the RepBottleneck module respectively, and then the processing results of the fifth common convolution layer and the RepBottleneck module are sequentially spliced, fused and subjected to common convolution operations to obtain a second fusion feature; The first fusion feature and the second fusion feature are sequentially concatenated and fused and subjected to ordinary convolution operations to output a final feature map.

3. The remote sensing image target detection method based on the improved YOLOv8 model according to claim 2, characterized in that: The FocalNext module includes a plurality of FocalNextBlock modules which are arranged in parallel and stacked, and a split random partitioning method is adopted to control the feature maps entering each FocalNextBlock module.

4. The remote sensing image target detection method based on the improved YOLOv8 model according to claim 2, characterized in that: The RepBottleneck module includes a RepConv re-parameterized convolution layer, a normal convolution layer and a residual connection path. In the training stage, the RepConv re-parameterized convolution layer is divided into 3×3 convolution and 1×1 convolution, and is normalized by a BN layer respectively, and then the SiLU activation function is used for feature adjustment; in the prediction stage, the RepConv re-parameterized convolution layer is re-parameterized into an equivalent 3×3 convolution, and then the SiLU activation function is used for feature adjustment.

5. The remote sensing image target detection method based on the improved YOLOv8 model according to claim 1, characterized in that: The input of the ASCPA adaptive spatial correlation pyramid attention module is connected to the SPPF module, and its output is upsampled and concatenated with the C2F feature map from the upper layer of the Neck feature enhancement network.

6. The remote sensing image target detection method based on the improved YOLOv8 model according to claim 1, characterized in that: The loss function of the improved YOLOv8 model is set as follows Among them, L Dt Represents the distance loss, L Sp represents shape loss, ω is the difficulty coefficient, ω=(1-IoU inner ) γ , γ is the focusing parameter, I inter Represents the overlapping area between the scaled inner target box Inner Target Box and the inner anchor box InnerAnchor Box, U union Represents the union area of ​​the scaled inner target box and the inner anchor box.

Images