Remote sensing vehicle target detection method based on feature fusion and attention mechanism

Abstract

The invention provides a remote sensing vehicle target detection method based on feature fusion and an attention mechanism. The method comprises the following steps: step 1, improving a YOLOv8 network to form a remote sensing vehicle target detection network; wherein the YOLOv8 network is improved, specifically, all C2f modules in a backbone network are replaced with focusing feature extraction modules (FFEM), and the FFEM modules are mainly obtained after a CA attention mechanism is fused into splicing operation in the C2f modules; 2, training the remote sensing vehicle target detection network to obtain a remote sensing vehicle target detection model; and step 3, inputting a to-be-detected remote sensing vehicle image into the remote sensing vehicle target detection model to obtain a detection result. According to the method, the small vehicle target can be well detected under the condition that the background of the remote sensing image is complex.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image target detection, and in particular to a remote sensing vehicle target detection method based on feature fusion and attention mechanism. Background Art

[0002] Remote sensing technology has developed rapidly and is widely used in various fields such as search and rescue, exploration, geological disaster monitoring, urban planning, land use and agricultural survey. Remote sensing image target detection has become an important technology, which uses image processing technology and computer vision algorithms to accurately detect and locate objects of interest (such as vehicles, aircraft and ships, etc.) in images. However, vehicles in remote sensing images have the characteristics of small targets, small number of pixels and complex background. Most existing detection methods only have good detection effects on large targets, and poor detection performance for small targets.

[0003] Due to the particularity of remote sensing images, compared with vehicle detection on natural images, remote sensing vehicle target detection usually faces the following challenges: (1) Due to the complex background of remote sensing images, many targets are occluded, making it difficult for the model to extract sufficient features. (2) Vehicles in remote sensing images are usually small and easily interfered by noise, which greatly tests the model's ability to extract and fuse features. (3) Existing models are sensitive to target size when detecting targets, resulting in low detector performance. Summary of the invention

[0004] In order to at least partially solve the above three challenges, the present invention provides a remote sensing vehicle target detection method based on feature fusion and attention mechanism.

[0005] The present invention provides a remote sensing vehicle target detection method based on feature fusion and attention mechanism, comprising:

[0006] Step 1: Improve the YOLOv8 network to form a remote sensing vehicle target detection network; wherein, the improvement of the YOLOv8 network specifically includes: replacing all C2f modules in the backbone network with focused feature extraction modules FFEM (Focused Feature Extraction Module1), wherein the FFEM is mainly obtained by integrating the CA attention mechanism into the splicing operation in the C2f module;

[0007] Step 2: training the remote sensing vehicle target detection network to obtain a remote sensing vehicle target detection model;

[0008] Step 3: Input the remote sensing vehicle image to be detected into the remote sensing vehicle target detection model to obtain the detection result.

[0009] Furthermore, the feature extraction process of the CA attention mechanism includes:

[0010] The feature map X output after the splicing operation in the C2f module is pooled in the horizontal and vertical directions to obtain two feature maps z h and z w ;

[0011] The feature map z h and z w Perform concatenation, convolution, normalization and activation operations in the spatial dimension to obtain the feature map f;

[0012] Split the feature map f along the spatial dimension to obtain the feature map f h and f w ;

[0013] The feature map f h and f w Convolution and activation operations are performed in turn to obtain the attention weight g h and g w ;

[0014] The feature map X output after the concatenation operation in the C2f module is combined with the attention weight g h and g w Multiply them together to get the weighted feature map Y.

[0015] Furthermore, the YOLOv8 network is improved, including:

[0016] The path aggregation network in the neck network is replaced by a triple-weighted feature fusion network TWFFPN (TripleWeighted Feature Fusion Pyramid Network); the TWFFPN mainly adds a small target layer P2 layer to the structure of BiFPN, and changes the feature fusion method of the P2 layer and the P5 layer, so that the output feature map of the P2 layer is obtained by fusion of the initial features and intermediate features from the current layer and the intermediate features from the P3 layer; the output feature maps of the P3 layer and the P4 layer are both obtained by fusion of the initial features and intermediate features from the current layer and the output feature map from the previous layer; the output feature map of the P5 layer is obtained by fusion of the initial features from the current layer, the intermediate features of the P4 layer and the output feature map of the P4 layer;

[0017] Correspondingly, the output feature maps of the P3 layer, P4 layer and P5 layer are used as the three detection heads of the remote sensing vehicle target detection network.

[0018] Furthermore, the output feature map of the P2 layer Output feature map of P3 layer Output feature map of P4 layer And the output feature map of the P5 layer They are:

[0019]

[0020]

[0021] Among them, ω i ' j Represents learnable weights, i=2,3,4,5,j=1,2,3,ε n Indicates the preset value, n=1,2,3,4,P i in represents the initial features of the i-th layer, P i td Represents the intermediate features of the i-th layer, Resize represents an upsampling operation or a downsampling operation, and Conv represents a convolution operation.

[0022] Furthermore, in the process of training the remote sensing vehicle target detection network, NWD (Normalized Wasserstein Distance) loss is used as the bounding box loss.

[0023] Beneficial effects of the present invention:

[0024] (1) The present invention designs a focused feature extraction module, which improves the feature extraction capability of the model by integrating the CA attention mechanism into the C2f module and replacing it with the C2f module in the original backbone network.

[0025] (2) The present invention designs a triple-weight feature fusion network, which can better fuse deep semantic information and shallow detail information, thereby enhancing the model's ability to detect small targets.

[0026] (3) The present invention uses NWD loss as the bounding box regression loss of the model to alleviate the problem that the model is extremely sensitive to different target sizes and improve the model convergence speed.

[0027] Experiments on DOTA and DIOR datasets show that compared with existing detection methods, the method of the present invention has better performance in remote sensing image target detection. At the same time, it can be seen from the ablation experiment that each improved module proposed in the present invention can effectively improve the detection accuracy. Therefore, compared with the existing detection algorithm, the method proposed in the present invention is more suitable for target detection in remote sensing images, and has important practical significance for the detection of remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1A flowchart of a remote sensing vehicle target detection method based on feature fusion and attention mechanism provided by an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a remote sensing vehicle target detection network structure provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the structure of a focus feature extraction module (FFEM) provided in an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a triple weight feature fusion network (TWFFPN) provided in an embodiment of the present invention;

[0032] Figure 5 The detection results of the DOTA data set provided by the embodiment of the present invention; wherein, (a1-a2) are input remote sensing images, (b1-b2) are the detection results of the method of the present invention, and (c1-c2) are the detection results of YOLOv8-N;

[0033] Figure 6 The detection results of the DIOR dataset provided by the embodiment of the present invention, wherein (a1-a2) are the input remote sensing images, (b1-b2) are the detection results of the method of the present invention, and (c1-c2) are the detection results of YOLOv8-N. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Example 1

[0036] like Figure 1 As shown, an embodiment of the present invention provides a remote sensing vehicle target detection method based on feature fusion and attention mechanism, comprising the following steps:

[0037] S101. Improve the YOLOv8-N network to form a remote sensing vehicle target detection network;

[0038] Specifically, for the original YOLOv8-N network, the backbone network will be disturbed by the complex background when extracting features, resulting in insufficient feature extraction by the model and failure to make special processing for some important features. Therefore, in the embodiment, the YOLOv8-N network is improved, specifically including: Figure 2As shown in Figure 1, all C2f modules in the backbone network are replaced with the focused feature extraction module FFEM, so that the backbone network can extract more feature information and focus on important features. The structure of the FFEM module is shown in Figure 1. Figure 3 As shown in Figure 1, it is mainly obtained by integrating the Coordinate Attention (CA) attention mechanism into the concatenation operation of the C2f module.

[0039] Specifically, the feature extraction process of the CA attention mechanism includes:

[0040] First, the feature map X output after the splicing operation in the C2f module is pooled in the horizontal and vertical directions to obtain two feature maps z h and z w ;

[0041] It can be understood that if the number of channels of feature map X is C, the height is H, and the width is W, then the two feature maps z h and z w The sizes of are C*H*1 and C*1*W respectively; the calculation formula of the relevant feature map is as follows:

[0042]

[0043] Among them, x c and z c denote the c-th channel of the input and output feature maps respectively.

[0044] Secondly, the feature map z h and z w Concat, convolution, normalization and activation operations are performed in sequence in the spatial dimension to obtain the feature map f. The process can be expressed by the following formula:

[0045] f=δ(F1([z h ,z w ]))

[0046] Among them, [z h ,z w ] represents the Concat operation, F1 represents the dimensionality reduction operation using a 1*1 convolution kernel, and δ represents the activation operation. It can be understood that using 1*1 convolution can reduce the number of channels, and finally using normalization and activation operations can avoid gradient explosion or gradient disappearance, and introduce nonlinear factors to improve the model's expressiveness.

[0047] Then, the feature map f is split along the spatial dimension to obtain the feature map f h and f w ; The feature map f h and f wConvolution and activation operations are performed in turn to obtain the attention weight g h and g w ; The process can be expressed by the following formula:

[0048] g h =σ(F h (f h ))

[0049] g w =σ(F w (f w ))

[0050] Among them, F h and F w Indicates that a 1*1 convolution kernel is used for convolution operation, and σ represents an activation operation. It can be understood that the 1*1 convolution operation of this process is intended to increase the number of channels to the same as the number of channels of the input feature map X.

[0051] Finally, the feature map X output after the concatenation operation in the C2f module is combined with the attention weight g h and g w Multiply them together to get the weighted feature map Y, that is: Y = X × g h ×g w .

[0052] S102, constructing a remote sensing vehicle data set, and using the data set to train the remote sensing vehicle target detection network to obtain a remote sensing vehicle target detection model;

[0053] S103: Inputting the remote sensing vehicle image to be detected into the remote sensing vehicle target detection model to obtain a detection result.

[0054] Specifically, this embodiment uses the YOLOv8-N network as the basic framework, and then improves the framework to obtain an improved YOLOv8-N network, such as Figure 2 The detection process of the improved YOLOv8-N network is as follows: first, the backbone network is used to extract the feature information of the input remote sensing image. In order to detect targets of different sizes, the backbone network outputs three feature maps of different sizes; then, the neck network is used to fuse the three extracted feature maps of different sizes to obtain three feature maps of different sizes with rich context information; finally, the detection head is used to detect the fused feature maps to obtain the final detection result.

[0055] Example 2

[0056] In order to further improve the feature fusion performance of the model, based on the above embodiment, the embodiment of the present invention further improves the YOLOv8-N network, such as Figure 2As shown, the path aggregation network (PAN) in the neck network is replaced by a triple weight feature fusion network (TWFFPN).

[0057] Specifically, the original path aggregation network leads to low fusion efficiency between effective features, and the detection of small vehicles cannot achieve ideal results. In order to enhance the fusion between multi-scale features and promote the propagation of context information, this embodiment proposes a TWFFPN network, such as Figure 4 As shown in (b), the TWFFPN network is mainly based on BiFPN (such as Figure 4 A small target layer P2 is added to the structure of (as shown in FIG. 1 ), and the feature fusion method of the P2 and P5 layers is changed, so that the output feature map of the P2 layer is obtained by fusion of the initial features and intermediate features from this layer and the intermediate features from the P3 layer; the output feature maps of the P3 and P4 layers are both obtained by fusion of the initial features and intermediate features from this layer and the output feature map from the layer above; the output feature map of the P5 layer is obtained by fusion of the initial features from this layer, the intermediate features of the P4 layer and the output feature map of the P4 layer.

[0058] The TWFFPN structure still retains the original cross-scale connection and weighted feature fusion structure in BiFPN, and adds a small target detection layer on this basis to achieve cross-level data cross-fusion and better integrate feature information at different levels, such as Figure 4 As shown by the orange line in (b). In addition, in order to avoid excessive model calculation complexity, the P2 layer is not directly output as the detection head. The feature map of the P2-P5 layer is used as an example to illustrate the processing flow of the TWFFPN structure fusion method:

[0059] First, introduce the small target layer P2. For the P2 layer, in order to obtain the final output feature map In addition to the initial features and intermediate features Participate in integration, It also participates in the fusion process to improve the feature richness of the P2 layer and enhance the location information in the network structure. In order not to increase the amount of calculation, the P2 layer is not directly output as a detection head.

[0060]

[0061] Secondly, for the P3 and P4 layers, the output feature maps are composed of the initial features from this layer. and Intermediate Features and And the output features of the previous layer and Fusion obtained.

[0062]

[0063]

[0064] Finally, for the P5 layer, in order to obtain the final output feature map In addition to the initial features And the output features of the fourth layer Participate in fusion and introduce intermediate features This improves the feature loss caused by continuous convolution.

[0065]

[0066] Among them, ω i ' j represents the learnable weights, i=2,3,4,5,j=1,2,3; ε n Indicates the preset value, n = 1, 2, 3, 4, used to avoid numerical instability; P i in represents the initial features of the i-th layer, P i td Represents the intermediate features of the i-th layer, Resize represents an upsampling operation or a downsampling operation, and Conv represents a convolution operation.

[0067] Example 3

[0068] In order to help the model regress the bounding box more accurately and improve the convergence speed of the model, based on the above embodiments, the embodiment of the present invention uses NWD loss as the bounding box loss of the model, which is defined as follows:

[0069] L NWD =1-NWD(N a ,N b )

[0070]

[0071] Among them, NWD (N a ,N b ) represents the similarity between bounding box distributions, represents the two-dimensional Gaussian distribution of the bounding box, c xi ,c yi represents the center coordinates of the inscribed ellipse of the bounding box i, They represent the lengths of the major and minor axes of the inscribed ellipse of the bounding box i, respectively; C represents a constant closely related to the dataset.

[0072] NWD loss performs regression by measuring the similarity between the two-dimensional Gaussian distributions of bounding boxes. It can measure the similarity between non-overlapping or mutually contained bounding boxes and maintain consistent sensitivity to objects of different sizes.

[0073] In order to verify the effectiveness of the detection method of the present invention, the present invention also provides the following experimental data.

[0074] In this experiment, two vehicle datasets were constructed from the DOTA and DIOR datasets. The images in the DOTA dataset come from Google Earth, GF-2 satellite, etc., and have the characteristics of wide coverage, large scale changes, and dense object arrangement. This experiment uses the DOTAV1.0 version, which has 15 categories, and 1817 images with vehicles were extracted from it as one of the research datasets. Another dataset used for detection is DIOR, which has the characteristics of scene visibility and object occlusion. Since the present invention mainly studies vehicle detection, 6421 images containing vehicles were extracted as one of the research datasets. For the vehicle datasets extracted above, the training set, validation set, and test set are divided according to the ratio of 7:2:1. The above two datasets were used for experiments to verify the effectiveness of the model proposed in the present invention.

[0075] (I) Experimental environment and training parameters

[0076] The experimental platform was built with Ubuntu as the operating system and PyTorch as the framework. In the experiment, the CPU was Intel(R) Xeon(R) Platinum 8350C and the GPU was RTX 3080Ti (12GB). In addition, no pre-trained weights were used during the experiment; the learning rate was set to 0.01; the image size was 640*640; the optimizer used SGD; the batch size was set to 16, and the epoch value was 200.

[0077] (II) Experimental results

[0078] (1) Experiments were conducted on the DOTA dataset and compared with other detection methods, as shown in Table 1.

[0079] Table 1 Comparison of experimental results of different methods on DOTA dataset

[0080]

[0081] As can be seen from Table 1, compared with the YOLO series models, the detection accuracy of SSD still needs to be improved, and its parameters and computational complexity are very high. The AP value of YOLOv5-S on large vehicles is 93%, which achieved the best results among the comparison models. However, YOLOv5-S is not as good as HVDNet in terms of key indicators mAP0.5 and mAP0.5:0.95, and the AP value of HVDNet on large vehicles is second only to YOLOv5-S. In addition, the parameters and FLOPs of HVDNet are smaller than those of YOLOv5-S, which are 4.1M and 15.6G, respectively. The AP values ​​of YOLOv7-Tiny and YOLOX-S on large vehicles are similar, but there is still a big gap compared with the HVDNet model of the present invention. In addition, compared with the state-of-the-art networks YOLOv10-N and the baseline YOLOv8n, the parameters and FLOPs of the proposed model are slightly increased, but the highest values ​​​​are achieved in the key indicators mAP0.5 and mAP0.5:0.95, which are 89.6% and 68.6% respectively.

[0082] In order to more intuitively show the advantages of the proposed detection method, the present invention provides the detection results of several different images, such as Figure 5 As shown in the figure, (ai) is the input image, (bi) is the detection result of the method proposed in the present invention, and (ci) is the detection result of YOLOv8-N.

[0083] Depend on Figure 5 From the yellow box in (c1), we can see that YOLOv8-N missed a small white vehicle, while the method proposed in this paper can accurately detect it. Figure 5 (a2) shows that there are tree branches blocking the image, and Figure 5 The detection results in (c2) show that YOLOv8-N missed two small vehicles and one large vehicle in this case, while the model proposed in the present invention can correctly detect the occluded objects.

[0084] (2) In order to verify the effectiveness of each proposed module, ablation experiments were performed on the DOTA dataset, as shown in Table 2.

[0085] Table 2 Ablation experiments on the DOTA dataset

[0086]

[0087] This experiment uses YOLOv8-N as the baseline, and the method proposed in this invention achieves 89.6% mAP on the DOTA dataset. Each improved module proposed in this invention can improve the detection performance to a certain extent. Compared with the basic network YOLOv8-N, the AP value of small vehicles is improved by 3%, the AP value of large vehicles is improved by 1.7%, the recall rate R is improved by 2%, the mAP0.5 is improved by 2.4%, and the mAP0.5:0.95 is improved by 4.3%. It can be seen that when the modules are added to the model cumulatively, the effect gradually improves.

[0088] (3) In addition to the experiments on the DOTA dataset, in order to further verify the effectiveness of the model of the present invention, experiments were also conducted on the DIOR dataset and compared with other detection methods, as shown in Table 3.

[0089] Table 3 Comparison of experimental results of different methods on DIOR dataset

[0090]

[0091]

[0092] The experimental results in Table 3 clearly show that SSD achieves the highest accuracy value, but its FLOPs and parameter amount increase by 54.5G and 23.2M respectively compared with the baseline YOLOv8-N. The recall R values ​​of YOLOv5-S and HVDNet are similar, but the latter has fewer parameters and FLOPs. Although the detection results of the HVDNet model are not optimal in terms of FLOPs and parameter amount, the highest detection values ​​are achieved in the key indicators mAP0.5 and mAP0.5:0.95. Specifically, compared with the baseline YOLOv8-N, the mAP0.5 and mAP0.5:0.95 of the HVDNet of the present invention increase by 3.3% and 2.7% respectively, and the increase in the number of parameters is not obvious. In addition, compared with the most advanced network YOLOv10-N, the HVDNet of the present invention achieves the highest mAP value. In summary, the experimental results in Table 3 once again confirm that the proposed model strikes a balance between detection accuracy and parameter amount.

[0093] In order to more intuitively show the advantages of the proposed detection method, the detection results of several different images are given, such as Figure 6 As shown in the figure, (ai) is the input image, (bi) is the detection result of the method proposed in the present invention, and (ci) is the detection result of YOLOv8-N.

[0094] Depend on Figure 6Comparison between (b1) and (c1) shows that YOLOv8-N not only missed the vehicle, but also misdetected two vehicles, as shown in the yellow and green boxes in the figure. However, the detection method we proposed avoids this situation. Figure 6 As shown in the yellow box in (c2), YOLOv8-N missed two cars, one of which was blocked by shadows, but the model proposed in the present invention can detect it correctly.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote sensing vehicle target detection method based on feature fusion and attention mechanism, characterized in that: include: Step 1: Improve the YOLOv8 network to form a remote sensing vehicle target detection network; wherein the improvement of the YOLOv8 network specifically includes: replacing all C2f modules in the backbone network with a focused feature extraction module FFEM, wherein the FFEM is mainly obtained by integrating the CA attention mechanism into the splicing operation in the C2f module; Step 2: training the remote sensing vehicle target detection network to obtain a remote sensing vehicle target detection model; Step 3: Input the remote sensing vehicle image to be detected into the remote sensing vehicle target detection model to obtain the detection result.

2. According to claim 1, a remote sensing vehicle target detection method based on feature fusion and attention mechanism is characterized in that: The feature extraction process of the CA attention mechanism includes: The feature map X output after the splicing operation in the C2f module is pooled in the horizontal and vertical directions to obtain two feature maps z h and z w ; The feature map z h and z w Perform concatenation, convolution, normalization and activation operations in the spatial dimension to obtain the feature map f; Split the feature map f along the spatial dimension to obtain the feature map f h and f w ; The feature map f h and f w Convolution and activation operations are performed in turn to obtain the attention weight g h and g w ; The feature map X output after the concatenation operation in the C2f module is combined with the attention weight g h and g w Multiply them together to get the weighted feature map Y.

3. According to claim 1, a remote sensing vehicle target detection method based on feature fusion and attention mechanism is characterized in that: Improvements to the YOLOv8 network include: The path aggregation network in the neck network is replaced by a three-weight feature fusion network TWFFPN; the TWFFPN mainly adds a small target layer P2 layer to the structure of BiFPN, and changes the feature fusion method of the P2 layer and the P5 layer, so that the output feature map of the P2 layer is obtained by fusion of the initial features and intermediate features from the current layer and the intermediate features from the P3 layer; the output feature maps of the P3 layer and the P4 layer are both obtained by fusion of the initial features and intermediate features from the current layer and the output feature map from the previous layer; the output feature map of the P5 layer is obtained by fusion of the initial features from the current layer, the intermediate features of the P4 layer and the output feature map of the P4 layer; Correspondingly, the output feature maps of the P3 layer, P4 layer and P5 layer are used as the three detection heads of the remote sensing vehicle target detection network.

4. A remote sensing vehicle target detection method based on feature fusion and attention mechanism according to claim 3, characterized in that: Output feature map of P2 layer Output feature map of P3 layer Output feature map of P4 layer And the output feature map of the P5 layer They are: Among them, ω i ' j Represents learnable weights, i=2,3,4,5,j=1,2,3,ε n Indicates the preset value, n=1,2,3,4,P i in represents the initial features of the i-th layer, P i td Represents the intermediate features of the i-th layer, Resize represents an upsampling operation or a downsampling operation, and Conv represents a convolution operation.

5. The remote sensing vehicle target detection method based on feature fusion and attention mechanism according to claim 1 is characterized in that: In the process of training the remote sensing vehicle target detection network, NWD loss is used as the bounding box loss.

Images