A remote sensing image target detection method based on feature fusion

Abstract

The present invention provides a remote sensing image target detection method based on feature fusion, and the scheme is as follows: (1) preprocessing the remote sensing image target detection data set, and dividing it into a training set, a validation set and a test set; (2) constructing a pyramid feature refinement unit, and performing multi-scale learning on features with three paths of different receptive fields; (3) constructing a pyramid feature refinement module, and replacing the C3 module in the path aggregation network with it, so that the network can perform more sufficient and effective feature fusion; (4) constructing a lightweight decoupled detection head, and replacing the original detection head of YOLOv5 with it, and effectively alleviating the conflict problem between the classification task and the regression task under the condition of introducing a small number of model parameters, and improving the detection accuracy; (5) training and testing the improved network. The present invention can achieve good detection effect on remote sensing image data sets and provide support for remote sensing intelligent interpretation.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and in particular to a remote sensing image target detection method based on feature fusion. Background Art

[0002] With the continuous development of aerospace remote sensing technology, remote sensing image data has been continuously enriched. The detection, identification and positioning of targets in remote sensing images, such as vehicles, ships and aircraft, have been widely used in disaster control, traffic planning, national defense security and other fields.

[0003] In order to make full use of the semantic information of deep feature maps and the texture information of shallow feature maps to improve detection accuracy, existing algorithms often use FPN or PANet to complete the fusion of multi-scale feature maps. Both FPN and PANet use single-scale convolution to enhance the fusion effect, ignoring the multi-scale expression characteristics of multi-layer features, resulting in poor fusion effect and further affecting detection accuracy. In addition, the conflict between classification tasks and regression tasks is also a well-known problem in the field of target detection. The introduction of decoupled detection heads effectively alleviates this problem and improves detection accuracy. However, while the existing decoupled detection heads improve accuracy, they also greatly increase the model's trainable parameters and floating-point operations, resulting in slower model detection speed. Summary of the invention

[0004] In order to solve the above problems, the present invention takes YOLOv5 as a benchmark, proposes a remote sensing image target detection method based on feature fusion, and uses a public remote sensing image target detection data set (such as DOTA) to train and test the improved network. First, a pyramid feature refinement unit is constructed, which enhances the representation ability of features in a multi-scale learning manner, and a pyramid feature refinement module is constructed by stacking pyramid feature refinement units. The pyramid feature refinement module replaces C3 in PANet to enhance the fusion effect of multi-scale feature maps; secondly, a lightweight decoupled detection head is constructed, which effectively alleviates the conflict problem between classification tasks and regression tasks while introducing a small number of model parameters, and significantly improves the detection accuracy. In order to further reduce the model parameters, the Ghost Shuffle convolution is introduced, and the convolution replaces the 3×3 standard convolution in the pyramid feature refinement unit and the lightweight decoupled detection head.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A remote sensing image target detection method based on feature fusion, comprising:

[0007] Step 1: Obtain a remote sensing image target detection dataset and preprocess the data; the preprocessing includes format conversion and data set division; specifically, the dataset is randomly divided into a training set, a validation set, and a test set; the image label is converted into a format suitable for the YOLOv5 network;

[0008] Step 2: Improve the YOLOv5 network, including:

[0009] The C3 module in the feature fusion network PANet is replaced by the Pyramid Feature Refinement Module (PFR Module). Compared with the C3 structure of PANet, which only enhances the fusion effect of the feature map after the Concat operation through a single-scale 3×3 standard convolution, ignoring the multi-scale expression characteristics of multi-layer features, the Pyramid Feature Refinement Module is based on the idea of ​​multi-scale learning, and learns the feature map after the Concat operation with receptive fields of different sizes to enhance the representation ability of the fused features.

[0010] The detection head is replaced with the constructed Lightweight Decoupled DetectorHead. Compared with the original detection head of YOLOv5, which uses shared parameters for classification and regression, the Lightweight Decoupled DetectorHead uses two parallel branches, each of which learns parameters independently, alleviating the conflict between classification and regression tasks.

[0011] Step 3: Train the improved YOLOv5 network based on the divided data set and save the best model;

[0012] Step 4: Test the saved best model based on the divided data set to obtain the remote sensing image target detection results.

[0013] Furthermore, the pyramid feature refinement module is composed of three types of operations: standard convolution, pyramid feature refinement unit (Pyramid Feature Refinement Unit, PFR-Unit) and Concat, where each layer of standard convolution includes (1×1Conv+BN+SiLU). After the backbone network CSPDarknet-53 completes feature extraction, three feature maps C2∈R of different scales are obtained. 80×80×256 ,C3∈R 40×40×512 , and C4∈R 20×20×1024 , which is used as the input of the feature fusion network. Taking the fusion of feature maps C3 and C4 as an example, the fusion process of two feature maps of different scales is as follows: First, the high-dimensional feature map C4 is reduced in dimension and upsampled to obtain a size of C4 ′ ∈R 40×40×512Feature map, and then the feature maps C3 and C4 are concatenated through the Concat operation ′ The fusion of feature maps of different scales is completed by splicing in the channel dimension, and the feature map with a size of 40×40×1024 is output. Finally, the pyramid feature refinement module is used to enhance the fusion effect, and the feature map with a size of 40×40×512 is output.

[0014] Furthermore, the pyramid feature refinement module includes two branch paths, one including only standard convolution, and the other consisting of standard convolution and pyramid feature refinement units, wherein each layer of standard convolution includes 1×1Conv+BN+SiLU.

[0015] Furthermore, the pyramid feature refinement unit is composed of standard convolution and GSConv, wherein each layer of standard convolution includes 1×1Conv+BN+SiLU, and GSConv includes 3×3GSConv+BN+SiLU; through experimental discussion, the number of GSConv in the pyramid refinement unit is set to 3, wherein the receptive field of 2 series-connected 3×3GSConv is equivalent to a GSconv with a convolution kernel size of 5×5, and the receptive field of 3 series-connected 3×3GSConv is equivalent to a GSconv with a convolution kernel size of 7×7. The pyramid refinement unit uses receptive fields of different sizes to perform multi-scale learning on feature maps that are simply fused using the Concat operation, enhance the representation ability of the fused feature maps, and use them as input to the next layer and the detection head.

[0016] Furthermore, the specific operation process of the pyramid feature refinement module is as follows: first, two 1×1 standard convolutions are used to divide the input into two parts with the same number of channels. This operation not only reduces the amount of network calculation and the occupation of video memory, but also ensures the performance of the network. Then, one of the branches is passed through three pyramid feature refinement units in turn to enhance the feature representation ability. Finally, the feature maps on the two branches are spliced ​​and convolved.

[0017] Furthermore, the specific operation process of the pyramid feature refinement unit is as follows: first, the nonlinearity of the network is increased by 1×1 standard convolution, so that the network can express more complex features, and then the fused feature map is multi-scale learned through the paths whose receptive fields are equivalent to 3×3GSConv, 5×5GSConv, and 7×7GSConv. The output feature map size of each path is 40×40×512, and then Concat and 1×1 standard convolution operations are connected, and the final output is 40×40×512.

[0018] Furthermore, the lightweight decoupled detection head is composed of standard convolution and GSConv, wherein the standard convolution includes 1×1Conv+BN+SiLU, and the GSConv includes 3×3GSConv+BN+SiLU; the lightweight decoupled detection head performs a convolution operation on the input detection feature map through the shared 1×1 standard convolution and 3×3GSConv, and then connects two parallel branches, one for the classification task and the other for the regression task.

[0019] Furthermore, the feature fusion network outputs three detection feature maps of sizes 80×80×256, 40×40×512, and 20×20×1024, which are used to predict large, medium, and small targets, respectively; a lightweight decoupled detection head is connected after each detection feature map; in the lightweight decoupled detection head, a 1×1 standard convolution is first used to reduce the feature map dimensions from 256, 512, and 1024 to 256, respectively, followed by a shared 3×3GSConv, and the outputs are 80×80×256, 40×40×256, and 20×20×256, respectively, and finally there are two parallel branches, one for the task division and the other for the regression task.

[0020] Furthermore, in step 3, during training, the number of epochs is set to 300, the batch-size is 16, the initial learning rate is 0.01, the weight delay is set to 0.0005, the momentum is set to 0.9, and the optimization method is the stochastic gradient descent algorithm. After 300 iterations of training, the loss value and accuracy are both stable, and the best model parameters are saved.

[0021] Furthermore, the testing of the best model saved based on the divided data set includes:

[0022] The preprocessed remote sensing image is sent to the CSPDarknet-53 feature extraction network, and after multiple downsampling operations, multiple feature maps of different scales are obtained;

[0023] The feature fusion network based on the improved YOLOv5 network performs feature fusion on the feature maps of different scales and outputs multiple detection feature maps of different scales.

[0024] The output detection feature maps of multiple different scales are sent to the detection network of the improved YOLOv5 network for target recognition and positioning.

[0025] Furthermore, the Ghost Shuffle convolution is introduced, and this convolution is used to replace the 3×3 standard convolution in the pyramid feature refinement unit and the lightweight decoupled detection head.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The invention improves the YOLOv5 network and has the following advantages compared with the original YOLOv5 algorithm: a pyramid feature refinement module is constructed to replace C3 in PANet, so that the network can perform more sufficient and effective feature fusion, and the utilization rate of the original features is improved; a lightweight decoupled detection head is constructed, and the conflict between the classification task and the regression task is effectively alleviated when a small number of model parameters are introduced, thereby improving the accuracy of remote sensing image target detection; and the parameters of the model are further reduced by introducing GSConv. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of a flow chart of a remote sensing image target detection method based on feature fusion according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a network structure of a remote sensing image target detection method based on feature fusion according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a pyramid feature refinement module according to an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a pyramid feature refinement unit according to an embodiment of the present invention;

[0032] Figure 5 It is a schematic diagram of the structure of a lightweight decoupling detection head according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0034] like Figure 1 As shown, a remote sensing image target detection method based on feature fusion includes:

[0035] Step 1: Obtain a remote sensing image target detection dataset and preprocess the data; the preprocessing includes: randomly dividing the dataset into a training set, a validation set, and a test set; converting the image labels into a format that is suitable for the YOLOv5 network. As an implementable method, the remote sensing image target detection dataset uses DOTA.

[0036] Step 2: Improve the YOLOv5 network, including: replacing the C3 module in the feature fusion network PANet with the pyramid feature refinement module, and replacing the detection head with the constructed lightweight decoupled detection head; the improved YOLOv5 network is as follows Figure 2 As shown;

[0037] Step 3: Train the improved YOLOv5 network based on the divided data set and save the best model (specifically save the parameters corresponding to the best model);

[0038] Step 4: Test the saved best model based on the divided data set to obtain the remote sensing image target detection results.

[0039] Furthermore, in the step 2, the specific method of replacing the C3 module in the feature fusion network PANet with the pyramid feature refinement module is:

[0040] The pyramid feature refinement module is composed of three types of operations: standard convolution, pyramid feature refinement unit and Concat. The specific structure of the pyramid feature refinement module is as follows: Figure 3 As shown, the pyramid feature refinement unit is Figure 4 As shown in the figure, each layer of standard convolution includes (1×1Conv+BN+SiLU). The pyramid feature refinement unit consists of standard convolution and GSConv, each layer of standard convolution includes (1×1Conv+BN+SiLU), and GSConv includes (3×3GSConv+BN+SiLU). Through experimental discussion, the number of GSConv in the pyramid refinement unit is set to 3, where the receptive field of 2 series-connected 3×3GSConv is equivalent to a GSconv with a convolution kernel size of 5×5, and the receptive field of 3 series-connected 3×3GSConv is equivalent to a GSconv with a convolution kernel size of 7×7. The pyramid refinement unit uses receptive fields of different sizes to perform multi-scale learning on feature maps that are simply fused using the Concat operation, enhance the representation ability of the fused feature maps, and use them as the input of the next layer and the detection head.

[0041] After the backbone network CSPDarknet-53 completes feature extraction, three feature maps C2∈R of different scales are obtained 80×80×256 ,C3∈R 40×40×512 , and C4∈R 20×20×1024 , which is used as the input of the feature fusion network. Taking the fusion of feature maps C3 and C4 as an example, the fusion process of two feature maps of different scales is as follows: First, the high-dimensional feature map C4 is reduced in dimension and upsampled to obtain a size of C4 ′ ∈R 40×40×512 Feature map, and then the feature maps C3 and C4 are concatenated through the Concat operation ′ The fusion of feature maps of different scales is completed by splicing in the channel dimension, and the feature map with a size of 40×40×1024 is output. Finally, the pyramid refinement module is used to enhance the fusion effect, and the feature map with a size of 40×40×512 is output.

[0042] The specific operation process of the pyramid feature refinement module is as follows: first, use two 1×1 standard convolutions to divide the input into two parts with the same number of channels. This operation not only reduces the amount of network calculation and the occupation of video memory, but also ensures the performance of the network. Then, one of the branches is passed through three pyramid feature refinement units in turn to enhance the feature representation ability. Finally, the feature maps on the two branches are spliced ​​and convolved.

[0043] The specific operation process of the pyramid feature refinement unit is as follows: first, the nonlinearity of the network is increased through a 1×1 standard convolution, so that the network can express more complex features. Then, the fused feature map is subjected to multi-scale learning through paths that are equivalent to 3×3GSConv, 5×5GSConv, and 7×7GSConv. The output feature map size of each path is 40×40×512. After that, Concat and 1×1 standard convolution operations are connected, and the final output is 40×40×512.

[0044] Furthermore, in step 2, the specific method of replacing the detection head with the constructed lightweight decoupling detection head is:

[0045] The lightweight decoupled detection head model structure consists of standard convolution and GSConv, where the standard convolution includes (1×1Conv+BN+SiLU) and the GSConv includes (3×3GSConv+BN+SiLU). Figure 5 shown.

[0046] After the feature fusion network, three detection feature maps of size 80×80×256, 40×40×512, and 20×20×1024 are output, which are used to predict large, medium, and small targets respectively. A lightweight decoupled detection head is connected after each detection feature map. In the lightweight decoupled detection head, a 1×1 standard convolution is first used to reduce the feature map dimensions from 256, 512, and 1024 to 256, respectively. This operation is mainly to reduce model parameters and computation. Next is a shared 3×3GSConv, with outputs of 80×80×256, 40×40×256, and 20×20×256, respectively. Finally, there are two parallel branches, one for classification tasks and the other for regression tasks.

[0047] Furthermore, in step 3, during training, the number of iterations epoch is set to 300, the batch-size is 16, the initial learning rate is 0.01, the weight delay is set to 0.0005, the momentum is set to 0.9, and the optimization method is the stochastic gradient descent algorithm. After 300 iterations of training, the loss value and accuracy are both stable, and the best model parameters at this time are saved.

[0048] Furthermore, the testing of the best model saved based on the divided data set includes:

[0049] The preprocessed remote sensing image is sent to the CSPDarknet-53 feature extraction network, and after multiple downsampling operations, multiple feature maps of different scales are obtained;

[0050] The feature fusion network based on the improved YOLOv5 network performs feature fusion on the feature maps of different scales and outputs multiple detection feature maps of different scales.

[0051] The output detection feature maps of multiple different scales are sent to the detection network of the improved YOLOv5 network for target recognition and positioning.

[0052] Furthermore, it also includes:

[0053] The Ghost Shuffle convolution is introduced and used to replace the 3×3 standard convolution in the pyramid feature refinement module and the lightweight decoupled detection head.

[0054] It is worth noting that the improved method proposed in the present invention is applicable to situations where targets are small and densely arranged, and improves detection accuracy while ensuring speed.

[0055] In summary, the present invention improves the YOLOv5 network, and has the following advantages compared with the original YOLOv5 algorithm: a pyramid feature refinement module is constructed and used to replace C3 in PANet, so that the network can perform more sufficient and effective feature fusion, and the utilization rate of the original features is improved; a lightweight decoupled detection head is constructed, and the conflict between the classification task and the regression task is effectively alleviated by introducing a small number of model parameters, thereby improving the accuracy of remote sensing image target detection; and the parameters of the model are further reduced by introducing GSConv.

[0056] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A remote sensing image target detection method based on feature fusion, characterized in that: include: Step 1: Obtain remote sensing image target detection dataset and preprocess the data; The preprocessing includes format conversion and data set division; Step 2: Improve the YOLOv5 network, including: replacing the C3 module in the feature fusion network PANet with a pyramid feature refinement module, and replacing the detection head with a constructed lightweight decoupled detection head; The pyramid feature refinement module is composed of three types of operations: standard convolution, pyramid feature refinement unit and Concat. The pyramid feature refinement module contains two branch paths, one of which only contains standard convolution and the other consists of standard convolution and pyramid feature refinement unit, wherein each layer of standard convolution includes 1×1Conv+BN+SiLU. The specific operation process of the pyramid feature refinement module is as follows: first, two 1×1 standard convolutions are used to divide the input into two parts with the same number of channels, then one of the branches is sequentially passed through three pyramid feature refinement units to enhance the representation ability of the features, and finally the feature maps on the two branches are spliced ​​and convolved; The pyramid feature refinement unit is composed of standard convolution and GSConv, wherein each layer of standard convolution includes 1×1Conv+BN+SiLU, and GSConv includes 3×3GSConv+BN+SiLU; the pyramid feature refinement unit performs GSConv operation on the input feature map through three paths, and each path realizes multi-scale learning of features by connecting different numbers of 3×3GSConv in series; The specific operation process of the pyramid feature refinement unit is as follows: first, the nonlinearity of the network is increased by 1×1 standard convolution, so that the network can express more complex features, and then the fused feature map is multi-scaled learned through paths with receptive field sizes equivalent to 3×3GSCon v, 5×5GSConv, and 7×7GSConv. The output feature map size of each path is 40×40×512, and then the Concat and 1×1 standard convolution operations are connected, and the final output is 40×40×512; The lightweight decoupled detection head consists of standard convolution and GSConv, where the standard convolution includes 1×1Conv+BN+SiLU, GSConv includes 3×3GSConv+BN+SiLU; the lightweight decoupled detection head performs convolution operations on the input detection feature map through shared 1×1 standard convolution and 3×3GSConv, and then connects two parallel branches, one for classification task and the other for regression task; after the feature fusion network, three detection feature maps of size 80×80×256, 40×40×512, and 20×20×1024 are output, which are used to predict large, medium and small targets respectively; a lightweight decoupled detection head is connected after each detection feature map; in the lightweight decoupled detection head, first, a 1x1 standard convolution is used to reduce the feature map dimensions from 256, 512, and 1024 to 256, respectively, followed by a shared 3×3GSConv, and the outputs are 80×80×256, 40×40×256, and 20×20×256, respectively, and finally two parallel branches, one for classification task and the other for regression task; Step 3: Train the improved YOLOv5 network based on the divided data set and save the best model; Step 4: Test the saved best model based on the divided data set to obtain the remote sensing image target detection results.

2. The method for remote sensing image target detection based on feature fusion according to claim 1, characterized in that: The testing of the best model saved based on the divided data set includes: The preprocessed remote sensing image is sent to the CSPDarknet-53 feature extraction network, and after multiple downsampling operations, multiple feature maps of different scales are obtained; The feature fusion network based on the improved YOLOv5 network performs feature fusion on the feature maps of different scales and outputs multiple detection feature maps of different scales. The output detection feature maps of multiple different scales are sent to the detection network of the improved YOLOv5 network for target recognition and positioning.

3. The method for remote sensing image target detection based on feature fusion according to claim 1, characterized in that: Also includes: The Ghost Shuffle convolution is introduced and used to replace the 3×3 standard convolution in the pyramid feature refinement module and the lightweight decoupled detection head.

Images