Remote sensing target detection method based on target feature enhanced multi-scale fusion network

Abstract

The invention discloses a remote sensing target detection method based on a target feature enhanced multi-scale fusion network, and the method comprises the steps: constructing a multi-layer feature aggregation module as a main feature extraction module, and better extracting the target features of each category; a multi-channel feature fusion module is constructed, the network layer parameter quantity is reduced, a cross-stage hierarchical structure is adopted to design an MCFCSP module, richer gradient combination and feature expression are realized, and the network feature fusion capability is enhanced; by adopting a top-down and bottom-up feature fusion strategy, information interaction between different levels of features is promoted; fractional Fourier transform convolution is constructed, and parameters are adjusted according to the characteristics of small targets and the change of scenes to adapt to small targets of different types, different sizes and different complexities. The method provided by the invention has the characteristics of high detection precision, low calculation complexity, strong feature representation capability and the like, can effectively improve the detection precision of the small target, and further improves the detection performance of the small target of the remote sensing image.

Description

Technical Field

[0001] The invention relates to a remote sensing target detection method based on a target feature enhanced multi-scale fusion network, and belongs to the technical field of remote sensing target detection. Background Art

[0002] In recent years, the rapid development of artificial satellite and aerial photography technology has greatly promoted the continuous expansion of the application scope of remote sensing technology. Remote sensing images are playing an increasingly important role in both daily life and military fields.

[0003] With the rapid development of deep learning-based target detection algorithms such as R-CNN, SSD, RetinaNet and YOLO series in the field of remote sensing image processing, the target detection effect of remote sensing images has been significantly improved, but there are still many limitations when dealing with small target detection in complex backgrounds. To this end, researchers are looking for solutions from multiple angles.

[0004] Zhang Yin et al. (Zhang Yin, Zhu Guiyi, Shi Tianjun et al. Small target detection in remote sensing images based on feature fusion and attention [J]. Acta Optica Sinica, 2022, 42(24): 140-150) proposed a novel "cascade attention" method, which enhances the ability to capture small targets by fusing multiple receptive field information in the underlying feature map. Zhang Kuo et al. (Zhang Kuo, Chen Zhangjin, Qiao Dong et al. Real-time detection of remote sensing images based on receptive field and feature enhancement [J]. Progress in Laser & Optoelectronics, 2023, 60(02): 331-340) proposed a lightweight neural network based on receptive field and feature enhancement, which uses deep separable convolution to reduce the number of parameters and speed up detection, and introduces modules such as receptive field enhancement and attention mechanism to improve detection accuracy. Qu et al. (QUJ S, SU C, ZHANG ZW, et al. An Improved SSD Network for Small Object Detection Based on Dilated Convolution and Feature Fusion[J].IEEE Access, 2020, 8: 82832-82843) proposed a new feature fusion method based on dilated convolution, which effectively improved the detection effect of small targets. Xue Junda et al. (Xue Junda, Zhu Jiajia, Zhang Jing, et al. Target detection in optical remote sensing images based on FFC-SSD model[J]. Acta Optica Sinica, 2022, 42(12):138-148) proposed a FFC-SSD model based on target box grouping clustering and efficient feature fusion. The anti-pooling strategy is used to reduce parameters and computation, enhance shallow features and fuse them with high-level semantics, and improve the accuracy of feature map output. Wu Luobing et al. (Wu Luobing, Gu Yuhai, Wu Wenhao, et al. Remote sensing rotating target detection based on multi-scale feature extraction [J]. Laser & Optoelectronics Progress, 2023, 60(12): 453-461) proposed a rotating remote sensing target detection algorithm based on multi-scale feature extraction. The receptive field expansion module was designed by combining the dilated convolution and an adaptive feature fusion structure was embedded to improve the model's detection capability for multi-scale targets in complex environments.Jiang et al. (Jiang S, Yao W, Wang MS, et al. An Optimized Deep Neural Network Detecting Small and NarrowRectangular Objects in Google Earth Images[J]. IEEE Journal of SelectedTopics in Applied Earth Observations and Remote Sensing, 2020, 13: 1068-1081) proposed an optimal deep neural network model based on deep learning, which combines the object features in the image with the construction of the original data, effectively improving the detection performance of small-sized objects. Teng et al. (Teng Z, Duan Y, Liu YN, Zhang BP, et al. Global to Local: Clip-LSTM-Based Object Detection From Remote Sensing Images, IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-13, 2022) proposed a global-to-local object detection network model, which eliminates the influence of complex backgrounds through global context clues extracted by multi-scale perception modules, and designs adaptive anchor modules to alleviate semantic scale differences. Zhao et al. (Zhao, Q. Wang, Y. Wu, Q. Cao and Q. Ran, Target Detection Model Distillation Using Feature Transition and Label Registration for Remote Sensing Imagery, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 5416-5426, 2022) designed a lightweight attention network that uses multi-scale feature transformation and label registration methods to improve the model learning ability, thereby improving the detection accuracy of small targets in remote sensing images.

[0005] The above methods have different improvements, but they still show certain limitations when dealing with small target detection. Especially when facing remote sensing images containing complex scene information, these algorithms may not be able to fully extract features, and the details of small targets are often ignored or completely missed. In addition, although the feature pyramid fusion technology solves the problem of scale difference detection, there is still a lot of room for improvement in the fusion of high-level semantic information and low-level spatial information.

[0006] Based on the above analysis, due to the large scale span, small target size, unbalanced distribution and dense distribution of remote sensing images, there are often problems of false detection or missed detection when performing target detection. Therefore, how to improve the target detection effect of remote sensing images, especially the detection effect of small-sized targets, has become an urgent problem to be solved. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a remote sensing target detection method based on a target feature enhanced multi-scale fusion network, which solves the problem of difficulty in detecting small targets in remote sensing images under complex backgrounds and realizes accurate detection of small targets under different environmental conditions.

[0008] The present invention adopts the following technical solutions to solve the above technical problems: The remote sensing target detection method based on target feature enhanced multi-scale fusion network includes the following steps: Step 1: obtain a remote sensing image dataset, annotate the dataset, and divide the annotated dataset into a training set and a test set; Step 2: construct a target feature enhanced multi-scale fusion network model, including a backbone network, a neck network, and a detection head; A multi-layer feature aggregation module and MP module are constructed in the backbone network to extract features from the input image and obtain feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image respectively. A multi-channel feature fusion module is constructed in the neck network. Based on the multi-channel feature fusion module, a cross-stage network MCFCSP module is designed. The top-down and bottom-up feature fusion strategies are adopted to fuse feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, and the fused feature maps with sizes of 1 / 8, 1 / 16 and 1 / 32 of the input image are obtained. Introducing fractional Fourier transform into the detection head to construct fractional Fourier transform convolution for remote sensing target detection; Step 3, using the training set to train the target feature enhanced multi-scale fusion network model constructed in step 2 to obtain the optimal parameter network model, that is, the trained remote sensing target detection model; Step 4: Input the test set into the trained remote sensing target detection model and output the remote sensing target detection results.

[0009] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects: 1. The present invention designs a multi-layer feature aggregation module (MFAM) in the backbone network to capture the shape and structure of the target, thereby improving the feature extraction capability of convolution and better extracting target features of various categories.

[0010] 2. The present invention constructs a multi-channel feature fusion module (MCF) to perform fine processing on the neck, and adopts a cross-stage aggregation method to design the MCFCSP module to achieve richer gradient combinations and feature expressions, which can effectively reduce the amount of network calculations and enhance the feature fusion capability of the network.

[0011] 3. The present invention constructs a bidirectional multi-scale feature fusion module (BMFM) and adopts top-down and bottom-up feature fusion strategies to effectively promote information interaction between features at different levels. It not only enhances the expression ability of each level, but also makes full use of feature information at different scales, thereby improving the performance of target detection.

[0012] 4. The present invention introduces fractional Fourier transform to construct fractional Fourier transform convolution (FrFTConv), and adjusts the parameters of fractional Fourier transform according to the characteristics of small targets and changes in scenes to adapt to small targets of different types, sizes and complexities, improve the accuracy and robustness of small target detection, and thus better realize the precise detection of small targets under different environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a remote sensing target detection method based on a target feature enhanced multi-scale fusion network of the present invention; Figure 2 It is the overall framework diagram of the target feature enhanced multi-scale fusion network model (ETF-Mnet) proposed in the present invention; Figure 3 It is a structural diagram of the multi-layer feature aggregation module (MFAM) proposed in the present invention; Figure 4 It is a structural diagram of the multi-channel feature fusion module (MCF) proposed in the present invention; Figure 5 It is a structural diagram of the MCFCSP module proposed by the present invention; Figure 6 It is a structural diagram of the bidirectional multi-scale feature fusion structure (BMFM) proposed in the present invention; Figure 7 It is a structural diagram of the fractional Fourier transform convolution module (FrFTConv) proposed in the present invention; Figure 8 It is the Fn network structure diagram proposed by the present invention; Fig. 9 : is a comparison diagram of the detection effect of the ablation experiment of ETF-Mnet provided in the embodiment on the DOTA dataset and the NWPU VHR-10 dataset with YOLOv7 as the benchmark model, wherein (a) is the original image, (b) is the benchmark model, (c) is the ELAN module in the backbone replaced by the MFAM module, (d) is the ELAN module in the neck replaced by the MCFCSP module, (e) is the addition of BMFM on the basis of YOLOv7, (f) is the replacement of the detection head of YOLOv7 with the FrFTConv module, and (g) is ETF-Mnet; Fig.10 A comparison diagram of target detection effects of ETF-Mnet and YOLOv5s, YOLOv7 and YOLOv8s algorithms provided in the embodiment, wherein (a) is the original image, (b) is YOLOv5s, (c) is YOLOv7, (d) is YOLOv8s, and (e) is ETF-Mnet; Fig.11 A comparison diagram of target detection effects of ETF-Mnet, YOLOv5s, YOLOv7 and YOLOv8s algorithms provided in the embodiment under complex and dense target backgrounds, wherein (a) is the original image, (b) is YOLOv5s, (c) is YOLOv7, (d) is YOLOv8s, and (e) is ETF-Mnet; Fig.12 A comparison diagram of target detection missed detection effects of ETF-Mnet and YOLOv5s, YOLOv7 and YOLOv8s algorithms provided in the embodiment, wherein (a) is the original image, (b) is YOLOv5s, (c) is YOLOv7, (d) is YOLOv8s, and (e) is ETF-Mnet; Fig.13 A comparison diagram of the target detection misdetection effects of ETF-Mnet and YOLOv5s, YOLOv7 and YOLOv8s algorithms provided in the embodiment, where (a) is the original image, (b) is YOLOv5s, (c) is YOLOv7, (d) is YOLOv8s, and (e) is ETF-Mnet. DETAILED DESCRIPTION

[0014] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0015] like Figure 1 As shown, the present invention proposes a remote sensing target detection method based on target feature enhanced multi-scale fusion network, comprising the following steps: Step 1: Obtain a remote sensing image dataset, annotate the dataset, create a remote sensing image dataset in YOLO format, and divide it into a training set and a test set; Step 2: Build ETF-Mnet, which mainly includes three parts: backbone network, neck network and detection head; Step 3: Input the data set obtained in step 1 into the ETF-Mnet network constructed in step 2 for training and testing to obtain the optimal parameter model; Step 4: Use the optimal parameter model obtained in step 3 to perform target detection on the remote sensing image input in step 1.

[0016] The steps for step 1 are: Obtain DOTA and NWPU VHR-10 high-resolution remote sensing image datasets, of which the DOTA dataset contains 2806 images. ~ Remote sensing images with varying pixels, with a total of 15 annotated categories, namely ports, airplanes, ships, roundabouts, storage tanks, swimming pools, small vehicles, tennis courts, bridges, basketball courts, helicopters, athletic fields, large vehicles, football fields, and baseball fields. There are a total of 188,282 instances in the dataset, mostly medium-sized and small instances. The NWPU VHR-10 dataset was released by Northwestern Polytechnical University. It has 650 images containing targets and 150 background images without targets, totaling 800. There are a total of 3,896 targets in the image, including airplanes, ships, storage tanks, baseball fields, tennis courts, basketball courts, athletic fields, ports, bridges, and cars, totaling 10 categories. The annotated images are divided into training and test sets at a ratio of 8:2, and then image preprocessing is performed to cut the remote sensing images into pieces of size for later network training and testing.

[0017] In step 2, build ETF-Mnet, such as Figure 2 As shown, the steps including the backbone network, the neck network and the detection head are: Step 2.1: Build the network input port, perform data enhancement on the input image, and input the 640×640 image into the backbone network. The backbone network consists of several CBH (Conv+BN+Mish), several MFAM modules, and several MP modules. The specific steps are: The image undergoes 1 3×3 convolution (Conv) with a stride of 1, batch normalization (BN) and Mish activation function, and then 1 3×3 convolution with a stride of 2, batch normalization and Mish activation function, repeated twice, and then the MFAM module outputs a 1 / 4 feature map C2 with a channel number of 256. Then, the MP module is used to reduce the size and number of channels, and then the MFAM module is used to output a 1 / 8 feature map C3 with a channel number of 512. The MP module is used to reduce the size and number of channels, and then the MFAM module is used to output a 1 / 16 feature map C4 with a channel number of 1024. The MP module is used to reduce the size and number of channels, and then the MFAM module is used to output a 1 / 32 feature map C5 with a channel number of 1024.

[0018] The MP module reduces the image size and number of channels through Maxpool and CBH, and fuses features through a concatenation operation (ConCat). The specific steps are as follows: first, the input image of the module is subjected to maximum pooling, and then a 1×1 convolution with a step size of 1, batch normalization, and Mish activation function are performed to obtain feature map a; then, the input image of the module is subjected to a 1×1 convolution with a step size of 1, batch normalization, and Mish activation function, and then a 3×3 convolution with a step size of 2, batch normalization, and Mish activation function to obtain feature map b. Feature maps a and b are concatenated to obtain the final output image of the module.

[0019] like Figure 3 As shown in the figure, the MFAM module is composed of multiple residual cascade layers consisting of several CBH and ODCBH convolution layers with a convolution kernel of 1. The CBH convolution layer with a convolution kernel of 1 includes three parts: 1×1 convolution, batch normalization, and Mish activation function, which can better retain gradient information and improve the feature representation ability of the model. The convolution, batch normalization and Mish activation function are three parts. Convolution kernels of multiple scales are introduced, and the features of the convolution kernels are adaptively combined for feature extraction. The target features of different scales are captured, and more comprehensive and richer semantic information is provided to improve the detection accuracy of small targets. In addition, after being processed by multiple convolution modules, the network's ability to express nonlinear features is enhanced, and it can better adapt to targets of different scales and shapes. The input feature map of the module first passes through the CBH convolution layer, and then is spliced ​​with the feature map that has passed through the ODCBH convolution layer twice, and then is spliced ​​with the feature map that has passed through the ODCBH convolution layer twice again to obtain the final output image of the module. OMNI-Dimensional Dynamic Convolution (ODConv) learns attention in the four dimensions of the convolution kernel (space, input channel, output channel, number of kernels) through a multi-dimensional attention mechanism to enhance feature extraction capabilities.

[0020] Step 2.2: Construct the neck network, which is composed of a spatial pyramid pooling module (Spatial Pyramid Pooling Convolutional Spatial Pyramid Clustering, SPPCSPC) combined with cross-stage partial convolution, several MCF modules, several MCFCSP modules, several MP modules and upsampling operations. As the feature fusion part, the neck first adjusts the channels of the feature maps (C2 / C3 / C4 / C5) of the four scales output by the backbone network, and then performs upsampling and splicing operations to generate fused feature maps of four scales (P2 / P3 / P4 / P5). Then, the fused feature maps of these four scales are downsampled and spliced ​​using the maximum pooling module to obtain fused feature maps of three scales (N3 / N4 / N5). The specific steps are: pass the feature map C5 obtained in step 2.1 through the SPPCSPC module to obtain the feature map P5, then pass through the MCF module and upsampling operation, concatenate the obtained feature map P51 with the feature map obtained by the MCF module of C4, and then pass through the MCFCSP module to obtain a 1 / 16 feature map P4 with a channel number of 256, then pass through the MCF module and upsampling operation, concatenate the obtained feature map P41 with the feature map obtained by the MCF module of C3, and then pass through the MCFCSP module to obtain a 1 / 8 feature map P3 with a channel number of 128, then pass through the MCF module and upsampling operation, concatenate the obtained feature map P31 with the feature map obtained by the MCF module of C2, and then pass through the MCFCSP module to obtain a 1 / 4 feature map P2 with a channel number of 64. P2 passes through the FRB module and the MP module, and is spliced ​​with the characteristic map obtained by P3 through the FRB module, and then passes through the MCFCSP module to obtain the characteristic map N3, and then passes through the MP module and is spliced ​​with P4, and then passes through the MCFCSP module to obtain the characteristic map N4, and then passes through the MP module and is spliced ​​with P5, and then passes through the MCFCSP module to obtain the characteristic map N5.

[0021] Although the feature representation capabilities of P2 and P3 are enhanced by feature fusion, these layers still lack deep semantic information. To solve this problem, the present invention designs a feature refinement block (FRB). This module slices the input features and uses and Convolution is used to capture information in different directions, thereby enhancing the expressiveness of features. After point-by-point convolution, the module is connected to the original features through residual connections, which not only alleviates the gradient vanishing problem, but also enhances the feature transfer capability.

[0022] The SPPCSPC module is specifically as follows: first, the module input image undergoes a CBH operation with a convolution kernel of 1 to obtain a feature map c, then undergoes a CBH operation with a convolution kernel of 3, and then undergoes a CBH operation with a convolution kernel of 1 to obtain a feature map d, performs three maximum pooling operations on the feature map d in parallel, outputs three feature maps, adds these three feature maps to the feature map d, undergoes a CBH operation with a convolution kernel of 1, and then undergoes a CBH operation with a convolution kernel of 3, adds the obtained feature map to the feature map c, and finally, undergoes a CBH operation with a convolution kernel of 1 to obtain the final output image of the module.

[0023] like Figure 4 As shown in the figure, the MCF module is specifically as follows: if a feature map with the number of channels X1 is input into the module, the number of channels in the feature map is first reduced to one-fourth of the output channel number X2 through point-by-point convolution, and then the image is processed using three 3×3 convolutions and the SE attention mechanism respectively, the three processed feature maps are combined with the feature map before processing, and the channels of the data are mixed through the CA attention mechanism and channel mixing (Shuffle) operation to obtain a feature map with the number of channels X2.

[0024] The SE module focuses on channel information, which can balance the network channels well and improve performance by adding a small amount of calculation. The Coordinate Attention (CA) module introduces position encoding or position vectors, which enables the model to better capture the details and local features of small targets and improve the quality and perception of feature representation. Therefore, the MCF module uses the SE attention mechanism to enhance the features of important channels and weaken the features of unimportant channels, and uses the CA attention mechanism to automatically adjust the attention weights of different scales through the learned coordinate information, so as to effectively handle the changes of small targets at different scales. Finally, the Shuffle operation is used to shuffle the channels of the data, and the information generated by the point-by-point convolution is infiltrated into each part of the information generated by the 3×3 convolution, which increases randomness and improves the generalization ability of the network.

[0025] like Figure 5As shown in the figure, the MCFCSP module uses three MCF BottleNeck stacks to form a residual structure, which aims to fuse shallow geometric structure information and deep semantic information to enhance target detection capabilities. The shallow residual module has high-resolution output features, which can better represent geometric detail information and target location information, but lacks semantic information. After multiple convolution operations, the deep residual module has a strong ability to represent semantic information, but lacks spatial geometric feature information. Therefore, by fusing the features of deep and shallow residual modules, comprehensive detection of remote sensing target information can be achieved. First, the module input image is convolved point by point, the number of channels is adjusted and the features are mixed to increase the network's expression ability. Then, three MCFBottleNecks capture more detailed information, further enhance the expression ability of features of different scales, and better capture multi-scale targets. Finally, the fused feature information is spliced ​​and the features are output through point-by-point convolution. The specific steps are as follows: first, the module input image undergoes one point-by-point convolution, then repeats the MCF BottleNeck operation three times, combines the obtained feature map with the feature map that has not undergone the MCF BottleNeck operation, and then undergoes one more point-by-point convolution to obtain the final output image of the module.

[0026] The MCF BottleNeck operation performs two MCF operations and one point-by-point convolution on the input, and finally combines the two feature maps as the output feature map of the operation.

[0027] like Figure 6 As shown, in the feature fusion part, the bidirectional multi-scale feature fusion structure (BMFM) proposed in the present invention first adjusts the channels of the four scales (C2 / C3 / C4 / C5) output by the backbone network, and then splices each scale with the feature map of the same size after upsampling from top to bottom to obtain the fused feature maps of four scales (P2 / P3 / P4 / P5), and finally downsamples and fuses the feature maps of different scales from bottom to top. Specifically, the channel dimension of the output fused feature maps (P2 / P3 / P4 / P5) of the four scales is increased, and then starting from the largest scale, the feature maps of each scale are spliced ​​with the feature maps of the same size after downsampling one by one to obtain the fused feature maps of three scales (N3 / N4 / N5).

[0028] Step 2.3: The three feature maps (N3 / N4 / N5) output by the neck network in step 2.2 are respectively subjected to fractional Fourier transform convolution (FrFTConv) to complete the detection of image targets. Fractional Fourier transform convolution uses multi-branch structure and residual structure to capture more feature information. First, a 1×1 convolution is performed on the input feature map, and the output is evenly divided into four parts. Each part is passed to a different Fn (n=1, 2, 3, 4) module, and fractional Fourier transform processing is performed at frequencies (f) of 0.25, 0.5, 0.75, and 1.0, respectively. The feature maps processed by each module are spliced, and then spliced ​​with the feature map before segmentation to obtain the final output image of the module.

[0029] like Figure 7 As shown in the figure, the FrFTConv module uses a multi-branch structure and a residual structure to capture more feature information. In the main branch, the input feature map first passes through a convolution layer with a convolution kernel size of 1, and is evenly divided into four parts. Each part is passed to a different Fn module and processed at frequencies of different sizes. It can simultaneously obtain low-order (close to the time domain) and high-order (close to the frequency domain) features, thereby smoothly transitioning from the time domain to the frequency domain, providing the network model with a more detailed perception of feature changes, thereby enabling the network model to learn more complex and rich feature representations. The Fn module structure diagram is shown in the figure. Figure 8 shown.

[0030] In the FrFT filter of the FrFTConv module, the input features are first decomposed to generate 2D grid coordinate matrices in the X and Y directions. The filter components in the X and Y directions are calculated using the FrFT formula, and finally multiplied to obtain the final filter output. For each output channel, the corresponding filter component is calculated: Among them, the filter component calculation formula in the X direction is as follows: , The filter component in the Y direction is calculated as follows: , The final filter component calculation formula is as follows: , in, f is the frequency parameter, p is the fractional ordinal parameter.

[0031] Step 3 specifically includes: Step 3.1: Input 80% of the training set into the model, and set the training parameters: initial learning rate, momentum parameter, decay coefficient, batch size, and use the pre-trained weight file for training to obtain the optimal parameter model.

[0032] Step 3.2: Input 20% of the test set into the best parameter model obtained through training for testing. During the testing process, the accuracy, recall rate and average accuracy of the network model are obtained.

[0033] The detection effect of the ablation experiment of ETF-Mnet provided in the embodiment on the DOTA dataset and the NWPU VHR-10 dataset using YOLOv7 as the benchmark model is compared as follows: Fig. 9 As shown in (a)-(g).

[0034] In order to prove the effectiveness of ETF-Mnet provided in this embodiment, the DOTA and NWPU VHR-10 remote sensing datasets are used to train and verify the model. The target detection effect comparison between ETF-Mnet provided in this embodiment and YOLOv5s, YOLOv7 and YOLOv8s algorithms is shown in the figure below: Fig.10 (a)-(e), Fig.11 As shown in (a)-(e), from the perspective of the detection accuracy of each category of targets, the ETF-Mnet proposed in this embodiment has a great advantage in improving the detection accuracy of small targets; the comparison of missed detection effects is shown in Fig.12 As shown in (a)-(e), the comparison of false detection effects is shown in Fig.13 As shown in (a)-(e), it can be seen that the ETF-Mnet provided in the embodiment can better detect the target and has great advantages in improving the target detection accuracy, and has significant superiority in complex backgrounds and higher accuracy.

[0035] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned remote sensing target detection method based on target feature enhanced multi-scale fusion network are implemented.

[0036] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the aforementioned remote sensing target detection method based on target feature enhanced multi-scale fusion network are implemented.

[0037] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0038] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0039] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0040] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0041] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A remote sensing target detection method based on target feature enhanced multi-scale fusion network, characterized in that: The steps include: Step 1: obtain a remote sensing image dataset, annotate the dataset, and divide the annotated dataset into a training set and a test set; Step 2: construct a target feature enhanced multi-scale fusion network model, including a backbone network, a neck network, and a detection head; A multi-layer feature aggregation module and MP module are constructed in the backbone network to extract features from the input image and obtain feature maps with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image respectively. A multi-channel feature fusion module is constructed in the neck network. Based on the multi-channel feature fusion module, a cross-stage network MCFCSP module is designed. The top-down and bottom-up feature fusion strategies are adopted to fuse feature maps with sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image, and the fused feature maps with sizes of 1 / 8, 1 / 16 and 1 / 32 of the input image are obtained. Introducing fractional Fourier transform into the detection head to construct fractional Fourier transform convolution for remote sensing target detection; Step 3, using the training set to train the target feature enhanced multi-scale fusion network model constructed in step 2 to obtain the optimal parameter network model, that is, the trained remote sensing target detection model; Step 4: Input the test set into the trained remote sensing target detection model and output the remote sensing target detection results.

2. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 1 is characterized in that: In the step 1, a remote sensing image data set is obtained, and images in the data set are annotated to obtain an annotated data set, where the annotated data set includes the original image and the corresponding label image; The original image and the corresponding label image are cropped After the images of different sizes are collected, the labeled dataset is randomly divided into training and test sets in a ratio of 8:

2.

3. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 1 is characterized in that: In the step 2, the backbone network includes a first CBH module, a second CBH module, a third CBH module, a fourth CBH module, a first multi-layer feature aggregation module, a first MP module, a second multi-layer feature aggregation module, a second MP module, a third multi-layer feature aggregation module, a third MP module, and a fourth multi-layer feature aggregation module connected in sequence; The structures of the first to fourth CBH modules are the same, including convolutional layers, batch normalization, and Mish activation functions. In the first and third CBH modules, the convolutional kernels of the convolutional layers are both 3 and the step sizes are both 1. In the second and fourth CBH modules, the convolutional kernels of the convolutional layers are both 3 and the step sizes are both 2. The structures of the first to third MP modules are the same, and all include the fifth to seventh CBH modules and the first maximum pooling layer; in the fifth and sixth CBH modules, the convolution kernel of the convolution layer is 1, and the step size is 1; in the seventh CBH module, the convolution kernel of the convolution layer is 3, and the step size is 2; the input of each MP module is sequentially generated through the first maximum pooling layer and the fifth CBH module to generate feature map a, and the input of each MP module is sequentially generated through the sixth and seventh CBH modules to generate feature map b, a and b are spliced ​​to obtain the output of each MP module; The input end of the first CBH module is connected to the input image, and the first, second, third and fourth multi-layer feature aggregation modules correspond to feature maps C2, C3, C4 and C5 with output sizes of 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the input image respectively.

4. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 3 is characterized in that: The first, second, third and fourth multi-layer feature aggregation modules have the same structure, and all include the eighth to tenth CBH modules and the first to fourth ODCBH modules; in the eighth to tenth CBH modules, the convolution kernel of the convolution layer is 1 and the step size is 1; the first to fourth ODCBH modules have the same structure, and all include a full-dimensional dynamic convolution layer, batch normalization and a Mish activation function; The input of each multi-layer feature aggregation module passes through the eighth CBH module, the first ODCBH module, the second ODCBH module, the third ODCBH module and the fourth ODCBH module in sequence to obtain the output of the fourth ODCBH module; the input of each multi-layer feature aggregation module passes through the ninth CBH module to obtain the output of the ninth CBH module; the output of the ninth CBH module is spliced ​​with the output of the eighth CBH module, the output of the second ODCBH module and the output of the fourth ODCBH module, and the spliced ​​result passes through the tenth CBH module to obtain the output of each multi-layer feature aggregation module.

5. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 1 is characterized in that: The neck network includes a spatial pyramid pooling module combined with cross-stage partial convolution, the first to sixth multi-channel feature fusion modules, the fourth to sixth MP modules, the first to sixth MCFCSP modules and the first to second feature refinement modules; The spatial pyramid pooling module combined with cross-stage partial convolution includes the eleventh to seventeenth CBH modules and the second to fourth maximum pooling layers; in the eleventh, thirteenth, fourteenth, sixteenth and seventeenth CBH modules, the convolution kernel of the convolution layer is 1 and the step size is 1; in the twelfth and fifteenth CBH modules, the convolution kernel of the convolution layer is 3 and the step size is 1; the input of the spatial pyramid pooling module combined with cross-stage partial convolution passes through the eleventh CBH module, the twelfth CBH module and the thirteenth CBH module in sequence to obtain the output of the thirteenth CBH module; the output of the thirteenth CBH module passes through the second, third and fourth The maximum pooling layer, the output of the thirteenth CBH module is spliced ​​with the output of the second maximum pooling layer, the output of the third maximum pooling layer and the output of the fourth maximum pooling layer, and the splicing result is sequentially passed through the fourteenth and fifteenth CBH modules to obtain the output of the fifteenth CBH module; the input of the spatial pyramid pooling module combined with the cross-stage partial convolution is passed through the sixteenth CBH module to obtain the output of the sixteenth CBH module, the output of the sixteenth CBH module is spliced ​​with the output of the fifteenth CBH module, and then passed through the seventeenth CBH module, and the output of the seventeenth CBH module is used as the output of the spatial pyramid pooling module combined with the cross-stage partial convolution; The structures of the fourth to sixth MP modules are the same, and all include the fifth to seventh CBH modules and the first maximum pooling layer; in the fifth and sixth CBH modules, the convolution kernel of the convolution layer is 1, and the step size is 1; in the seventh CBH module, the convolution kernel of the convolution layer is 3, and the step size is 2; the input of each MP module is sequentially generated through the first maximum pooling layer and the fifth CBH module to generate a feature map a, and the input of each MP module is sequentially generated through the sixth and seventh CBH modules to generate a feature map b, and a and b are spliced ​​to obtain the output of each MP module; The backbone network outputs a feature map C5 of 1 / 32 of the input image, which is output by a spatial pyramid pooling module combined with cross-stage partial convolution to output a feature map P5. After P5 passes through the first multi-channel feature fusion module and an upsampling operation, a feature map P51 is obtained; the backbone network outputs a feature map C4 of 1 / 16 of the input image, which passes through the second multi-channel feature fusion module, and the output of the second multi-channel feature fusion module is spliced ​​with P51 to obtain a feature map P52. P52 passes through the first MCFCSP module to obtain a feature map P4. P4 passes through the third multi-channel feature fusion module and an upsampling operation to obtain a feature map P41; the backbone network outputs a feature map C3 of 1 / 8 of the input image, which passes through the fourth multi-channel feature fusion module, and the output of the fourth multi-channel feature fusion module is spliced ​​with P41 to obtain a feature map P42. P42 passes through the second MCFCSP module to obtain a feature map P3. P3 passes through the fifth multi-channel feature fusion module and an upsampling operation to obtain a feature map P31; the backbone network outputs a feature map C4 of 1 / 16 of the input image, which passes through the second multi-channel feature fusion module, and the output of the second multi-channel feature fusion module is spliced ​​with P41 to obtain a feature map P42. P42 passes through the second MCFCSP module to obtain a feature map P31. After the feature map C2 of 1 / 4 of the input image passes through the sixth multi-channel feature fusion module, the output of the sixth multi-channel feature fusion module is spliced ​​with P31 to obtain the feature map P32, and P32 passes through the third MCFCSP module to obtain the feature map P2; P2 passes through the first feature refinement module and the fourth MP module in sequence to obtain the output of the fourth MP module, P3 passes through the second feature refinement module to obtain the output of the second feature refinement module, the output of the second feature refinement module is spliced ​​with the output of the fourth MP module, and the splicing result passes through the fourth MCFCSP module to obtain the fused feature map N3; N3 passes through the fifth MP module to obtain the output of the fifth MP module, the output of the fifth MP module is spliced ​​with P4, and the splicing result passes through the fifth MCFCSP module to obtain the fused feature map N4; N4 passes through the sixth MP module to obtain the output of the sixth MP module, the output of the sixth MP module is spliced ​​with P5, and the splicing result passes through the sixth MCFCSP module to obtain the fused feature map N5; N3, N4 and N5 are the outputs of the neck network.

6. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 5 is characterized in that: The structures of the first to sixth MCFCSP modules are the same, and all include the second to fourth point-by-point convolution layers and the first to third MCF BottleNeck modules; the convolution kernels of the second to fourth point-by-point convolution layers are all 1; the structures of the first to third MCFBottleNeck modules are the same, and all include the seventh to eighth multi-channel feature fusion modules and the fifth point-by-point convolution layer, the input of each MCF BottleNeck module is sequentially passed through the seventh and eighth multi-channel feature fusion modules to obtain the output of the eighth multi-channel feature fusion module, the input of each MCF BottleNeck module is passed through the fifth point-by-point convolution layer to obtain the output of the fifth point-by-point convolution layer, the output of the fifth point-by-point convolution layer is spliced ​​with the output of the eighth multi-channel feature fusion module to obtain the output of each MCF BottleNeck module; The input of each MCFCSP module passes through the second point-by-point convolution layer, the first MCF BottleNeck module, the second MCFBottleNeck module and the third MCF BottleNeck module in sequence to obtain the output of the third MCF BottleNeck module; the input of each MCFCSP module passes through the third point-by-point convolution layer to obtain the output of the third point-by-point convolution layer, the output of the third point-by-point convolution layer is spliced ​​with the output of the third MCF BottleNeck module, and then passes through the fourth point-by-point convolution layer to obtain the output of each MCFCSP module; The structures of the first to eighth multi-channel feature fusion modules are the same, including the first point-by-point convolution layer, the first to third convolution layers, the first to third SE attention mechanisms, and the CA attention mechanism; the convolution kernel of the first point-by-point convolution layer is 1, and the convolution kernels of the first to third convolution layers are all 3; The number of output channels of each multi-channel feature fusion module is set to X2, and the number of channels of the input feature map of each multi-channel feature fusion module is reduced to one quarter of X2 through the first point-by-point convolution layer to obtain feature map E1; E1 is subjected to the first convolution layer and the first SE attention mechanism to obtain feature map E2; E1 is subjected to the second convolution layer and the second SE attention mechanism to obtain feature map E3; E1 is subjected to the third convolution layer and the third SE attention mechanism to obtain feature map E4; E4 is spliced ​​with E3, E2 and E1, and the splicing result is subjected to the CA attention mechanism and channel mixing operation to obtain the output of each multi-channel feature fusion module.

7. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 5 is characterized in that: The structures of the first to second feature refinement modules are the same, and both include the eighteenth CBH module and the fourth to fifth convolutional layers; the convolution kernel of the convolutional layer in the eighteenth CBH module is 1, and the step size is 1; the convolution kernel of the fourth convolutional layer is , the convolution kernel of the fifth convolutional layer is ; The feature map input to each feature refinement module is sliced, and the number of channels of the input feature map is divided into two halves, one half of which passes through the fourth convolutional layer and the other half passes through the fifth convolutional layer. The output of the fourth convolutional layer is spliced ​​with the output of the fifth convolutional layer, and then passes through the eighteenth CBH module. The output of the eighteenth CBH module is spliced ​​with the feature map input to each feature refinement module to obtain the output of each feature refinement module.

8. The remote sensing target detection method based on target feature enhanced multi-scale fusion network according to claim 1 is characterized in that: The detection head includes the first to third fractional Fourier transform convolution modules. The fused feature map of the neck network output with a size of 1 / 8 of the input image is passed through the first fractional Fourier transform convolution module to perform large-scale target detection; the fused feature map of the neck network output with a size of 1 / 16 of the input image is passed through the second fractional Fourier transform convolution module to perform medium-scale target detection; the fused feature map of the neck network output with a size of 1 / 32 of the input image is passed through the third fractional Fourier transform convolution module to perform small-scale target detection.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the remote sensing target detection method based on the target feature enhanced multi-scale fusion network as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the remote sensing target detection method based on the target feature enhanced multi-scale fusion network as described in any one of claims 1 to 8 are implemented.

Images