A ship detection method and device based on frequency domain feature fusion and a storage medium

Abstract

The application provides a ship detection method and device based on frequency domain feature fusion and a storage medium, and belongs to the technical field of radar detection. The method comprises the following steps: inputting an acquired current remote sensing image into a ship detection model constructed based on a single-stage target detection method; generating frequency domain features corresponding to the current remote sensing image by a pre-constructed frequency domain feature extraction module in the ship detection model according to the current remote sensing image; generating spatial domain features corresponding to the current remote sensing image by a pre-constructed spatial domain feature extraction module in the ship detection model according to the current remote sensing image; generating fused features by the ship detection model according to the frequency domain features and the spatial domain features; and generating a detection result corresponding to the current remote sensing image by the ship detection model according to the fused features. The application aims to provide a ship detection method which can improve the accuracy and has real-time performance.

Description

Technical Field

[0001] This application relates to the technical field of radar detection, and more specifically, to a ship detection method, apparatus, and storage medium based on frequency domain feature fusion. Background Technology

[0002] Compared to optical and multispectral satellites, Synthetic Aperture Radar (SAR) satellites can adapt to varying marine climates and achieve all-weather ship detection. Early ship detection using SAR images primarily employed three algorithms: constant false alarm rate (CFAR), polarization-based, and geometric feature-based. However, CFAR algorithms rely on accurate statistical models, which are often difficult to implement. Polarization-based methods are susceptible to sea clutter noise, leading to performance degradation. Geometric feature-based algorithms rely on expert experience to build template libraries, which are prone to mismatches in noisy environments and have high computational costs and poor transferability.

[0003] In recent years, with the rise of artificial intelligence and the improvement of remote sensing image quality, deep learning-based SAR image ship detection has achieved great success. Deep learning-based SAR image ship detection can be divided into two categories: two-stage detection methods and single-stage detection methods. However, two-stage detection methods are slow and have low real-time efficiency because they require processing a large number of candidate regions separately. Single-stage detection methods mainly rely on the spatial domain features of remote sensing images, but sea clutter often affects the spatial domain, leading to insufficient extraction of ship features and lower detection accuracy. Therefore, in ship detection, there is an urgent need for detection methods that can improve accuracy while maintaining real-time performance. Summary of the Invention

[0004] This application provides a ship detection method, apparatus, and storage medium based on frequency domain feature fusion, aiming to provide a detection method that can improve accuracy and has real-time performance.

[0005] In a first aspect, embodiments of this application provide a ship detection method based on frequency domain feature fusion, the method comprising:

[0006] The acquired current remote sensing image is input into the ship detection model, which is constructed based on a single-stage target detection method;

[0007] Based on the current remote sensing image, the frequency domain features corresponding to the current remote sensing image are generated through the frequency domain feature extraction module pre-built in the ship detection model.

[0008] Based on the current remote sensing image, spatial domain features corresponding to the current remote sensing image are generated through the pre-built spatial domain feature extraction module in the ship detection model.

[0009] The ship detection model generates fused features based on the frequency domain features and the spatial domain features;

[0010] The ship detection model generates the detection result corresponding to the current remote sensing image based on the fused features.

[0011] Optionally, based on the current remote sensing image, frequency domain features corresponding to the current remote sensing image are generated through a frequency domain feature extraction module pre-built in the ship detection model, including:

[0012] The ship detection model pre-constructs a frequency domain feature extraction module to extract multi-level frequency domain features from the current remote sensing image.

[0013] Optionally, based on the current remote sensing image, spatial domain features corresponding to the current remote sensing image are generated through a pre-built spatial domain feature extraction module in the ship detection model, including:

[0014] The spatial domain feature extraction module is pre-built in the ship detection model to extract the spatial domain features of the current remote sensing image step by step.

[0015] From the high-level to the low-level direction of the current remote sensing image, the spatial domain features of the current remote sensing image at adjacent levels are fused to enhance the semantic information of each level of features.

[0016] Optionally, the ship detection model generates fused features based on the frequency domain features and the spatial domain features, including:

[0017] Using the PAN method, multi-level frequency domain features and multi-level spatial domain features are spliced ​​layer by layer in order from low to high level to generate multi-level fused features.

[0018] Optionally, the process of pre-constructing the frequency domain feature extraction module includes:

[0019] A preprocessing layer is constructed, wherein the preprocessing layer includes performing two-dimensional Fourier transform, batch normalization processing, and activation function calculation on the current remote sensing image;

[0020] A multi-level convolution operation module is constructed, wherein the channel dimension of the multi-level convolution operation module increases sequentially.

[0021] Optionally, a spatial domain feature extraction module is pre-built, including:

[0022] Construct a CSPNet structure, wherein the CSPNet structure is used to sequentially generate the semantic features of the multi-level spatial domain of the current remote sensing image in order of channel dimension from low to high;

[0023] An FPN structure is constructed, wherein the FPN structure fuses the semantic features of the spatial domains of adjacent levels in order from high to low, thereby obtaining multi-level spatial domain features with uniform semantic information distribution.

[0024] Optionally, the method further includes:

[0025] In any of the processes of constructing the frequency domain feature extraction module, constructing the spatial domain feature extraction module, and applying the PAN method, an optimization module is added, wherein the optimization module is used to optimize the convolution operation results to reduce memory consumption.

[0026] Optionally, the method further includes:

[0027] Based on the domain differences in remote sensing image data, the frequency domain feature extraction module is fine-tuned during the transfer learning process to adapt the ship detection model to remote sensing image data with domain differences.

[0028] Secondly, embodiments of this application provide a ship detection device based on frequency domain feature fusion, the device comprising:

[0029] The input module is used to input the acquired current remote sensing image into the ship detection model, wherein the ship detection model is constructed based on a single-stage target detection method;

[0030] The frequency domain feature generation module is used to generate frequency domain features corresponding to the current remote sensing image based on the current remote sensing image and through the frequency domain feature extraction module pre-built in the ship detection model.

[0031] The spatial domain feature generation module is used to generate spatial domain features corresponding to the current remote sensing image based on the current remote sensing image and through the spatial domain feature extraction module pre-built in the ship detection model.

[0032] The fusion module is used by the ship detection model to generate fused features based on the frequency domain features and the spatial domain features;

[0033] The detection result generation module is used by the ship detection model to generate the detection result corresponding to the current remote sensing image based on the fused features.

[0034] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the ship detection method based on frequency domain feature fusion described in the first aspect of the embodiments.

[0035] Beneficial effects:

[0036] Compared to existing models, the ship detection model provided in this embodiment is based on a single-stage detection method. While ensuring detection speed, it extracts frequency domain features and spatial domain features from the current remote sensing image, and then fuses the frequency domain features and spatial domain features. Based on the fused features, it further determines the detection result corresponding to the current remote sensing image. The ship detection model generates detection results with higher accuracy based on richer features, thus improving both accuracy and real-time performance. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the steps of a ship detection method based on frequency domain feature fusion proposed in an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of the structure of a frequency domain feature extraction module proposed in an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of the optimization module in the frequency domain feature extraction module provided in an embodiment of this application;

[0041] Figure 4 This is a schematic diagram of the bottleneck unit proposed in an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the spatial domain feature extraction module proposed in one embodiment of this application;

[0043] Figure 6 This is a schematic diagram of the structure of an SPP unit proposed in an embodiment of this application;

[0044] Figure 7 This is a schematic diagram of the overall structure of a ship detection model proposed in an embodiment of this application;

[0045] Figure 8This is a schematic diagram of a remote sensing image filtered in different frequency bands according to an embodiment of this application;

[0046] Figure 9 This is a functional block diagram of a ship detection device based on frequency domain feature fusion proposed in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Synthetic Aperture Radar (SAR) satellites can adapt to varying marine climates and achieve all-weather ship detection. Early SAR image ship detection algorithms mainly included constant false alarm rate (CFAR) algorithms, polarization-based methods, and geometric feature-based detection algorithms.

[0049] However, constant false alarm rate (CFAR) algorithms rely on accurate statistical models, which are often difficult to implement; polarization-based methods are susceptible to sea clutter noise, leading to performance degradation; geometric feature-based detection algorithms rely on expert experience to build template libraries, which are prone to mismatches when noise is high, and have high computational costs and poor transferability; moreover, the models provided by these traditional methods are often difficult to adjust parameters, are often not robust enough for images of different resolutions, and have slow running speeds.

[0050] With the rise of artificial intelligence and the improvement of remote sensing image quality, deep learning-based SAR image ship detection has achieved great success. Deep learning-based SAR image ship detection can be divided into two categories:

[0051] One type is the two-stage detection method, such as the one based on R-CNN, which uses Selective Search to generate proposed regions, and then performs bounding box regression and classification on the proposed regions. Since the two-stage detection method needs to process a large number of candidate regions separately, the inference speed is slow and the real-time performance of the detection process is poor.

[0052] Another type is the single-stage detection method, such as the existing method You Only Look Once. YOLO treats the target detection problem as a regression problem. It has a fast detection speed, but it is poor in detecting small targets and groups of targets, and its localization effect is also poor. Furthermore, existing single-stage detection methods often ignore the frequency domain feature information of remote sensing images. At the same time, remote sensing images are often affected by sea clutter in the spatial domain. Therefore, single-stage detection methods have the problem of insufficient ability to extract ship features, resulting in low accuracy.

[0053] Based on this, the embodiments of this application provide a ship detection method based on frequency domain feature fusion, which can improve both detection accuracy and real-time performance during detection.

[0054] Figure 1 The diagram illustrates a flowchart of a ship detection method based on frequency domain feature fusion according to an embodiment of this application. The method specifically includes the following steps:

[0055] S101: Input the acquired current remote sensing image into the ship detection model.

[0056] Remote sensing images are two-dimensional images currently detected by synthetic aperture radar (SAR), and can be labeled as I(x, y).

[0057] The ship detection model is built based on a single-stage target detection method, so it has the characteristic of fast detection speed. For example, the ship detection model in this embodiment can adopt YOLO-MSF, and the ship detection model can output the corresponding detection results according to the current remote sensing image.

[0058] S102: Based on the current remote sensing image, the frequency domain features corresponding to the current remote sensing image are generated through the frequency domain feature extraction module pre-built in the ship detection model.

[0059] However, while current single-stage target detection methods offer fast detection speeds, their ability to extract ship features is insufficient, resulting in low accuracy of the final detection results. Therefore, this method incorporates a frequency domain feature extraction module into the ship detection model. By adding frequency domain feature extraction to the existing remote sensing image, richer features can be extracted. Consequently, the ship detection model, based on these richer ship features, can generate more accurate detection results.

[0060] In one feasible implementation, a frequency domain feature extraction module can be pre-built in the ship detection model. After the current remote sensing image is input into the pre-built frequency domain feature extraction module, the frequency domain features in the current remote sensing image can be generated through the frequency domain feature extraction module.

[0061] Furthermore, the frequency domain feature extraction module in this embodiment adopts a multi-level frequency domain feature extraction method. Specifically, the multi-level frequency domain feature extraction method can fully characterize the frequency domain features of ships in remote sensing images. Moreover, in the target domain, by fine-tuning the frequency domain feature extraction module alone, better target detection performance can be achieved with a small amount of computation, and it has good domain generalization performance.

[0062] Reference Figure 2 The diagram shows a schematic of the frequency domain feature extraction module provided in an embodiment of this application. The frequency domain feature extraction module uses a convolutional neural network, where FTLayer is a preprocessing layer used to preprocess the current remote sensing image I(x,y).

[0063] Specifically, the preprocessing layer FTLayer includes performing a two-dimensional Fourier transform on the current remote sensing image I(x, y). If the two-dimensional Fourier transform operation is denoted as F(·), and the result of the two-dimensional Fourier transform is denoted as F(u, v), then the operational relationship of the preprocessing layer can be expressed as:

[0064] F(u, v) = F(I(x, y))

[0065] In other implementations, the preprocessing layer FTLayer may further include a batch normalization process, assuming that the input batch to be processed is... Batch normalization can then be expressed as:

[0066]

[0067] Where, x i This represents the vector corresponding to the i-th sample. This represents the mean of the samples input in this batch; This represents the variance of the input samples in this batch; γ is the scaling parameter; and β is the translation parameter.

[0068] The formula for the activation function SiLu in the preprocessing layer can be:

[0069]

[0070] Then, the preprocessed remote sensing image F(u, v) is subjected to multiple convolution operations through a multi-level convolution operation module, and the channel dimension of the multi-level convolution operation module increases sequentially; Conv(c1, c2, k, s) represents the convolution operation, where c1 represents the input channel dimension, c2 represents the output channel dimension, k represents the convolution kernel side length k, and s represents the stride of the convolution operation.

[0071] like Figure 2In the frequency domain feature extraction module, an optimization module C3 is added during multiple convolution operations. The optimization module C3 optimizes the results of each convolution operation, which can enhance the learning ability of the convolutional neural network of the frequency domain feature extraction module and reduce memory consumption. In C3(c1, c2), c1 represents the dimension of the input channel and c2 represents the dimension of the output channel.

[0072] Reference Figure 3 This diagram illustrates the structure of the optimization module within the frequency domain feature extraction module provided in an embodiment of this application. Figure 3 As shown, the optimization module C3 first performs two Conv(c1, c2, 1, 1) convolution operations on the current convolution operation result, and then inputs the result of one of the Conv(c1, c2, 1, 1) convolution operations into the bottleneck unit Bottleneck, and then performs a concat operation on the result of the other Conv(c1, c2, 1, 1) convolution operation. Finally, it performs a Conv(c1, c2, 1, 1) convolution operation on the result of the concat operation and outputs it.

[0073] Reference Figure 4 The diagram shows a schematic of the bottleneck unit provided in an embodiment of this application. In one feasible implementation, bs, c / 2, 2, h in the bottleneck unit Bottleneck are the outputs of the convolution operation.

[0074] The frequency domain feature extraction module provided in this application extracts multi-level frequency domain features from the current remote sensing image through layer-by-layer convolution. Lower-level convolutional layers have smaller receptive fields but can extract more detailed features, enabling the ship detection model network to model more refined local frequency domain information relationships. Higher-level convolutional layers have larger receptive fields and can model a wider range of frequency domain information relationships. Therefore, the frequency domain feature extraction module can fully extract rich and diverse frequency domain features at different levels. For example, in... Figure 2 In the diagram, P3, P4, and P5 represent feature maps of different levels, with P3 being a lower-level feature map and P5 being a higher-level feature map.

[0075] S103: Based on the current remote sensing image, the spatial domain features corresponding to the current remote sensing image are generated through the pre-built spatial domain feature extraction module in the ship detection model.

[0076] In one feasible implementation, a spatial domain feature extraction module can be pre-built in the ship detection model. After obtaining the current remote sensing image, in addition to inputting it into the frequency domain feature extraction module to extract frequency domain features, it can also input it into the spatial domain feature extraction module to extract spatial domain features.

[0077] Reference Figure 5The diagram shows a schematic of the spatial domain feature extraction module provided in an embodiment of this application. In one embodiment, the spatial domain feature extraction module can adopt the structure of CSPNet and FPN. By extracting the semantic information of the spatial domain through the structure of CSPNet and FPN, the feature map with low resolution strong semantic information and the feature map with high resolution weak semantic information can be fused with less computation to obtain spatial domain features of different scales with average distribution.

[0078] Specifically, Figure 5 The left column in the image shows the CSPNet structure. In order to allow the model to learn more features, the CSPNet structure includes an optimization module C3 in addition to the standard convolution.

[0079] Reference Figure 3 The diagram shown illustrates the optimization module C3. The CSPNet structure containing the optimization module C3 divides the input into two branches, performs convolution operations on each branch to halve the number of channels, then performs a Bottleneck operation on one branch, and finally concatenates the two branches by channel. This structure allows the model to learn more features.

[0080] Figure 5 The right column in the diagram represents the FPN structure. The size of the feature map increases progressively through upsampling, resulting in increasingly fine-grained low-level semantic information. Each time the feature map passes through an Upsample layer, its size is doubled, and it is then concatenated with the Backbone feature map via channel-wise concatenation. The FPN structure works from top to bottom, fusing feature maps extracted from adjacent levels of the backbone network through bitwise addition. High-level semantic features are propagated layer by layer to enhance the semantic information contained in each level of feature map.

[0081] In this embodiment, the spatial domain feature extraction module extracts semantic information of the spatial domain step by step using the backbone network of CSPNet structure. FPN fuses feature maps of adjacent levels from high-level semantic features to low-level semantic features. It can fuse feature maps with low-resolution strong semantic information and high-resolution weak semantic information with less increase in computation, so that the semantic information and localization information of feature maps of different scales are more even.

[0082] Reference Figure 6The diagram shows a schematic of the SPP unit provided in this embodiment. The SPP unit in step 9 of the spatial domain feature extraction module is used to transform a feature map of arbitrary size into a feature vector of fixed size. In SPP(c1,c2,[n1,n2,n3]), c1 represents the dimension of the input channel, c2 represents the dimension of the output channel, and n1, n2, and n3 represent the window size of the pooling operation of SPP. Specifically, for the input feature map, each channel is divided into n1*n1, n2*n2, and n3*n3 blocks respectively, and max pooling is performed on each block. The pooling results are concatted to finally obtain a feature vector with a fixed output size and dimension. The SPP unit enables the network to be compatible with features of multiple scales.

[0083] In this embodiment, the window size of the pooling operation of the SPP unit is 5 for n1, 9 for n2, and 13 for n3. For the input feature map, it is divided into 5*5, 9*9, and 13*13 respectively for max pooling. The three pooling results are concatted and then output as a feature vector with a fixed size dimension through a convolution operation with a kernel side length of 1 and a stride of 1.

[0084] In step 15 of the spatial domain feature extraction module, c1 in Upsample(c1,c2,n) represents the dimension of the input channel, c2 represents the dimension of the output channel, and n represents the upsampling factor. The upsampling algorithm uses nearest neighbor interpolation.

[0085] By inputting the current remote sensing image into the spatial domain feature extraction module, semantic information of the spatial domain is extracted through the structure of CSPNet and FPN. With less computation, feature maps with strong semantic information at low resolution and feature maps with weak semantic information at high resolution are fused to obtain spatial domain features of different scales with average distribution.

[0086] S104: The ship detection model generates fused features based on the frequency domain features and the spatial domain features.

[0087] By obtaining frequency domain features and spatial domain features from the current remote sensing image respectively, fusing the frequency domain features and spatial domain features, and then generating detection results based on the fused features, the accuracy of the results is more precise.

[0088] See Figure 7This diagram illustrates the overall structure of the ship detection model provided in this embodiment. Since different fusion methods are used during the fusion of frequency domain features and spatial domain features, the accuracy of the final detection result will be greatly affected. In order to fully fuse the frequency domain features with the spatial domain features generated by FPN, this embodiment adopts the PAN method for fusion. Through the PAN method, multi-level frequency domain features and multi-level spatial domain features are spliced ​​layer by layer in the order from low level to high level to generate multi-level fused features.

[0089] In the feature fusion network of this embodiment, PAN obtains feature maps with progressively smaller size and higher semantic information from the output of FPN through convolution. Each time the feature map passes through a convolutional layer, its size is reduced to half of its original size, and the number of channels is doubled by concatenating it with the FPN feature map according to the number of channels.

[0090] like Figure 7 As shown, the frequency domain feature extraction module extracts multi-level frequency domain features and fuses them sequentially at the corresponding scale level of the PAN using a concat method, from low-level, detailed, small receptive field to high-level, large receptive field, stitching together the frequency domain features and spatial domain features layer by layer.

[0091] For example, the low-level frequency domain feature map output in step 5 of the frequency domain feature extraction module, the low-level spatial domain feature map output in step 2 of the spatial domain feature extraction module, and the feature map output in step 15 of the FPN are fused in step 16 by concatenation; the intermediate-level frequency domain feature map output in step 7 of the frequency domain feature extraction module, the intermediate-level spatial domain feature map output in step 14 of the spatial domain feature extraction module, and the feature map output in step 18 of the PAN are fused in step 19 by concatenation; and the high-level frequency domain feature map output in step 9 of the frequency domain feature extraction module, the high-level spatial domain feature map output in step 10 of the spatial domain feature extraction module, and the feature map output in step 21 of the PAN are fused in step 22 by concatenation.

[0092] The concatenated features exhibit spatial domain features in some channels and frequency domain features in others. Then, convolution operations are used to fuse the features between channels.

[0093] The ship detection model provided in this embodiment includes two connection structures: FPN and PAN. The FPN fuses feature maps extracted from adjacent levels of the backbone network in a top-down manner through bitwise addition. High-level semantic features are propagated layer by layer to enhance the semantic information contained in each level of feature map. At the same time, the PAN fuses frequency domain features and spatial domain features from the bottom up. The localization information of the low-level frequency domain feature maps is used to enhance the high-level frequency domain feature maps, which can better extract multi-level frequency domain and spatial domain features and fuse multi-level frequency domain and spatial domain features.

[0094] By using PAN fusion, a multi-scale feature fusion method, frequency domain features are combined with a single-stage target detection network, resulting in better network detection performance. At the same time, it improves the real-time performance of target detection compared to existing multi-stage target detection networks.

[0095] Because existing frequency domain feature extraction methods only utilize shallow frequency domain feature information, the detection network can only use single-scale frequency domain feature information with a small receptive field. However, the multi-level frequency domain feature extraction and multi-scale feature fusion method adopted in this embodiment enables the ship detection model network to model the relationship between a wider range of frequency domain information, thereby mining richer frequency domain information, and thus outputting more accurate detection results based on rich frequency and spatial domain information.

[0096] In other implementations, the fusion of frequency domain features and spatial domain features can also be performed using AHLC (Adjacent Higher-Level Composition) or SLC (Same Level Composition).

[0097] Specifically, the AHLC method fuses high-level features from adjacent backbone networks into the current level, while the SLC method fuses semantic features at the same level from adjacent backbone networks into the current level. However, this method does not use Add during fusion, but instead uses Concat, because directly adding frequency domain features and spatial domain features will cause the spatial domain features with larger activation values ​​to be overwhelmed by the frequency domain features with lower activation values.

[0098] Table 1 compares the detection results generated by the ship detection model using different feature fusion methods.

[0099] Table 1 Comparison of results for different feature fusion methods

[0100] method mAP(%) Computational complexity (GFLOPs) SLC 96.1 28.2 AHLC 96.9 27.7 Integrating PAN 97.2 24.8

[0101] As can be seen from Table 1, although the ship detection model performs well in terms of accuracy and computational cost under different fusion methods, the mean average precision (mAP) is the highest when using the PAN method for fusion. mAP is an indicator of detection accuracy in target detection, and the higher the value, the higher the detection accuracy. At the same time, the computational cost of the PAN method is the lowest, indicating that this fusion method has the advantages of both high detection accuracy and low computational cost.

[0102] S105: The ship detection model generates the detection result corresponding to the current remote sensing image based on the fused features.

[0103] Compared to existing models, the ship detection model provided in this embodiment is based on a single-stage detection method. While ensuring detection speed, it extracts frequency domain features and spatial domain features from the current remote sensing image, and then fuses the frequency domain features and spatial domain features. Based on the fused features, it further determines the detection result corresponding to the current remote sensing image. The ship detection model generates detection results with higher accuracy based on richer features, thus improving both accuracy and real-time performance.

[0104] Reference Figure 8 This shows a schematic diagram of remote sensing images after filtering in different frequency bands, such as... Figure 8 Although the low-frequency components in remote sensing images contain less noise, the outlines of ships are not obvious. Although the high-frequency components contain the outline information of the image, most of the noise in remote sensing images is concentrated in the high-frequency region, while the mid-frequency components are of relatively better quality. Therefore, the ship detection model provided in this embodiment can remove high-frequency noise in the frequency domain space and explicitly integrate the extracted frequency domain features into the model network, which plays a role in resisting noise interference and improving data quality, thereby improving the accuracy of the detection results.

[0105] In one feasible implementation, during the training of the ship detection model YOLO-MSF, the training set can be defined as the pre-training set of SSDD. Considering that training the model with sufficient data samples in the source domain will have a better effect, the target domain training set can be selected as the first 100 data in the HRSID dataset, trained for 50 epochs, and then the data in the HRSID dataset with indices from 101 to 500 can be used as the target domain test set.

[0106] The ship detection model YOLO-MSF provided in this embodiment is compared with several ship detection models obtained by applying existing detection methods, as shown in Table 2. The existing models include Faster R-CNN, M-DFaster RCNN, YOLOV5s, and YOLOV5m.

[0107] Table 2 Comparison of detection performance of different models

[0108] method mAP(%) Computational complexity (GFLOPs) Inference time (ms) Faster R-CNN 94.5 129.0 93.9 MD Faster R-CNN 96.2 147.5 105.8 YOLOV5s 96.3 15.8 6.6 YOLOV5m 96.7 48.0 14.5 YOLO-MSF 97.2 24.8 11.7

[0109] As can be seen from Table 2, Faster R-CNN and M-DFaster R-CNN obtained by applying the two-stage detection method have significantly longer inference times than other models, resulting in greater latency and poorer real-time performance.

[0110] For YOLOV5s and YOLOV5m, which use a single-stage detection method, the inference time is greatly shortened while ensuring detection accuracy. Among them, YOLOV5s has better overall performance than YOLOV5m.

[0111] Based on the YOLO-MSF provided in this embodiment, compared with YOLOV5s, after adding a frequency domain feature extraction module and adopting a suitable feature fusion method, the mAP index is improved by 0.9%. Moreover, even with the addition of a frequency domain feature extraction module, the cost of the introduced computational load and inference time is relatively small. Compared with YOLOV5m, YOLO-MSF has advantages in detection accuracy, computational load, and inference time. Therefore, it can be seen that the method proposed in this embodiment can achieve better results in the trade-off between detection accuracy and detection speed.

[0112] In one feasible implementation, the method further includes:

[0113] S106: Based on the domain differences of remote sensing image data, the frequency domain feature extraction module is fine-tuned during the transfer learning process to adapt the ship detection model to remote sensing image data with domain differences.

[0114] Due to differences in satellite imaging parameters and the different preprocessing procedures for raw remote sensing data, there are often domain differences between remote sensing image data. Therefore, if there are domain differences between the data in actual applications and the training set, the performance of the ship detection model will decrease. In order to reduce such performance degradation, transfer learning is generally used to fine-tune the overall model.

[0115] Fine-tuning of the ship detection model includes three different approaches: fine-tuning the overall model, i.e., updating all parameters in the ship detection model's network; or fine-tuning the frozen backbone network, i.e., adjusting the frequency domain feature extraction module, bottleneck layer (including FPN and PAN structures), and detection head (such as...) in the ship detection model's network. Figure 7 The Head parameter is updated while other parameters are not updated; or only the frequency domain feature extraction module is fine-tuned, i.e., only the parameters in the frequency domain feature extraction module are updated while other parameters are not updated.

[0116] This embodiment uses three different fine-tuning methods and the target domain training set to conduct experiments on the ship detection model YOLO-MSF. The experimental results are shown in Table 3.

[0117] Table 3 Comparison of models obtained from transfer learning under different fine-tuning methods

[0118] method mAP(%) Computational complexity (GFLOPs) Overall fine-tuning 88.9％ 24.8 Frozen backbone network 89.8％ 14.4 Fine-tuning frequency domain feature extraction module 90.0％ 9.0

[0119] As can be seen from Table 3, if the overall model is fine-tuned, the mAP is the lowest among the three methods, and the computational cost is also the highest among the three methods. That is, fine-tuning the overall model will consume a lot of computing resources and the training time is long. However, if only the frequency domain feature extraction module is fine-tuned, the mAP is the highest among the three methods, and the computational cost is significantly less than fine-tuning the entire model and fine-tuning the frozen backbone network.

[0120] Furthermore, since the number of data samples available for training in the target domain is relatively small, overfitting can occur when fine-tuning a network with a large parameter scale, resulting in poor performance of the network on the target domain test set. If only the frequency domain feature extraction module is fine-tuned, fewer parameters need to be updated, which can better prevent overfitting and make the ship detection model perform better in the target domain.

[0121] Therefore, the ship detection model provided in this embodiment not only retains the fast detection speed of the single-stage detection method, but also extracts multi-level frequency domain features and multi-level spatial domain features, and then performs multi-scale fusion of the frequency domain and spatial domain features to obtain more accurate detection results with richer features. In order to solve the problem of domain differences, the frequency domain feature extraction module can be fine-tuned separately to perform transfer learning on the target domain, reducing the number of parameters required for training and speeding up the training speed. At the same time, it also helps to prevent overfitting, making the ship detection model more applicable in different scenarios.

[0122] This application has at least the following effects:

[0123] 1. Applying frequency domain features to models using single-stage detection methods can make fuller use of the spatial and frequency domain features of remote sensing images, suppress noise, and improve the accuracy of detection results.

[0124] 2. When a frequency domain feature extraction module is introduced into the model applying the single-stage detection method, the amount of computation introduced is small, and thus the inference time cost is small. It can improve the accuracy while still ensuring the real-time performance of the ship detection model.

[0125] 3. When extracting frequency domain features, a multi-level frequency domain feature extraction method is adopted, which can model the relationship between a wider range of frequency domain information and thus mine richer frequency domain information; and the multi-level frequency domain features are fused with multi-level spatial domain features, so that more accurate detection results can be obtained based on richer features.

[0126] 4. When there are domain differences in the target domain, by fine-tuning the frequency domain feature extraction module, transfer learning can be performed with less computational cost, and the ship detection model obtained by transfer learning has better performance, making the model more applicable.

[0127] Reference Figure 9 This diagram illustrates a functional block diagram of a ship detection device based on frequency domain feature fusion according to an embodiment of this application. The device includes:

[0128] The input module 100 is used to input the acquired current remote sensing image into the ship detection model, wherein the ship detection model is constructed based on a single-stage target detection method;

[0129] The frequency domain feature generation module 200 is used to generate frequency domain features corresponding to the current remote sensing image based on the current remote sensing image and through the frequency domain feature extraction module pre-built in the ship detection model.

[0130] The spatial domain feature generation module 300 is used to generate spatial domain features corresponding to the current remote sensing image based on the current remote sensing image and through the spatial domain feature extraction module pre-built in the ship detection model.

[0131] The fusion module 400 is used by the ship detection model to generate fused features based on the frequency domain features and the spatial domain features;

[0132] The detection result generation module 500 is used by the ship detection model to generate the detection result corresponding to the current remote sensing image based on the fused features.

[0133] Optionally, the frequency domain feature generation module includes:

[0134] The multi-level frequency domain feature generation unit is used to extract multi-level frequency domain features from the current remote sensing image by using the frequency domain feature extraction module pre-built in the ship detection model.

[0135] Optionally, the spatial domain feature generation module includes:

[0136] The first spatial domain feature generation unit is used to extract the spatial domain features of the current remote sensing image step by step through the spatial domain feature extraction module pre-built in the ship detection model.

[0137] The second spatial domain feature generation unit is used to fuse the spatial domain features of the current remote sensing image at adjacent levels from high to low levels to enhance the semantic information of each level of features.

[0138] Optionally, the fusion module includes:

[0139] The multi-level fusion unit is used to stitch together multi-level frequency domain features and multi-level spatial domain features layer by layer in the order from low to high using the PAN method to generate multi-level fused features.

[0140] Optionally, the device further includes:

[0141] The frequency domain feature generation module is used to construct a preprocessing layer, wherein the preprocessing layer includes performing two-dimensional Fourier transform, batch normalization processing, and activation function calculation on the current remote sensing image; and constructing a multi-level convolution operation module, wherein the channel dimension of the multi-level convolution operation module increases sequentially.

[0142] Optionally, the device further includes:

[0143] The spatial domain feature generation module is used to construct a CSPNet structure, wherein the CSPNet structure is used to sequentially generate multi-level spatial domain semantic features of the current remote sensing image in order of channel dimension from low to high; and to construct an FPN structure, wherein the FPN structure is used to fuse the semantic features of adjacent spatial domains in order of high to low levels to obtain multi-level spatial domain features with uniform semantic information distribution.

[0144] Optionally, the device further includes:

[0145] An optimization module construction module is used to add an optimization module during any of the processes of constructing the frequency domain feature extraction module, constructing the spatial domain feature extraction module, and applying the PAN method. The optimization module is used to optimize the convolution operation results to reduce memory consumption.

[0146] Optionally, the device further includes:

[0147] The fine-tuning module is used to fine-tune the frequency domain feature extraction module during the transfer learning process based on the domain differences of the remote sensing image data, so that the ship detection model can adapt to the remote sensing image data with domain differences.

[0148] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the ship detection method based on frequency domain feature fusion described in the embodiments.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0155] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0156] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A ship detection method based on frequency domain feature fusion, characterized in that, The method includes: The acquired current remote sensing image is input into a ship detection model, which is constructed based on a single-stage target detection method. Based on the current remote sensing image, a frequency domain feature extraction module pre-built in the ship detection model is used to generate frequency domain features corresponding to the current remote sensing image. The process of pre-building the frequency domain feature extraction module includes: constructing a preprocessing layer, wherein the preprocessing layer includes performing two-dimensional Fourier transform, batch normalization, and activation function calculation on the current remote sensing image; and constructing a multi-level convolution operation module, wherein the channel dimension of the multi-level convolution operation module increases sequentially. Based on the current remote sensing image, spatial domain features corresponding to the current remote sensing image are generated through a pre-constructed spatial domain feature extraction module in the ship detection model. This includes: generating multi-level spatial domain semantic features of the current remote sensing image sequentially in order of channel dimension from low to high using the CSPNet structure in the pre-constructed spatial domain feature extraction module; and fusing the semantic features of adjacent spatial domains level by level in order of layer from high to low using the FPN structure in the pre-constructed spatial domain feature extraction module to obtain multi-level spatial domain features with uniform semantic information distribution. The ship detection model generates fused features based on the frequency domain features and the spatial domain features, including: using the PAN method, splicing multi-level frequency domain features and multi-level spatial domain features layer by layer in order from low to high level to generate multi-level fused features; the multi-level fused features are spatial domain features in some channels and frequency domain features in some channels. The ship detection model generates the detection result corresponding to the current remote sensing image based on the fused features.

2. The method of claim 1, wherein, Based on the current remote sensing image, the frequency domain features corresponding to the current remote sensing image are generated through a pre-built frequency domain feature extraction module in the ship detection model, including: The frequency domain feature extraction module is pre-built in the ship detection model to extract multi-level frequency domain features from the current remote sensing image.

3. The method of claim 2, wherein, Based on the current remote sensing image, spatial domain features corresponding to the current remote sensing image are generated through a pre-built spatial domain feature extraction module in the ship detection model, including: The spatial domain feature extraction module is pre-built in the ship detection model to extract the spatial domain features of the current remote sensing image step by step. From the high-level to the low-level direction of the current remote sensing image, the spatial domain features of the current remote sensing image at adjacent levels are fused to enhance the semantic information of each level of features.

4. The method of claim 1, wherein, The method further includes: In any of the processes of constructing the frequency domain feature extraction module, constructing the spatial domain feature extraction module, and applying the PAN method, an optimization module is added, wherein the optimization module is used to optimize the convolution operation results to reduce memory consumption.

5. The method of claim 1, wherein, The method further includes: Based on the domain differences in remote sensing image data, the frequency domain feature extraction module is fine-tuned during the transfer learning process to adapt the ship detection model to remote sensing image data with domain differences.

6. A ship detection device based on frequency domain feature fusion, characterized in that, The device includes: The input module is used to input the acquired current remote sensing image into the ship detection model, wherein the ship detection model is constructed based on a single-stage target detection method; The frequency domain feature generation module is used to generate frequency domain features corresponding to the current remote sensing image based on the current remote sensing image and through the frequency domain feature extraction module pre-built in the ship detection model. The process of pre-building the frequency domain feature extraction module includes: building a preprocessing layer, wherein the preprocessing layer includes performing two-dimensional Fourier transform, batch normalization processing, and activation function calculation on the current remote sensing image; and building a multi-level convolution operation module, wherein the channel dimension of the multi-level convolution operation module increases sequentially. The spatial domain feature generation module is used to generate spatial domain features corresponding to the current remote sensing image based on the current remote sensing image and through the spatial domain feature extraction module pre-built in the ship detection model. The fusion module is used by the ship detection model to generate fused features based on the frequency domain features and the spatial domain features; The detection result generation module is used by the ship detection model to generate the detection result corresponding to the current remote sensing image based on the fused features; The spatial domain feature generation module is further configured to generate multi-level spatial domain semantic features of the current remote sensing image sequentially in order of channel dimension from low to high using the CSPNet structure in the pre-constructed spatial domain feature extraction module; and to fuse the semantic features of adjacent spatial domains step by step in order of high to low using the FPN structure in the pre-constructed spatial domain feature extraction module to obtain multi-level spatial domain features with uniform semantic information distribution. The fusion module is further configured to stitch together multi-level frequency domain features and multi-level spatial domain features layer by layer in a PAN manner, from low to high level, to generate multi-level fused features; the multi-level fused features are spatial domain features in some channels and frequency domain features in others.

7. A computer readable storage medium characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the ship detection method based on frequency domain feature fusion as described in any one of claims 1 to 5.

Images