Deep learning ship detection method based on visible light and SAR image fusion

Abstract

The invention discloses a deep learning ship detection method based on visible light and SAR image fusion. The method comprises the steps that firstly, registered visible light and SAR images are acquired to construct a multi-modal data set, a heterogeneous double-branch deep learning network is constructed, optical texture features and scattering structure features are extracted, and multi-scale initial feature representations are formed respectively; inputting the extracted features into a multi-modal attention enhancement module, and performing deep interaction on bimodal features to suppress noise and enhance complementary information; in the feature pyramid neck network, a single-scale deformable attention residual module is utilized to adaptively sense the geometric deformation of the ship, and a detection result is output; and finally, inputting the training set into the network, updating parameters through back propagation to obtain a trained detection model, and inputting the test set into the model to obtain a ship detection result. According to the method, the detection bottleneck of a single mode under complex sea conditions and severe weather is solved, and the detection precision and robustness of a multi-scale ship target are improved.

Description

Technical Field

[0001] This invention relates to the field of radar and optical remote sensing target detection technology, and in particular to a deep learning-based ship detection method based on the fusion of visible light and SAR images. Background Technology

[0002] With the deepening implementation of the maritime power strategy, maritime situational awareness has become a crucial link in safeguarding national maritime security and ensuring maritime traffic order. Currently, maritime surveillance mainly relies on two methods: visible light (RGB) and synthetic aperture radar (SAR). Visible light images have rich textures, making them easy for the human eye to interpret, but their image quality is highly susceptible to changes in lighting and weather. While SAR radar offers all-weather detection advantages, its imaging lacks color and texture information and is severely affected by speckle noise, resulting in a high false alarm rate. Single-mode sensors are no longer sufficient to meet the demands for high-precision detection in complex sea conditions.

[0003] In recent years, deep learning has made significant progress in the field of target detection, and multimodal fusion technology has gradually attracted attention. Invention patent CN 121236370 A discloses a method, system, and device for small target detection based on visible light and SAR fusion. This method involves adapting and preprocessing visible light images and synthetic aperture radar (SAR) images, performing cross-modal feature registration on the preprocessed images, enhancing the registered multimodal features using a bidirectional adversarial generative network, and adaptively fusing the enhanced multimodal features using a spatial-channel dual-path attention mechanism to generate a fused feature map. Small target detection is then performed based on this fused feature map. However, this method uses a homogeneous backbone network to extract features, ignoring the significant differences in the imaging mechanisms of the two modalities, resulting in weak feature extraction specificity. Furthermore, traditional feature fusion methods often involve simple channel stitching, lacking deep cross-modal interaction and filtering mechanisms. When a modality is affected by noise interference, invalid features can severely impact detection performance. Meanwhile, conventional network structures with fixed convolutional kernels are difficult to adapt to the characteristics of large aspect ratios and multiple rotations of sea surface targets, and are prone to introducing background noise.

[0004] Therefore, there is an urgent need to invent a ship detection method that can specifically adapt to the two completely different imaging mechanisms of visible light and SAR images, adopt more discriminative single-mode features, suppress invalid features of single modes in adverse weather or high-noise environments, and improve the accuracy and robustness of ship detection in complex environments. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based ship detection method based on the fusion of visible light and SAR images, which is robust, has high detection accuracy, and is highly adaptable to various environments.

[0006] The technical solution to achieve the purpose of this invention is: a deep learning-based ship detection method based on the fusion of visible light and SAR images, comprising the following steps:

[0007] Step 1: Obtain registered visible light images and SAR images of the sea area to be detected, construct a multimodal ship detection dataset, add data augmentation samples to the dataset, and divide the multimodal ship detection dataset into training set and test set;

[0008] Step 2: Construct a deep learning network based on dual-branch feature extraction. This network includes an RGB branch feature extraction network, a SAR branch feature extraction network, a multimodal enhanced attention module, a feature pyramid neck network, and a multi-scale detection head. Then, initialize the parameters of the deep learning network.

[0009] Step 3: Input the training set into the deep learning network, and use the RGB branch feature extraction network to extract optical texture features and the SAR branch feature extraction network to extract scattering structure features, forming a multi-scale initial feature representation;

[0010] Step 4: Input the initial feature representation of the multi-scale into the multimodal cross-enhanced channel attention module and the multimodal self-enhanced spatial attention module for feature fusion and calibration. Then, the detection box is adaptively processed through the feature pyramid neck network and the deformable attention residual module. Finally, the detection result is output through the multi-scale detection head and the loss is calculated. The network parameters are updated through backpropagation to obtain the trained ship detection model.

[0011] Step 5: Input the test set into the trained ship detection model to verify the performance of the ship detection model and obtain the ship category and location detection results.

[0012] Compared with the prior art, the present invention has the following significant advantages: (1) It adopts a heterogeneous dual-branch feature extraction architecture. The RGB branch uses CNN to capture texture details, and the SAR branch uses scattering perception Transformer to aggregate discrete scattering features and suppress speckle noise, thereby adapting to the two completely different imaging mechanisms and extracting more discriminative single-modal features; (2) It adopts a multimodal enhanced attention module, introduces the cross-enhancement mechanism of the channel dimension and the self-enhancement mechanism of the spatial dimension, and realizes deep interaction between modes by calculating the weight of the mixed features as a global adjustment factor, suppressing the invalid features of single modes in bad weather or high noise environment, and improving the robustness of fused features; (3) It introduces a deformable attention residual module in the neck of the feature pyramid to replace the traditional convolution unit. Using the single-scale deformable self-attention mechanism, it can adaptively adjust the feature sampling position according to the slender structure and rotational attitude of the ship target, solve the problem of limited receptive field of fixed convolution kernel, and further improve the detection and positioning accuracy of multi-scale and multi-attitude ship targets. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a deep learning-based ship detection method based on the fusion of visible light and SAR images according to the present invention.

[0014] Figure 2 This is a schematic diagram of the structure of the deep learning network based on dual-branch feature extraction constructed in this invention.

[0015] Figure 3 This is a partial data illustration of the visible light and SAR multimodal dataset constructed in an embodiment of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0017] like Figure 1 As shown, the present invention discloses a deep learning-based ship detection method based on visible light and SAR image fusion, comprising the following steps:

[0018] Step 1: Obtain registered visible light and SAR images of the sea area to be detected, construct a multimodal ship detection dataset, add data augmentation samples to the dataset, and divide the multimodal ship detection dataset into training and testing sets, as follows:

[0019] A multimodal ship detection dataset was constructed using a CycleGAN-based style transfer method. The LEVIR dataset was selected as the visible light domain data, and the HRSID and FUSAR datasets were selected as the SAR domain data. The CycleGAN network was used to learn the mapping relationship between the visible light and SAR image domains, and the visible light images were mapped to the SAR image domain, thereby constructing a semantically consistent and pixel-level registered dual-modal ship detection dataset.

[0020] Step 2: Construct a deep learning network based on dual-branch feature extraction. This network includes an RGB branch feature extraction network, a SAR branch feature extraction network, a multimodal enhanced attention module, a feature pyramid neck network, and a multi-scale detection head. Then, initialize the parameters of the deep learning network.

[0021] As a specific example, the basic building block of the RGB branch feature extraction network is the CBS module, which consists of concatenated convolutional layers, batch normalization layers, and SiLU activation functions. Its calculation formula is as follows:

[0022] (1)

[0023] In the formula, For the input feature map, This represents the convolution operation. Indicates batch normalization, This represents the activation function.

[0024] As a specific example, the SAR branch feature extraction network consists of multiple stacked scattering sensing Transformer modules, which include a window sparse self-attention mechanism and a denoising feedforward network.

[0025] The window-based sparse self-attention mechanism divides the feature map into non-overlapping windows and calculates self-attention only within each window. The calculation formula is as follows:

[0026] (2)

[0027] in These are query, key, and value vectors, respectively. This is the scaling factor, with a value of 64. For relative position encoding offset;

[0028] The denoising feedforward network is coupled with a depthwise separable convolutional branch in parallel with the first-layer linear mapping to suppress speckle noise and enhance local spatial correlation. Its calculation formula is as follows:

[0029] (3)

[0030] In the formula, Represents a linear mapping layer. This represents the activation function. This indicates a depthwise separable convolution operation.

[0031] As a specific example, the multimodal enhanced attention module includes a multimodal cross-enhanced channel attention module and a multimodal self-enhanced spatial attention module.

[0032] As a specific example, the feature pyramid neck network includes a deformable attention residual module. This module adopts a single-scale computational architecture adapted to the feature pyramid neck network. It is deployed in the P4 and P5 level semantic paths of the feature pyramid neck network, replacing the original C3k2 convolutional processing unit. Structurally, the deformable attention residual module removes the cross-attention operations between multi-scale levels and performs single-scale deformable self-attention computation only on the feature map of the current feature scale to reduce computational complexity and focus on the geometric deformation features of the ship target.

[0033] Step 3: Input the training set into the deep learning network, and use the RGB branch feature extraction network to extract optical texture features and the SAR branch feature extraction network to extract scattering structure features, forming a multi-scale initial feature representation;

[0034] Step 4: Input the initial feature representations at multiple scales into the multimodal cross-enhanced channel attention module and the multimodal self-enhanced spatial attention module for feature fusion and calibration. Then, perform adaptive processing of the detection boxes through the feature pyramid neck network and the deformable attention residual module. Finally, output the detection results through the multi-scale detection head and calculate the loss. Update the network parameters through backpropagation to obtain the trained ship detection model, as detailed below:

[0035] Step 4.1: The multimodal cross-enhanced channel attention module generates hybrid features through convolution and introduces hybrid feature weights as global adjustment factors. The original weights of RGB and SAR are dynamically calibrated, and a cross-injection strategy is adopted to inversely activate the features using the calibrated weights, as detailed below:

[0036] The multimodal cross-enhanced channel attention module first generates hybrid features through convolution and introduces hybrid feature weights as a global adjustment factor to dynamically calibrate the original weights of RGB and SAR. The formula for calculating the calibrated weights is as follows:

[0037] (4)

[0038] (5)

[0039] In the formula, , and These are the original channel weights generated by global average pooling and multilayer perceptron, respectively. Indicates broadcast multiplication;

[0040] Then, a cross-injection strategy is adopted, using the calibrated weights to inversely activate the features. The calculation formula is as follows:

[0041] (6)

[0042] (7)

[0043] In the formula, and For input features, and This refers to the output characteristics after channel enhancement.

[0044] Step 4.2: The multimodal self-enhancing spatial attention module calculates the channel averages of RGB and SAR features, concatenates them, and generates a shared global spatial context map through convolution and sigmoid activation. This constructs multi-scale local spatial response maps, which are then normalized using the global spatial context map. The normalized spatial attention map is then applied only to the corresponding modality, completing spatial self-enhancing, as detailed below:

[0045] The multimodal self-enhancing spatial attention module first calculates the channel averages of RGB and SAR features, concatenates them, and then generates a shared global spatial context graph through convolution and sigmoid activation. The calculation formula is:

[0046] (8)

[0047] (9)

[0048] In the formula, This represents the total number of channels, which is 512 based on the size of the input feature map. and This is a single-channel spatial response diagram. This represents the Sigmoid activation function. Indicates a splicing operation;

[0049] Then, multi-scale local spatial response maps are constructed and normalized using a global spatial context map. The calculation formula is as follows:

[0050] (10)

[0051] (11)

[0052] In the formula, This is a local spatial response diagram. This is the normalized spatial attention map. This indicates element-wise multiplication; the calculation process for the SAR mode is the same as that for the RGB mode.

[0053] Finally, the normalized spatial attention map is applied only to the corresponding modality itself to achieve spatial self-enhancement. The calculation formula is as follows:

[0054] (12)

[0055] (13)

[0056] In the formula, and This represents the final output feature of the multimodal enhanced attention module.

[0057] Step 4.3: The feature pyramid neck network and deformable attention residual module use fully connected layers to predict a set of offsets and weights for each attention head, and use bilinear interpolation to calculate the features of the sampling points for adaptive processing of the detection boxes, as detailed below:

[0058] The deformable attention residual module uses a fully connected layer to predict a set of offsets and weights for each attention head, and uses bilinear interpolation to calculate the features of the sampling points.

[0059] The formula for calculating the deformable self-attention at a single scale is as follows:

[0060] (14)

[0061] In the formula, To query elements, As the reference point coordinates, For input features; The number of attention heads is set to 8. This represents the number of sampling points, with a value of 4. and The first The first one in the attention. The predicted offset and attention weights for each sampling point; by restricting the sampling range in the formula to only the current single-scale feature map. The process is performed on top of the feature map without involving feature maps at other levels.

[0062] Step 4.4: Output the detection results through the multi-scale detection head and calculate the loss. Update the network parameters through backpropagation to obtain the trained ship detection model.

[0063] Step 5: Input the test set into the trained ship detection model to verify the performance of the ship detection model and obtain the ship category and location detection results.

[0064] Example

[0065] This embodiment provides a deep learning-based ship detection method based on the fusion of visible light and SAR images, such as... Figure 1 As shown, the specific steps include:

[0066] Step 1: Obtain registered visible light and SAR images of the sea area to be detected, construct a multimodal ship detection dataset, add data augmentation samples to the dataset, and divide the multimodal ship detection dataset into training and testing sets, as follows:

[0067] A style transfer network was used to simulate and generate pixel-level registered visible light and SAR image datasets. The visible light domain data was selected from the LEVIR dataset, and the SAR domain data was selected from the HRSID and FUSAR datasets. Since large-scale, rigorously registered heterogeneous data is difficult to obtain in real-world scenarios, this embodiment utilizes a CycleGAN network to learn the mapping relationship between the two image domains, mapping the abundant visible light data to the SAR image domain. During the simulation generation process, the input images were uniformly scaled to [resolution value missing]. To ensure the statistical characteristics of background noise in the generated SAR images are realistic, the ratio of sea surface to land images in the training samples was controlled at 8:2, the training batch size was set to 8, and the total number of training rounds was 200. The trained model was used for forward inference to construct the dataset, and finally a semantically consistent and pixel-level registered dual-modal ship detection dataset was constructed. The dataset was divided into a training set and a test set with a ratio of 7:3.

[0068] Step 2: Construct a deep learning network based on dual-branch feature extraction. This network includes an RGB branch feature extraction network, a SAR branch feature extraction network, a multimodal enhanced attention module, a feature pyramid neck network, and a multi-scale detection head, such as... Figure 2 As shown, the parameters of the deep learning network are initialized.

[0069] The RGB branch feature extraction network described above uses a convolutional neural network with CBS and C3k2 modules as its main components to extract texture features. The CBS module consists of a concatenated convolutional layer, a batch normalization layer, and a SiLU activation function, and its calculation formula is as follows:

[0070] (1)

[0071] In the formula, For the input feature map, This represents the convolution operation. Indicates batch normalization, This represents the activation function.

[0072] The SAR branch feature extraction network consists of multiple stacked scattering sensing Transformer modules, which include a window sparse self-attention mechanism and a denoising feedforward network.

[0073] The window-based sparse self-attention mechanism divides the feature map into non-overlapping windows, and calculates self-attention within each window using the following formula:

[0074] (2)

[0075] in These are query, key, and value vectors, respectively. This is the scaling factor, with a value of 64. For relative position encoding offset;

[0076] The denoising feedforward network is coupled with a depthwise separable convolutional branch in parallel with the first-layer linear mapping to suppress speckle noise and enhance local spatial correlation. Its calculation formula is as follows:

[0077] (3)

[0078] In the formula, Represents a linear mapping layer. This represents the activation function. This indicates a depthwise separable convolution operation.

[0079] As a specific example, the multimodal enhanced attention module includes a multimodal cross-enhanced channel attention module and a multimodal self-enhanced spatial attention module.

[0080] As a specific example, the feature pyramid neck network includes a deformable attention residual module. This module adopts a single-scale computational architecture adapted to the feature pyramid neck network. It is deployed in the P4 and P5 level semantic paths of the feature pyramid neck network, replacing the original C3k2 convolutional processing unit. Structurally, the deformable attention residual module removes the cross-attention operations between multi-scale levels and performs single-scale deformable self-attention computation only on the feature map of the current feature scale to reduce computational complexity and focus on the geometric deformation features of the ship target.

[0081] Step 3: Input the training set into the deep learning network, and use the RGB branch feature extraction network to extract optical texture features and the SAR branch feature extraction network to extract scattering structure features, forming a multi-scale initial feature representation;

[0082] Step 4: Input the initial feature representations at multiple scales into the multimodal cross-enhanced channel attention module and the multimodal self-enhanced spatial attention module for feature fusion and calibration. Then, perform adaptive processing of the detection boxes through the feature pyramid neck network and the deformable attention residual module. Finally, output the detection results through the multi-scale detection head and calculate the loss. Update the network parameters through backpropagation to obtain the trained ship detection model, as detailed below:

[0083] Step 4.1: The multimodal cross-enhanced channel attention module generates hybrid features through convolution and introduces hybrid feature weights as global adjustment factors. The original weights of RGB and SAR are dynamically calibrated, and a cross-injection strategy is adopted to inversely activate the features using the calibrated weights, as detailed below:

[0084] The multimodal cross-enhanced channel attention module first generates hybrid features through convolution and introduces hybrid feature weights as a global adjustment factor to dynamically calibrate the original weights of RGB and SAR. The formula for calculating the calibrated weights is as follows:

[0085] (4)

[0086] (5)

[0087] In the formula, , and These are the original channel weights generated by global average pooling and multilayer perceptron, respectively. Indicates broadcast multiplication;

[0088] Then, a cross-injection strategy is adopted, using the calibrated weights to inversely activate the features. The calculation formula is as follows:

[0089] (6)

[0090] (7)

[0091] In the formula, and For input features, and This refers to the output characteristics after channel enhancement.

[0092] Step 4.2: The multimodal self-enhancing spatial attention module calculates the channel averages of RGB and SAR features, concatenates them, and generates a shared global spatial context map through convolution and sigmoid activation. This constructs multi-scale local spatial response maps, which are then normalized using the global spatial context map. The normalized spatial attention map is then applied only to the corresponding modality, completing spatial self-enhancing, as detailed below:

[0093] The multimodal self-enhancing spatial attention module first calculates the channel averages of RGB and SAR features, concatenates them, and then generates a shared global spatial context graph through convolution and sigmoid activation. The calculation formula is:

[0094] (8)

[0095] (9)

[0096] In the formula, This represents the total number of channels, which is 512 based on the size of the input feature map. and This is a single-channel spatial response diagram. This represents the Sigmoid activation function. Indicates a splicing operation;

[0097] Then, multi-scale local spatial response maps are constructed and normalized using a global spatial context map. The calculation formula is as follows:

[0098] (10)

[0099] (11)

[0100] In the formula, This is a local spatial response diagram. This is the normalized spatial attention map. This indicates element-wise multiplication; the calculation process for the SAR mode is the same as that for the RGB mode.

[0101] Finally, the normalized spatial attention map is applied only to the corresponding modality itself to achieve spatial self-enhancement. The calculation formula is as follows:

[0102] (12)

[0103] (13)

[0104] In the formula, and This represents the final output feature of the multimodal enhanced attention module.

[0105] Step 4.3: The feature pyramid neck network and deformable attention residual module use fully connected layers to predict a set of offsets and weights for each attention head, and use bilinear interpolation to calculate the features of the sampling points for adaptive processing of the detection boxes, as detailed below:

[0106] The fused features are then processed through a feature pyramid neck network and a deformable attention residual module for adaptive bounding box processing. The deformable attention residual module utilizes a deformable multi-head attention mechanism to adaptively adjust the feature sampling position to adapt to the geometric deformation of the ship. The formula for calculating the deformable self-attention at a single scale is as follows:

[0107] (14)

[0108] In the formula, To query elements, As the reference point coordinates, For input features; The number of attention heads is set to 8. This represents the number of sampling points, with a value of 4. and The first The first one in the attention. The predicted offset and attention weights for each sampling point; by restricting the sampling range in the formula to only the current single-scale feature map. The process is performed on top of the feature map without involving feature maps at other levels.

[0109] Step 4.4: Output the detection results through the multi-scale detection head and calculate the loss. Update the network parameters through backpropagation to obtain the trained ship detection model.

[0110] Step 5: Input the test set into the trained ship detection model to verify the performance of the ship detection model and obtain the ship category and location detection results.

[0111] Table 1 shows the performance of ship multimodal detection under different network architectures. In this comparative experiment, YOLOv11 and RT-DETR were selected as single-modal baseline methods, and MCMA-Net and E2E-OSDet as multimodal fusion methods. Figure 3 This is a partial data illustration of the visible light and SAR multimodal dataset constructed in this embodiment. It can be seen that the method of this invention achieves an mAP50 of 93.6%, improving detection accuracy and demonstrating excellent performance in the high-precision positioning index mAP5095, while maintaining a low parameter count.

[0112] Table 1 Comparison of ship detection performance under different network architectures

[0113]

[0114] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based ship detection method based on visible light and SAR image fusion, characterized in that, Includes the following steps: Step 1: Obtain registered visible light images and SAR images of the sea area to be detected, construct a multimodal ship detection dataset, add data augmentation samples to the dataset, and divide the multimodal ship detection dataset into training set and test set; Step 2: Construct a deep learning network based on dual-branch feature extraction. This network includes an RGB branch feature extraction network, a SAR branch feature extraction network, a multimodal enhanced attention module, a feature pyramid neck network, and a multi-scale detection head. Then, initialize the parameters of the deep learning network. Step 3: Input the training set into the deep learning network, and use the RGB branch feature extraction network to extract optical texture features and the SAR branch feature extraction network to extract scattering structure features, forming a multi-scale initial feature representation; Step 4: Input the initial feature representation of the multi-scale into the multimodal cross-enhanced channel attention module and the multimodal self-enhanced spatial attention module for feature fusion and calibration. Then, the detection box is adaptively processed through the feature pyramid neck network and the deformable attention residual module. Finally, the detection result is output through the multi-scale detection head and the loss is calculated. The network parameters are updated through backpropagation to obtain the trained ship detection model. Step 5: Input the test set into the trained ship detection model to verify the performance of the ship detection model and obtain the ship category and location detection results.

2. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, Step 1 involves acquiring registered visible light and SAR images of the sea area to be detected and constructing a multimodal ship detection dataset, as detailed below: A multimodal ship detection dataset was constructed using a CycleGAN-based style transfer method. The LEVIR dataset was selected as the visible light domain data, and the HRSID and FUSAR datasets were selected as the SAR domain data. The CycleGAN network was used to learn the mapping relationship between the visible light and SAR image domains, and the visible light images were mapped to the SAR image domain, thereby constructing a semantically consistent and pixel-level registered dual-modal ship detection dataset.

3. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, The basic building block of the RGB branch feature extraction network described in step 2 is the CBS module, which consists of a concatenated convolutional layer, a batch normalization layer, and a SiLU activation function. The calculation formula is as follows: （1） In the formula, For the input feature map, This represents the convolution operation. Indicates batch normalization, This represents the activation function.

4. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, The SAR branch feature extraction network described in step 2 consists of multiple stacked scattering sensing Transformer modules, which include a window sparse self-attention mechanism and a denoising feedforward network. The window-based sparse self-attention mechanism divides the feature map into non-overlapping windows and calculates self-attention only within each window. The calculation formula is as follows: （2） in These are query, key, and value vectors, respectively. This is the scaling factor, with a value of 64. For relative position encoding offset; The denoising feedforward network is coupled with a depthwise separable convolutional branch in parallel with the first-layer linear mapping to suppress speckle noise and enhance local spatial correlation. Its calculation formula is as follows: （3） In the formula, Represents a linear mapping layer. This represents the activation function. This indicates a depthwise separable convolution operation.

5. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, The multimodal enhanced attention module mentioned in step 2 includes a multimodal cross-enhanced channel attention module and a multimodal self-enhanced spatial attention module.

6. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, The feature pyramid neck network described in step 2 includes a deformable attention residual module. This module adopts a single-scale computational architecture adapted to the feature pyramid neck network. It is deployed in the P4 and P5 level semantic paths of the feature pyramid neck network, replacing the original C3k2 convolutional processing unit. The deformable attention residual module removes the cross-attention operation between multi-scale levels and performs single-scale deformable self-attention computation only on the feature map of the current feature scale to reduce computational complexity and focus on the geometric deformation features of the ship target.

7. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 1, characterized in that, Step 4 describes inputting the initial feature representations at multiple scales into the multimodal cross-enhanced channel attention module and the multimodal self-enhanced spatial attention module for feature fusion and calibration. Subsequently, the detection boxes undergo adaptive processing via the feature pyramid neck network and the deformable attention residual module. Finally, the detection results are output through the multi-scale detection head, and the loss is calculated. Backpropagation updates the network parameters to obtain the trained ship detection model, as detailed below: Step 4.1: The multimodal cross-enhanced channel attention module generates hybrid features through convolution and introduces hybrid feature weights as global adjustment factors. The original weights of RGB and SAR are dynamically calibrated, and a cross-injection strategy is adopted to use the calibrated weights to perform inverse activation of the features. Step 4.2: The multimodal self-enhancing spatial attention module calculates the channel average values ​​of RGB features and SAR features, concatenates them, and generates a shared global spatial context map through convolution and Sigmoid activation. It then constructs multi-scale local spatial response maps and normalizes them using the global spatial context map. The normalized spatial attention map is then applied only to the corresponding modality itself, thus completing spatial self-enhancing. Step 4.3: The feature pyramid neck network and deformable attention residual module use fully connected layers to predict the offset and weight of a set of sampling points for each attention head, and use bilinear interpolation to calculate the features of the sampling points for adaptive processing of the detection box. Step 4.4: Output the detection results through the multi-scale detection head and calculate the loss. Update the network parameters through backpropagation to obtain the trained ship detection model.

8. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 7, characterized in that, Step 4.1 describes a multimodal cross-enhanced channel attention module that generates hybrid features through convolution and introduces hybrid feature weights as a global adjustment factor. The original weights of RGB and SAR are dynamically calibrated, and a cross-injection strategy is employed to inversely activate the features using the calibrated weights, as detailed below: The multimodal cross-enhanced channel attention module first generates hybrid features through convolution and introduces hybrid feature weights as a global adjustment factor to dynamically calibrate the original weights of RGB and SAR. The formula for calculating the calibrated weights is as follows: （4） （5） In the formula, , and These are the original channel weights generated by global average pooling and multilayer perceptron, respectively. Indicates broadcast multiplication; Then, a cross-injection strategy is adopted, using the calibrated weights to inversely activate the features. The calculation formula is as follows: （6） （7） In the formula, and For input features, and This refers to the output characteristics after channel enhancement.

9. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 7, characterized in that, Step 4.2 describes a multimodal self-enhancing spatial attention module that calculates the channel averages of RGB and SAR features, concatenates them, and generates a shared global spatial context map through convolution and sigmoid activation. This constructs multi-scale local spatial response maps, which are then normalized using the global spatial context map. The normalized spatial attention map is then applied only to the corresponding modality, completing spatial self-enhancing, as detailed below: The multimodal self-enhancing spatial attention module first calculates the channel averages of RGB and SAR features, concatenates them, and then generates a shared global spatial context graph through convolution and sigmoid activation. The calculation formula is: （8） （9） In the formula, This represents the total number of channels, which is 512 based on the size of the input feature map. and This is a single-channel spatial response diagram. This represents the Sigmoid activation function. Indicates a splicing operation; Then, multi-scale local spatial response maps are constructed and normalized using a global spatial context map. The calculation formula is as follows: （10） （11） In the formula, This is a local spatial response diagram. This is the normalized spatial attention map. This indicates element-wise multiplication; the calculation process for the SAR mode is the same as that for the RGB mode. Finally, the normalized spatial attention map is applied only to the corresponding modality itself to achieve spatial self-enhancement. The calculation formula is as follows: （12） （13） In the formula, and This represents the final output feature of the multimodal enhanced attention module.

10. The deep learning-based ship detection method based on visible light and SAR image fusion according to claim 7, characterized in that, Step 4.3 describes a feature pyramid neck network and a deformable attention residual module that use fully connected layers to predict a set of offsets and weights for each attention head, and uses bilinear interpolation to calculate the features of the sampling points for adaptive processing of the detection boxes, as detailed below: The deformable attention residual module uses a fully connected layer to predict a set of offsets and weights for each attention head, and uses bilinear interpolation to calculate the features of the sampling points. The formula for calculating the deformable self-attention at a single scale is as follows: （14） In the formula, To query elements, As the reference point coordinates, For input features; The number of attention heads is set to 8. This represents the number of sampling points, with a value of 4. and The first The first one in the attention. The predicted offset and attention weights for each sampling point; By limiting the sampling range in the formula to only the current single-scale feature map. The process is performed on top of the feature map without involving feature maps at other levels.

Images