Ship detection method of SAR-optical multi-mode fusion network based on perceptual alignment

Abstract

The invention discloses a ship detection method of an SAR-optical multi-mode fusion network based on perceptual alignment. The ship detection method comprises five parts of preparing a data set, constructing the perceptual alignment multi-mode fusion network based on YOLOv8, establishing a ship detection model, testing the ship detection model and evaluating the ship detection model. On the basis of an original YOLOv8 network, an SAR-optical multi-mode fusion network based on perceptual alignment is constructed, that is, three mechanisms of a feature alignment and fusion module, uncertainty perceptual shape IoU loss and a distance matching probability fusion algorithm are introduced on the basis of the original YOLOv8, full utilization of perceptual alignment in ship detection is achieved, and the accuracy of ship detection is improved. Therefore, the ship detection model provided by the invention has better detection precision. Compared with existing six most advanced ship detection methods, the SAR-optical ship detection method has the advantage that higher SAR-optical ship detection precision is realized.

Description

Technical Field

[0001] This invention belongs to the field of multimodal image interpretation technology of Synthetic Aperture Radar (SAR) and high-resolution optical images, and relates to a ship detection method based on a perception-aligned SAR-optical multimodal fusion network. Background Technology

[0002] Remote sensing-based ship detection has become a highly influential tool with wide applications in various marine fields, including search and rescue operations, environmental monitoring, fisheries resource management, and trade and transportation logistics. Accurate and reliable ship identification has been achieved through the fusion of high-resolution optical imagery and synthetic aperture radar (SAR) data acquired from advanced satellite platforms. This capability is primarily attributed to the inherent advantages of these remote sensing modalities, particularly their wide coverage and all-weather operation. See the literature “Xu, Xiaowo, et al. Distribution-Based Anchor Assignment and Comprehensive Score Voting with Distance-Penalty IoU Loss for SAR RemoteSensing Ship Detection.” IEEE Transactions on Instrumentation and Measurement (2024).

[0003] The fusion of optical and SAR images has been extensively studied in remote sensing and is widely recognized for its ability to improve image interpretation and target detection performance. Multi-source data fusion collaboratively integrates complementary information from different sensors, generating richer data representations and significantly improving the accuracy and robustness of ship detection. SAR systems are robust in adverse weather and low-light conditions, while optical sensors perform well in good lighting conditions. By combining these two modalities, fusion-based detection methods achieve greater adaptability in varying atmospheric and lighting scenarios. Furthermore, integrating heterogeneous features extracted from both sensors helps mitigate the inherent limitations of single-sensor systems, such as false alarms and missed detections, thereby improving the reliability of detection algorithms.

[0004] An ideal approach to multimodal data acquisition is to integrate optical and SAR sensors onto a single satellite platform, employing consistent observation geometry and synchronized imaging time. This configuration ensures the generation of spatially and temporally aligned optical and SAR images. However, due to the large size, weight, and power consumption requirements of high-resolution optical and SAR sensors, co-locating them on the same satellite remains technically infeasible under current engineering constraints.

[0005] If optical and SAR sensors can be co-located on the same satellite platform while ensuring consistency between observation geometry and imaging timing, spatiotemporally registered multimodal images can be obtained. However, the enormous requirements for size, mass, and power consumption of high-resolution optical and SAR sensors, as well as the fundamental differences in observation geometry between the two types of sensors (SAR often uses side-looking sensors, while optical sensors are often preferred for near-zenith views), make it difficult to achieve ideal co-location on the same platform in engineering, and often results in at least one of them operating under suboptimal conditions. In practice, optical and SAR images are usually acquired in separate time slots and on separate tracks, resulting in differences between observation geometry and acquisition time. Although geometric registration can alleviate spatial misalignment, temporal deviations can lead to scene changes, resulting in partially registered and significantly misaligned target pairs.

[0006] Therefore, to address the aforementioned problems, this invention proposes a ship detection method based on a perception-aligned SAR-optical multimodal fusion network. This method integrates three improved fusion components at multiple levels. This hybrid architecture effectively combines optical and SAR information to improve detection performance under misalignment conditions. Summary of the Invention

[0007] This invention belongs to the field of multimodal image interpretation technology of synthetic aperture radar (SAR) and high-resolution optical images. It discloses a ship detection method based on a perceptually aligned SAR-optical multimodal fusion network to address the insufficient accuracy of existing detection techniques. The method mainly includes five parts: preparing the dataset, constructing a perceptually aligned multimodal fusion network based on YOLOv8, establishing a ship detection model, testing the ship detection model, and evaluating the ship detection model. Based on the original YOLOv8 network, this method constructs a perceptually aligned SAR-optical multimodal fusion network, thereby optimizing the network structure to improve accuracy. Experimental results on the well-known SAR-optical ship dataset (i.e., the QXS-SAROPT dataset) show that, compared with six state-of-the-art ship detection methods, including optical, SAR-based, and multimodal fusion models, this invention achieves higher SAR-optical ship detection accuracy.

[0008] To facilitate the description of the present invention, the following terms are defined first:

[0009] Definition 1: QXS-SAROPT dataset

[0010] The QXS-SAROP dataset is a SAR-optical fusion ship detection dataset that can be used to train deep learning models, allowing researchers to evaluate the performance of their algorithms on this unified dataset. The QXS-SAROP dataset contains 20,000 pairs of SAR and optical images extracted from multiple high-resolution Gaofen-3 and Google Earth scenes. QXS-SAROPT is the first dataset to provide very high-resolution (1m) SAR and optical satellite image patches registered across three major port cities globally. The QXS-SAROPT dataset can be obtained from the reference "Huang M, Xu Y, Qian L, et al. The QXS-SAROPT dataset for deep learning in SAR-optical datafusion[J]. arxiv preprint arxiv:2103.08259,2021."

[0011] Definition 2: Classical Convolutional Neural Network Methods

[0012] Classical convolutional neural networks (CNNs) refer to a class of feedforward neural networks that include convolutional computations and have a deep structure. CNNs are constructed by mimicking the visual perception mechanisms of biological systems, enabling both supervised and unsupervised learning. The shared parameters of the convolutional kernels within their hidden layers and the sparsity of inter-layer connections allow CNNs to extract features with relatively low computational cost. In recent years, CNNs have made rapid progress in computer vision, natural language processing, and speech recognition, and their powerful feature learning capabilities have attracted widespread attention from experts and scholars both domestically and internationally. For details on classic CNN methods, please refer to the literature “Zhang Suofei, Feng Ye, Wu Xiaofu. Progress in target detection algorithms based on deep convolutional neural networks [J / OL]. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2019(05):1-9. https: / / doi.org / 10.14132 / j.cnki.1673-5439.2019.05.010.”

[0013] Definition 3: Classic CNN Feature Extraction Method

[0014] Classical CNN feature extraction involves using a CNN to extract features from the original input image. In short, the original input image is transformed into a series of feature maps through convolutional operations on different features. In a CNN, the convolutional kernels in the convolutional layers continuously slide across the image for computation. Simultaneously, the max-pooling layer is responsible for taking the maximum value of each local block in the inner product result. Therefore, CNNs implement image feature extraction methods through convolutional layers and max-pooling layers. For a detailed explanation of classic CNN feature extraction, please refer to the website "https: / / blog.csdn.net / qq_30815237 / article / details / 86703620".

[0015] Definition 4: YOLOv8

[0016] YOLOv8 is a one-stage object detection algorithm that performs dense predictions directly on the feature map without using explicit region proposals or RoI alignment. The entire YOLOv8 algorithm can be divided into four parts: the first part is the backbone network, used to extract multi-scale features from the entire image; the second part is dense candidate generation (grid / anchor-free), used to directly regress candidate boxes and object confidence on the feature map; the third part is feature alignment / fusion (Neck), used for upsampling / downsampling and fusing multi-scale features; and the fourth part is the classification and regression network (Head) and post-processing, used for post-processing such as class prediction, bounding box refinement, and non-maximum suppression of candidate boxes. For details on YOLOv8, please see the literature "Muhammad Yaseen.What isYOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector.arXiv(Cornell University).2024;:undefined-undefined.doi:10.48550 / arxiv.2408.15857".

[0017] Definition 5: Convolutional Kernel

[0018] In image processing, a convolution kernel is a weighted average of pixels in a small region of an input image, which is then used to produce the corresponding pixels in the output image. The weights are defined by a function called the convolution kernel. The role of the convolution kernel is to extract features. A larger convolution kernel size means a larger receptive field, but of course, more parameters are required. As early as 1998, LeCun's LetNet-5 model showed that there are local correlations in the spatial domain of an image, and the convolution process is a way to extract these local correlations. For details on how to set the convolution kernel, please refer to the literature "LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.".

[0019] Definition 6: Classic method for setting convolution kernel size

[0020] The kernel size refers to the length, width, and depth of the convolution kernel, denoted as L×W×D, where L represents the length, W represents the width, and D represents the depth. Setting the kernel size means determining the specific values ​​of L, W, and D. Generally, to achieve the same receptive field, the smaller the kernel, the fewer parameters and computational cost are required. Specifically, the length and width of the kernel must be greater than 1 to increase the receptive field. Even with symmetrical zero-padding, kernels with even-sized kernels cannot guarantee that the input and output feature spectrum sizes remain unchanged. Generally, 3 is used as the kernel size. For details on setting the kernel size, please refer to the literature "Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.".

[0021] Definition 7: Classic method for setting the stride of a convolution kernel

[0022] The kernel stride refers to the length of the convolution kernel in each movement, denoted as S. Setting the kernel stride means determining the specific value of S. Generally, the larger the stride, the fewer features are extracted; conversely, the more features are extracted. Convolutional layers typically use a stride of 1, while max pooling layers typically use a stride of 2. For classic kernel stride setting methods, please refer to the literature "Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324."

[0023] Definition 8: Classical Convolutional Layer

[0024] A convolutional layer consists of several convolutional units, each with parameters optimized using the backpropagation algorithm. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract low-level features such as edges, lines, and corners, while more layers can iteratively extract more complex features from these low-level features. For a detailed explanation of classic convolutional layers, please refer to the website "https: / / www.zhihu.com / question / 49376084".

[0025] Definition 9: Standard Deformed Convolution Method

[0026] In deformable convolution, the standard convolution kernel enhances the offset Δp. n And during training, it adaptively learns to model the shape features of the target, and the calculation formula is: in, Indicates in p n Summation operator under the condition ∈R, p0 represents the initial position, p n Let w(p) represent the position of the nth pixel, R represent the convolution region, and w(p) represent the position of the nth pixel. n ) represents p n The corresponding weight parameter, Δp n Let represent the offset learned at the nth pixel position, and y(p0) represent the final output. For details on the standard deformable convolution method, please refer to the literature "Dai, J.;Qi, H.;Xiong, Y.;Li, Y.;Zhang, G.;Hu, H.;Wei, Y. Deformable Convolutional Networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017;pp.764–773."

[0027] Definition 10: The classic Adam algorithm

[0028] The classic Adam algorithm is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It iteratively updates the weights of a neural network based on training data. The Adam algorithm differs from traditional stochastic gradient descent. Stochastic gradient descent updates all weights with a single learning rate that remains unchanged during training. Adam, however, designs independent adaptive learning rates for different parameters by calculating the first and second moment estimates of the gradient. See the literature "Kingma, D.; Ba, J. Adam: AMethod for Stochastic Optimization. arXiv 2014, arXiv:1412.6980." for details.

[0029] Definition 11: Standard Content-Driven Attention Module Approach

[0030] Content-Guided Attention (CGA) obtains a unique SIM for each channel of the input features in a coarse-to-fine manner, while fully mixing channel attention weights and spatial attention weights to ensure information interaction. CGA assigns a unique SIM to each channel, guiding the model to focus on the important regions of each channel. Therefore, it can emphasize more useful information encoded in the features, effectively improving accuracy. For details on the standard Content-Guided Attention (CGA) module method, please refer to the literature "Zixuan Chen, Zewei He, Zhe-Ming Lu. DEA-Net: Single Image Dehazing Based on Detail-Enhanced Convolution and Content-Guided Attention. IEEE transactions on image processing: a publication of the IEEE Signal Processing Society. 2024;:undefined-undefined.doi:10.1109 / TIP.2024.3354108."

[0031] Definition 12: Standard Uncertainty-Aware Shape IoU Loss Method

[0032] Uncertainty-aware Shape-IoU Loss (US-Loss) aims to address the labeling misalignment problem between optical and SAR modalities in multimodal ship detection. This loss function comprises two main components: uncertainty quantization and shape-IoU localization, designed to improve the robustness and accuracy of multimodal target detection. Uncertainty quantization dynamically quantifies the reliability of bounding box annotations in RGB-SAR image centering using a cross-modal IoU metric. In target detection, IoU measures the localization accuracy between predicted and ground truth bounding boxes. The cross-modal IoU CM... IoU Defined as the IoU between the bounding boxes annotated in RGB-SAR image pairs: Box SAR and Box RGB These represent the ground truth bounding boxes in the SAR and RGB images, respectively. The formula for calculating the shape-IoU is: L Shape-NWDIoU =1-IoU+NWD+0.5×Ω shape For details on the standard uncertainty-aware shape IoU loss (US-Loss) method, please refer to the literature "Zhang, Hao, and Shuaijie Zhang. Shape-iou: More accurate metric considering bounding box shape and scale." arXiv preprint arXiv:2312.17663 (2023)."

[0033] Definition 13: Standard Distance Matching Probability Fusion Method

[0034] The standard distance matching probabilistic fusion (DMP-Fusion) method employs the distance-intersection-union (D-IoU) method and the Hungarian matching algorithm. D-IoU captures spatial and geometric differences by considering the distance between the center points when the bounding boxes do not overlap. The formula for calculating D-IoU is: Where A and B are detection boxes in different modes, b A ,b B Let represent the center point of the detection box, ρ represent the Euclidean distance between the two center points, and c represent the diagonal distance of the smallest rectangle that can simultaneously cover both detection boxes. To calculate the robust confidence score of the matching detection, Bayesian fusion is used, and the final formula for calculating its posterior probability is: p represents the probability of detecting the current result given that the target exists. i∈[0,1] represents the detection confidence of the i-th modality. The distance-matching probability fusion algorithm ensures that the unique features and prior information inherent in each modality are preserved, thus achieving a more robust and informative fusion result. Furthermore, it does not require multimodal data during the training phase, thus exhibiting stronger robustness to data misalignment and modality-specific degradation. Even in cases of target absence or false detection in a particular modality, decision fusion can still obtain accurate results by utilizing the detection results of other modalities. For details on Bayesian fusion, please refer to the literature “Zhao, Zixiang, et al. Bayesian fusion for infrared and visible images. Signal Processing 177(2020):107734.”

[0035] Definition 14: Standard testing method for network detection

[0036] The standard method for testing detection networks refers to performing a final test on a test set to obtain the detection results of the detection model on the test set. See the literature "C. Lu, and W. Li, "Ship Classification in High-Resolution SAR Images via Transfer Learning with Small Training Dataset," Sensors, vol. 19, no. 1, pp. 63, 2018."

[0037] Definition 15: Standard Evaluation Index Calculation Method

[0038] Precision P refers to the number of correctly predicted samples out of all predicted positive samples. Precision P is defined as follows: Where # represents a number, TP represents a true instance, and FP represents a true negative instance;

[0039] Recall R refers to the number of correctly predicted samples out of all positive samples. Recall R is defined as follows: Where # represents a number, TP represents a true negative, and FN represents a false negative;

[0040] Mean precision (mAP) balances both the precision (P) and recall (R) of the detection model. Mean precision (mAP) is defined as follows: Where P(R) represents a function with R as the independent variable and P as the dependent variable;

[0041] For details on how to calculate the above parameters, please refer to the reference "Li Hang. Statistical Learning Methods [M]. Beijing: Tsinghua University Press, 2012.". Definition 16: Classical weighted matrix W to be learned O

[0042] The weighted matrix W to be learned O That is, the matrix parameters are self-learned during CNN training. In deep learning, these matrix parameters are those that need to be updated by the optimizer during backpropagation, and are called parameters (such as weights). In this invention, the weighting matrix W... O This is achieved by calling the `torch.nn.Parameter` function in Python and training a 1×1 convolutional layer. The aim is to continuously modify the values ​​of the matrix parameter variables during the learning process to achieve optimization. The classic weighted matrix W to be learned... O For details, please see the website "https: / / www.jianshu.com / p / d8b77cc02410".

[0043] Definition 17: Paired optical images

[0044] In this invention, the term "pairing" refers to the correspondence between a SAR image and an optical image in the QXS-SAROPT dataset. Specifically, a paired image pair consists of a SAR image and an optical image that capture the same geographic area or scene and have undergone preprocessing (such as geometric correction, spatial registration, and radiometric correction) to ensure they are aligned at the pixel or feature level. The optical image in this case is called the paired optical image of the SAR image.

[0045] Definition 18: A method for combining traditional feature alignment and fusion modules with the YOLOv8 backbone network

[0046] In this invention, the "Feature Alignment and Fusion Module (FAFM)" is a modular network unit used to accept intermediate features from two modalities and output fused features at the same scale as the backbone network. According to this method, the combination of FAFM and the YOLOv8 backbone network follows these steps:

[0047] 1. FAFM is connected to the YOLOv8 backbone at the fifth, seventh, and ninth intermediate scales. FAFM receives two intermediate features from optical and SAR images at these scales.

[0048] 2. FAFM first applies deformable convolutional units as defined in Definition 9 to the two features respectively, and adaptively adjusts the local sampling position of each feature by learning the sampling offset, thereby reducing the geometric or projectional differences between the two features and obtaining the aligned two features.

[0049] 3. After merging the aligned features from the two channels, input the content described in Definition 11 to guide the attention module. This module adaptively generates attention weights based on the feature content to highlight important semantics and suppress modality-specific noise, and outputs weighted fused candidate features.

[0050] Definition 19: Hungarian Algorithm

[0051] The Hungarian algorithm is a deterministic, multinomial-time algorithm used to find complete one-to-one matchings from a given cost matrix in the bipartite matching (or allocation) problem, which minimizes the overall cost (or maximizes the overall benefit). This algorithm constructs a set of one-to-one allocation schemes by applying row and column transformations to the cost matrix, finding augmenting or alternating paths, and marking and adjusting uncovered elements, ultimately achieving optimal allocation costs. For a detailed explanation of the Hungarian algorithm steps, please refer to "https: / / blog.csdn.net / chichoxian / article / details / 147059278".

[0052] This invention provides a ship detection method based on a perception-aligned SAR-optical multimodal fusion network, which includes the following steps:

[0053] Step 1: Prepare the dataset

[0054] For the well-known QXS-SAROPT dataset provided in Definition 1, it is registered and cropped into 463 pairs of images, each with a spatial resolution of 256×256 pixels, to obtain a new QXS-SAROPT dataset, denoted as QXS-SAROPT_new.

[0055] The QXS-SAROPT_new dataset is divided into three parts in a 7:1:2 ratio to obtain a training set, a validation set, and a test set. The training set is denoted as Train_QXS-SAROPT, the validation set as Val_QXS-SAROPT, and the test set as Test_QXS-SAROPT.

[0056] Step 2: Construct a perceptual alignment multimodal fusion network based on YOLOv8

[0057] Step 2.1: Construct the backbone network, neck network, and detection head in the YOLOv8 method.

[0058] Using the backbone network, neck network, and detection head from the classic YOLOv8 method in Definition 4, a perceptually aligned multimodal fusion network is established, denoted as f1.

[0059] Step 2.2: Construct the uncertainty-aware shape IoU loss

[0060] The standard uncertainty-aware shape IoU loss method defined in Definition 12 is used to perform cross-modal IoU matching on any SAR image and its paired optical image in the training set Train_QXS-SAROPT obtained in step 1, and the IoU loss is obtained.

[0061] Step 2.3: Construct the Feature Alignment and Fusion Module (FAFM)

[0062] Using the standard deformable convolution method in Definition 9 and the standard content-guided attention module method in Definition 11, a feature alignment and fusion module (FAFM) is established, denoted as f2. The grouped feature fusion module f2 consists of two deformable convolutions from Definition 9 and one content-guided attention module from Definition 11, denoted as C1, C2, and C3 respectively.

[0063] By combining the traditional feature alignment and fusion module in Definition 18 with the YOLOv8 backbone network, C1, C2, and C3 are combined with the backbone network in the classic YOLOv8 network in Definition 4 to obtain a new backbone network.

[0064] Step 2.4: Construct a distance matching probability fusion algorithm

[0065] Using the classic CNN feature extraction method in Definition 3, the feature extraction sub-network processes any SAR image and its paired optical image in the training set Train_QXS-SAROPT obtained in step 1, respectively, and the SAR image features are denoted as F1 and the paired optical image features are denoted as F2.

[0066] The D-IoU calculation formula in the standard distance matching probability fusion method of Definition 13 is adopted. The value matrix is ​​obtained by performing D-IoU calculation on the F1 and F2 features;

[0067] Then, the Hungarian algorithm in Definition 19 is used for optimal allocation;

[0068] The Bayesian fusion calculation formula in the standard distance matching probability fusion method of Definition 13 is adopted. The posterior probability of the target is calculated.

[0069] This completes the construction of the perceptual alignment multimodal fusion network.

[0070] Step 3: Establish a ship detection model

[0071] On the multimodal fusion network completed in step 2, the training set Train_QXS-SAROPT and the validation set Val_QXS-SAROPT obtained in step 1 are used as inputs, and the classic Adam algorithm in Definition 10 is used for training. After training, the ship detection model is obtained, denoted as MHFNet YOLOv8.

[0072] Step 4: Test the ship detection model

[0073] On the ship detection model MHFNet YOLOv8 obtained in step 3, the test set Test_QXS-SAROPT obtained in step 1 is tested using the standard detection network testing method in definition 14, and the test result of the test set on the ship detection model is obtained, denoted as Result.

[0074] Step 5: Evaluate the ship detection model

[0075] Using the test results Result of the ship detection model obtained in step 4 as input, the standard evaluation index calculation method in Definition 15 is used to calculate the precision, recall, mean precision and F1 score, which are denoted as P, R, mAP and F1 respectively.

[0076] This concludes the entire method.

[0077] The innovation of this invention lies in introducing three mechanisms on the basis of the original YOLOv8: feature alignment and fusion module, uncertainty-aware shape IoU loss, and distance matching probability fusion algorithm. These mechanisms fully utilize perceptual alignment in ship detection, resulting in a ship detection model with superior detection accuracy. Extensive experiments on the QXS-SAROPT dataset demonstrate that this method outperforms state-of-the-art methods, improving average accuracy by 4.77% and F1 score by 4.93%.

[0078] The advantage of this invention is that it fully utilizes perceptual alignment in ship detection and provides a method for ship target detection in SAR-optical multimodal images to solve the problem of insufficient accuracy in existing ship target detection. Attached Figure Description

[0079] Figure 1 This is a schematic flowchart illustrating the method for detecting ship targets in SAR-optical images according to the present invention.

[0080] Figure 2 This table compares the ship target detection method in SAR-optical images of the present invention with six other ship detector methods in terms of precision, recall, mean precision, and F1 score on the QXS-SAROPT dataset. Detailed Implementation

[0081] The following is in conjunction with the appendix Figure 1 The present invention will be described in further detail below.

[0082] Step 1: Prepare the dataset

[0083] For the well-known QXS-SAROPT dataset provided in Definition 1, it is registered and cropped into 463 pairs of images, each with a spatial resolution of 256×256 pixels, to obtain a new QXS-SAROPT dataset, denoted as QXS-SAROPT_new.

[0084] The QXS-SAROPT_new dataset is divided into three parts in a 7:1:2 ratio to obtain a training set, a validation set, and a test set. The training set is denoted as Train_QXS-SAROPT, the validation set as Val_QXS-SAROPT, and the test set as Test_QXS-SAROPT.

[0085] Step 2: Construct a perceptual alignment multimodal fusion network based on YOLOv8

[0086] Step 2.1: Construct the backbone network, neck network, and detection head in the YOLOv8 method.

[0087] Using the backbone network, neck network, and detection head from the classic YOLOv8 method in Definition 4, a perceptually aligned multimodal fusion network is established, denoted as f1.

[0088] Step 2.2: Construct the uncertainty-aware shape IoU loss

[0089] The standard uncertainty-aware shape IoU loss method defined in Definition 12 is used to perform cross-modal IoU matching on any SAR image and its paired optical image in the training set Train_QXS-SAROPT obtained in step 1, and the IoU loss is obtained.

[0090] Step 2.3: Construct the Feature Alignment and Fusion Module (FAFM)

[0091] Using the standard deformable convolution method in Definition 9 and the standard content-guided attention module method in Definition 11, a feature alignment and fusion module (FAFM) is established, denoted as f2. The grouped feature fusion module f2 consists of two deformable convolutions from Definition 9 and one content-guided attention module from Definition 11, denoted as C1, C2, and C3 respectively.

[0092] By combining the traditional feature alignment and fusion module in Definition 18 with the YOLOv8 backbone network, C1, C2, and C3 are combined with the backbone network in the classic YOLOv8 network in Definition 4 to obtain a new backbone network.

[0093] Step 2.4: Construct a distance matching probability fusion algorithm

[0094] Using the classic CNN feature extraction method in Definition 3, the feature extraction sub-network processes any SAR image and its paired optical image in the training set Train_QXS-SAROPT obtained in step 1, respectively, and the SAR image features are denoted as F1 and the paired optical image features are denoted as F2.

[0095] The D-IoU calculation formula in the standard distance matching probability fusion method of Definition 13 is adopted. The value matrix is ​​obtained by performing D-IoU calculation on the F1 and F2 features;

[0096] Then, the Hungarian algorithm in Definition 19 is used for optimal allocation;

[0097] The Bayesian fusion calculation formula in the standard distance matching probability fusion method of Definition 13 is adopted. The posterior probability of the target is calculated.

[0098] This completes the construction of the perceptual alignment multimodal fusion network.

[0099] Step 3: Establish a ship detection model

[0100] On the multimodal fusion network completed in step 2, the training set Train_QXS-SAROPT and the validation set Val_QXS-SAROPT obtained in step 1 are used as inputs, and the classic Adam algorithm in Definition 10 is used for training. After training, the ship detection model is obtained, denoted as MHFNet YOLOv8.

[0101] Step 4: Test the ship detection model

[0102] On the ship detection model MHFNet YOLOv8 obtained in step 3, the test set Test_QXS-SAROPT obtained in step 1 is tested using the standard detection network testing method in definition 14, and the test result of the test set on the ship detection model is obtained, denoted as Result.

[0103] Step 5: Evaluate the ship detection model

[0104] Using the test results Result of the ship detection model obtained in step 4 as input, the standard evaluation index calculation method in Definition 15 is used to calculate the precision, recall, mean precision and F1 score, which are denoted as P, R, mAP and F1 respectively.

[0105] This concludes the entire method.

[0106] like Figure 2As shown, the average accuracy achieved by this invention on the well-known QXS-SAROPT dataset is 87.95%. Furthermore, this invention achieves higher detection accuracy than six existing state-of-the-art ship detection methods, demonstrating that it can achieve high-precision ship target detection in SAR-optical images.

Claims

1. A ship detection method based on perception-aligned SAR-optical multimodal fusion network, characterized by its... Comprising the following steps: Step 1, preparing the data set For the known QXS-SAROPT data set provided in definition 1, register and crop 463 pairs of images, each pair with a spatial resolution of 256x256 pixels, to obtain a new QXS-SAROPT data set, denoted as QXS-SAROPT_new; According to the ratio of 7:1:2, the QXS-SAROPT_new data set is divided into three parts, and the training set, validation set and test set are obtained, the training set is denoted as Train_QXS-SAROPT, the validation set is denoted as Val_QXS-SAROPT, and the test set is denoted as Test_QXS-SAROPT. Step 2, constructing a perception-aligned multi-modal fusion network based on YOLOv8 Step 2.1: Constructing the backbone network, neck network and detection head in the YOLOv8 method The backbone network, neck network and detection head in the classical YOLOv8 method in definition 4 are used to establish a perception-aligned multi-modal fusion network, denoted as f1. Step 2.2: Constructing an uncertainty-aware shape IoU loss The standard uncertainty-aware shape IoU loss method in definition 12 is used to match the IoU of any SAR image and the paired optical image in the training set Train_QXS-SAROPT obtained in step 1, to obtain the IoU loss. Step 2.3: Constructing a feature alignment and fusion module (FAFM) The standard deformation convolution method in definition 9 and the standard content-guided attention module method in definition 11 are used to establish a feature alignment and fusion module (FAFM), denoted as f2. The grouping feature fusion module f2 is composed of two deformation convolutions in definition 9 and one content-guided attention module in definition 11, denoted as C1, C2, and C3, respectively. The combination method of the traditional feature alignment and fusion module in definition 18 and the YOLOv8 backbone network is used to combine C1, C2, and C3 with the backbone network in the classical YOLOv8 network in definition 4, to obtain a new backbone network. Step 2.4: Constructing a distance matching probability fusion algorithm The classical CNN feature extraction method in definition 3 is used to process any SAR image and the paired optical image in the training set Train_QXS-SAROPT obtained in step 1 using a feature extraction subnetwork, to obtain SAR image features denoted as F1 and paired optical image features denoted as F2, respectively. Adopt the D-IoU calculation formula in the standard distance matching probability fusion method of definition 13 D-IoU calculation is performed on F1 and F2 features to obtain a value matrix; Subsequently, the Hungarian algorithm in definition 19 is used for optimal allocation. The Bayes fusion calculation formula in the standard distance matching probability fusion method defined in 13 is adopted The posterior probability of the target is calculated. At this point, the construction of the perception-aligned multi-modal fusion network is completed. Step 3, establishing a ship detection model On the multi-modal fusion network completed in step 2, the training set Train_QXS-SAROPT and the validation set Val_QXS-SAROPT obtained in step 1 are used as input, and the classical Adam algorithm in definition 10 is used for training. After training, a ship detection model is obtained, denoted as MHFNet YOLOv8. Step 4, testing the ship detection model The test set Test_QXS-SAROPT obtained in step 1 is tested on the ship detection model MHFNet YOLOv8 obtained in step 3 by using the standard detection network test method in definition 14 to obtain the test result of the test set on the ship detection model, denoted as Result. Step 5, evaluating the ship detection model The test result Result of the ship detection model obtained in step 4 is taken as input, and the precision, recall, average precision and F1-score are calculated by using the standard evaluation index calculation method in definition 15, denoted as P, R, mAP and F1 respectively. Thus, the whole method ends.

Images