Unmanned aerial vehicle view angle marine ship target detection method based on multi-modal data fusion

Abstract

The invention discloses an unmanned aerial vehicle view angle marine ship target detection method based on multi-modal data fusion, and belongs to the field of marine target detection, and the method comprises the steps: respectively inputting a collected visible light image and an infrared image into a multi-modal feature extraction network, and extracting shallow features; a Transform block is utilized to extract deep layer features from the shallow layer features; the method comprises the following steps of: aggregating output of each block by adopting a multi-head self-attention mechanism to obtain multi-directional multi-scale information, and refining compact features by using a feedforward network to obtain intermediate features; carrying out feature fusion by adopting a graph attention fusion block so as to aggregate local information, and obtaining fusion features; reconstructing the fusion features to obtain a fusion image; and an improved YOLO target detection algorithm is adopted to carry out marine ship target detection. According to the method, data of various modes such as visible light images and infrared imaging are fused, the advantages of the modes are fully utilized, the defects of single-mode data are overcome, and the precision, the real-time performance and the reliability of marine ship target detection are improved.

Description

Technical Field

[0001] This invention belongs to the field of marine target detection technology, specifically relating to a method for detecting marine vessels from the perspective of an unmanned aerial vehicle (UAV) based on multimodal data fusion. Background Technology

[0002] With the rapid development of the global economy and the continuous advancement of marine resource development, the activities of ships at sea are becoming increasingly frequent. Real-time monitoring of ships at sea is of vital importance for ensuring maritime traffic safety, protecting the marine ecological environment, and combating maritime crimes. In recent years, the rapid development of drone technology has provided an efficient, flexible, and relatively low-cost solution for target detection of ships at sea. Drones can quickly reach designated sea areas and conduct real-time aerial monitoring of ships, obtaining information such as their position, course, and speed, providing strong support for maritime traffic management, marine resource protection, and maritime law enforcement.

[0003] However, maritime vessel target detection faces numerous challenges. On one hand, the complex and ever-changing marine environment, with factors such as waves, sea fog, and lighting conditions interfering with images and video data acquired by drones, results in incomplete or unclear feature information about vessel targets, increasing the difficulty of target detection. On the other hand, maritime vessels vary in size, shape, and color, and their distribution on the sea surface is relatively sparse. Some vessels may be at a distance or obscured, making it difficult for drones to accurately identify and locate target vessels. Furthermore, the dynamic nature of maritime vessels, with unpredictable changes in their speed and course, places higher demands on the real-time performance and accuracy of target detection.

[0004] Currently, the detection of ships at sea from the perspective of unmanned aerial vehicles (UAVs) mainly relies on single-modal data, such as visible light images, radar signals, or infrared imaging. While these single-modal data have certain advantages under specific conditions, their limitations are also quite apparent in complex marine environments. To overcome the limitations of single-modal data, multimodal data fusion technology has gradually gained attention. Multimodal data fusion refers to the comprehensive processing and analysis of data from different sensors or different physical modalities to obtain more comprehensive and accurate target information. In the field of multi-sensor image fusion, existing methods mainly include traditional methods and deep learning-based methods, each with its own shortcomings.

[0005] Traditional methods utilize various image processing techniques to fuse visible light and infrared images across different domains, requiring manual design of image transformations and complex fusion rules. However, these rules often result in an imbalance of information between the two modalities in the fused image, making it difficult to fully integrate the advantages of both images and leading to poor fusion results.

[0006] Deep learning-based methods lack consideration for task-specific features. Due to the lack of real-world labeled data for infrared and visible light image fusion, most studies use natural image data to train neural networks, failing to adequately consider the specific characteristics of this task. This results in models that are insufficiently targeted when processing infrared and visible light image fusion. Furthermore, deep learning-based methods struggle to balance global and local features, primarily relying on convolution to extract local features. This leads to a small receptive field, limiting their ability to preserve the global contextual information of the source images and failing to capture overall image features and relationships between different regions effectively. Additionally, deep learning-based methods suffer from source image feature loss. Some techniques stitch the source images together before inputting them into a single-path network, potentially causing the loss of unique features and affecting the quality and information integrity of the fused image. Existing deep learning-based methods generally struggle to simultaneously and effectively extract high-frequency modal information and local symmetry information and retain this information in the fused image, resulting in defects in detail and structural representation.

[0007] In the field of maritime target detection, existing methods for detecting ships at sea have certain limitations, as follows:

[0008] 1. Poor performance in detecting small targets: The SeaDroneSee dataset contains a large number of small targets. For example, in a sea scene, the size of a small target may be less than 32 pixels, and existing methods struggle to effectively detect these small targets. Taking the SeaDroneSee target detection track as an example, the current best-performing model has an average accuracy of only 36%, which is significantly lower than the best model's mAP of over 60% in the common COCO benchmark.

[0009] 2. Significantly affected by sea surface interference: The presence of waves and other interference factors on the sea surface makes target detection tasks more difficult. Most existing methods are designed for natural scene images, and when directly applied to maritime drone capture scenarios, they cannot effectively cope with these interferences, leading to a decline in target detection performance.

[0010] In summary, existing maritime vessel target detection technologies suffer from numerous problems and shortcomings in complex marine environments, and existing multimodal data fusion methods are also insufficient to meet the high accuracy, real-time performance, and reliability requirements of maritime vessel target detection. Therefore, a UAV-based maritime vessel target detection method based on multimodal data fusion is needed. This method should fully utilize the advantages of each modality of data, overcome the limitations of single-modal data, improve the performance of maritime vessel target detection, and provide strong technical support for maritime traffic safety management and marine resource protection. Summary of the Invention

[0011] To address the shortcomings of existing maritime vessel target detection technologies in complex marine environments, such as low detection accuracy, poor real-time performance, and limitations of single-modal data, this invention proposes a UAV-based method for maritime vessel target detection based on multimodal data fusion. By fusing data from multiple modalities, including visible light images and infrared imaging, the method fully leverages the advantages of each modality, overcomes the deficiencies of single-modal data, and improves the accuracy, real-time performance, and reliability of maritime vessel target detection.

[0012] The technical solution adopted by this invention to solve the technical problem is as follows:

[0013] This invention provides a method for detecting maritime vessels from a UAV perspective based on multimodal data fusion, comprising the following steps:

[0014] S1: Input the acquired visible light image and infrared image into the multimodal feature extraction network to extract shallow features;

[0015] S2: Use Transformer blocks to extract deep features from shallow features;

[0016] S3: First, a multi-head self-attention mechanism is used to aggregate the output of each Transformer block to obtain multi-directional, multi-scale information. Then, a feedforward network is used to refine compact features and obtain intermediate features.

[0017] S4: Use graph attention fusion blocks to perform feature fusion to aggregate local information and obtain fused features;

[0018] S5: Reconstruct the fusion features to obtain the fused image;

[0019] S6: Use the improved YOLO target detection algorithm for maritime vessel target detection.

[0020] Furthermore, the extraction process of the shallow features is expressed as follows:

[0021]

[0022] in, This represents the features extracted from the infrared image at layer i-1. H represents the features extracted from the visible light image at layer i-1. SFE (·) represents the convolution operation. and These represent the shallow features extracted from visible light source images and infrared source images, respectively.

[0023] Furthermore, in step S2, the shallow features are first... Divide the area into D×D local blocks through shape manipulation, and reshape it into... The local window features are used, and a multi-head self-attention mechanism is applied to each Transformer block; for the local window features Using three weight matrices Use the projection key K, query Q, and value V:

[0024] {K,Q,V}={F At W K ,F At W Q ,F At W V} (2)

[0025] First, calculate the dot product between query Q and each key K, then normalize using the Softmax function to generate attention scores:

[0026]

[0027] Where Atten(K,Q,V) represents the attention score, d k The key dimension is represented by P, the learnable parameter associated with position encoding is represented by V, the value matrix is ​​represented by T, the matrix transpose operator is represented by k, and the key dimension is represented by k.

[0028] Furthermore, the feedforward network is composed of a multilayer perceptron containing Gaussian error linear units; both the multi-head self-attention mechanism and the multilayer perceptron employ layer normalization processing.

[0029]

[0030] in, Y represents the intermediate features after multi-head self-attention and residual connections, which serve as the input to the subsequent feedforward network. LN represents layer normalization, MHSA represents multi-head self-attention mechanism, FFN represents feedforward network, and K, Q, and V represent the key matrix, query matrix, and value matrix, respectively.

[0031] Furthermore, in step S4, the intermediate feature F In Block features P are obtained through a convolutional layer with stride s; the block features P are transformed into a matrix form of two-dimensional features M through a permutation operation; a graph is constructed based on the Euclidean distances of the row vectors of the two-dimensional feature matrix, i.e., the node features; the distances between all node pairs are calculated, and the adjacency matrix A is generated using the K-nearest neighbor method. adj The mathematical representation of the graph is as follows: Reshape the two-dimensional feature M into a node set M l ; Using a graph attention network to obtain a new node set M' l .

[0032] Furthermore, in step S4, a learnable shared linear transformation weight matrix W is introduced for each node, and a shared self-attention mechanism is used to calculate the feature vector of node j. The feature vector of node i Influence coefficient:

[0033]

[0034] Where 'a' represents the number of learnable parameters. A single-layer feedforward network, where LR represents the LeakyReLU activation function; the influence coefficients are normalized using the Softmax function:

[0035]

[0036] Where, N i The set of neighboring nodes of node i represents the range of neighboring nodes considered when calculating attention. This represents the feature vector of node l.

[0037] Furthermore, in step S4, the output features are generated by aggregating neighborhood features using a nonlinear function σ:

[0038]

[0039] in, N represents the output feature vector of node i after processing by the multi-head self-attention mechanism. i The set of neighboring nodes of node i represents the range of adjacent nodes considered when aggregating features;

[0040] A multi-head self-attention mechanism is used to enhance feature extraction capabilities by concatenating the output features of Z independent attention heads and then averaging them.

[0041]

[0042] in, W represents the normalized coefficient of the z-th attention head. z This represents the corresponding weight matrix; finally, the output feature vector of node i after processing by the multi-head self-attention mechanism is... Reconstruct the feature map into a 3D feature map, and obtain the output feature F through deconvolution transformation. Out .

[0043] Furthermore, in step S5, an image reconstruction is achieved using a loss function that effectively preserves complementary information from the source image. The loss function is defined as follows:

[0044]

[0045] in, Indicates SSIM loss. Let I represent the total variation loss function, λ represent the hyperparameter controlling the relative importance, and I represent the total variation loss function. f and I s Let the output image and source image be represented respectively; then the SSIM loss calculation formula is:

[0046]

[0047] The formula for calculating TV loss to preserve image gradient information is:

[0048] T(m,n)=I f (m,n)-I s (m,n) (12)

[0049]

[0050] Where T represents the difference between the source image and the fused image, ||2| represents the l2 norm, and (m,n) represent the horizontal and vertical coordinates of the image pixels, respectively.

[0051] Furthermore, in step S6, the improved YOLO object detection algorithm introduces the SimAM attention module and adds an additional prediction head to the neck and head networks, while combining test-time enhancement and weighted box fusion. First, the ELAN backbone network of YOLOv7 is used to extract feature maps; then, the SimAM attention module is used to assign 3D attention weights to the feature maps to highlight key target features in the shallow network and suppress irrelevant information; finally, test-time enhancement is used to transform and flip the image at different scales, and weighted box fusion is used to fuse the prediction results of multiple models.

[0052] This invention provides a UAV-based system for detecting maritime vessels from a multimodal data fusion perspective. The system is implemented using a multimodal data fusion framework and includes:

[0053] Data acquisition module: used to collect multimodal data; it uses the optical camera and infrared thermal imager mounted on the UAV to collect visible light and infrared images of ships at sea, respectively.

[0054] Data preprocessing module: used to preprocess the acquired multimodal data, including data correction, noise removal and data alignment;

[0055] Feature extraction and fusion module: used to extract feature information from each modality of data separately, and to fuse the features of different modalities of data through a fusion algorithm;

[0056] Target detection module: used to detect ships at sea based on the fused multimodal feature information and target detection algorithms;

[0057] Results optimization module: Used to optimize detection results, including target tracking, result correction and false detection elimination.

[0058] The beneficial effects of this invention are:

[0059] 1. This invention leverages the advantages of visible light and infrared images through multimodal data fusion, overcoming the limitations of single-modal data. It combines convolutional layers, Transformer blocks, and graph attention mechanisms to form a unique hybrid architecture: convolutional layers effectively extract shallow local semantic details, Transformer blocks capture the relationship between local and global features, and the graph attention fusion block uses a global self-similarity method to learn useful information, fusing complementary information from multi-scale receptive fields. This significantly improves the accuracy of ship target detection at sea, enabling accurate identification and location of ship targets of various sizes, shapes, and colors in complex marine environments, effectively solving the problem of poor performance in detecting small targets in existing methods.

[0060] 2. This invention dynamically adjusts the weights and fusion strategies of each modality of data based on factors such as the size, distance, and environmental conditions of the target vessel at sea. This dynamic adjustment mechanism can flexibly optimize detection performance according to specific scenarios. The introduction of this dynamic adjustment mechanism enables the invention to automatically adjust the target detection strategy according to different situations, reduce the impact of sea surface interference, and further improve the adaptability and flexibility of target detection. Especially in complex and ever-changing marine environments, it can significantly reduce false detections and false negatives, thereby improving target detection performance.

[0061] 3. This invention improves the reliability of maritime vessel target detection through multiple steps such as data preprocessing, feature extraction and fusion, target detection and result optimization. It can effectively reduce false detections and missed detections, and improve the stability and accuracy of target detection.

[0062] 4. By integrating the advantages of multiple modal data and combining deep learning and dynamic fusion strategies, this invention can better adapt to the complex and ever-changing marine environment and achieve efficient and accurate ship target detection. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of a multimodal data fusion framework.

[0064] Figure 2 This is a flowchart of the multimodal fusion feature information extraction and fusion process.

[0065] Figure 3 This is a structural diagram of the Transformer block.

[0066] Figure 4 The diagram shows the structure of the attention fusion block.

[0067] Figure 5 This is a structural diagram of the SimAM attention module with full 3D weights. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to the accompanying drawings.

[0069] In a first aspect, the present invention provides a UAV-based system for detecting maritime vessels from the perspective of multimodal data fusion.

[0070] like Figure 1 As shown, this invention provides a UAV-based system for detecting maritime vessels from a multimodal data fusion perspective, specifically employing a multimodal data fusion framework. This framework mainly includes: a data acquisition module, a data preprocessing module, a feature extraction and fusion module, a target detection module, and a result optimization module. The functions and roles of each module are as follows:

[0071] Data acquisition module: Primarily used for acquiring multimodal data. Specifically, it utilizes the optical camera and infrared thermal imager mounted on the UAV to acquire visible light and infrared images of ships at sea, respectively. The optical camera acquires visible light images of the ships, while the infrared thermal imager acquires infrared images.

[0072] The data preprocessing module is mainly used to preprocess the acquired multimodal data, including data correction, noise removal, and data alignment. These preprocessing methods can all be implemented using existing technologies. Specifically, visible light images are enhanced in quality through histogram equalization and filtering, while infrared images are improved in terms of target detectability through background suppression and target enhancement.

[0073] Feature extraction and fusion module: This module is mainly used to extract feature information from each modality of data separately, and then fuse the features of different modalities through a fusion algorithm. First, shallow local features of infrared and visible light images are extracted through two convolutional layers. Then, a Transformer block is used to further capture multi-scale global and local feature relationships. Specifically, the fusion algorithm employs a weighted fusion method, assigning different weights based on the characteristics and importance of each modality of data to fuse the extracted features.

[0074] Target detection module: Primarily used for detecting maritime vessels based on fused multimodal feature information using target detection algorithms. Specifically, the target detection algorithm can employ an improved YOLO algorithm, combined with multimodal fusion features, to quickly and accurately detect the target's location and category.

[0075] The result optimization module is mainly used to optimize the detection results, including target tracking, result correction, and false detection elimination. In this invention, the Kalman filter algorithm is primarily used to track the detected targets, improving the continuity and stability of target detection. Simultaneously, this invention also utilizes complementary information from multimodal data to correct the detection results, reducing false detections and missed detections.

[0076] This invention proposes a comprehensive multimodal data fusion framework capable of simultaneously processing data from multiple modalities, including visible light and infrared images. This framework optimizes the entire process from data acquisition to target recognition through multiple modules, including data preprocessing, feature extraction and fusion, target detection, and result optimization. Furthermore, this multimodal data fusion framework not only integrates data from various modalities but also dynamically adjusts the weights and fusion strategies of each modality, dynamically optimizing detection performance based on factors such as target size, distance, and environmental conditions, significantly improving the adaptability and reliability of the detection.

[0077] Secondly, this invention provides a method for detecting maritime vessels from a UAV perspective based on multimodal data fusion. This method is mainly implemented through the UAV-based maritime vessel target detection system based on multimodal data fusion provided in the first aspect.

[0078] The present invention provides a method for detecting maritime vessels from the perspective of an unmanned aerial vehicle (UAV) based on multimodal data fusion, the specific implementation process of which is as follows:

[0079] Step 1: Multimodal data acquisition and preprocessing;

[0080] S1.1: Collect multimodal data through the data acquisition module. Specifically, the visible light images of ships at sea can be collected using the optical camera mounted on the UAV, while the infrared thermal imager mounted on the UAV can collect infrared images of ships at sea.

[0081] S1.2: The acquired multimodal data is preprocessed using the data preprocessing module. Visible light images are enhanced in quality through histogram equalization and filtering; while infrared images are improved in detectability through background suppression and target enhancement.

[0082] Step 2: Multimodal feature extraction and fusion;

[0083] This invention proposes a feature extraction and fusion algorithm based on deep learning. The algorithm constructs a multimodal feature extraction network to extract feature information from visible light and infrared images respectively, and then fuses the features from different modalities.

[0084] Feature Extraction: First, shallow local features from infrared and visible light images are extracted using two convolutional layers. Then, Transformer blocks (the basic unit of the Transformer architecture, containing multi-head self-attention mechanisms and feedforward networks, typically used in conjunction with residual connections and layer normalization) are used to further capture multi-scale global and local feature relationships. The Transformer block divides the input features into local blocks, learns long-range dependencies across blocks through multi-head self-attention mechanisms and positional encoding, and refines the features using a feedforward network, ultimately outputting deep features containing complementary information in both spatial and channel dimensions. This design balances high-frequency details and global contextual information through hierarchical processing.

[0085] Multimodal feature fusion: This method fuses extracted visible light image features and infrared image features using a weighted fusion approach. Specifically, different weights are assigned based on the characteristics and importance of each modality, and the extracted features are weighted and summed to obtain the fused feature vector.

[0086] In this invention, a visible light source image is given. and infrared source images Where C, H, and W represent the number of channels, height, and width, respectively. Let be the set of real numbers, meaning that each element in the tensor belongs to the set of real numbers. The overall process consists of three steps: First, shallow and deep features are extracted using convolutional layers and Transformer blocks respectively; then, graph attention fusion blocks are used to effectively extract local and complementary information from the two modalities; finally, the fused image is recovered through a decoder. Customizing the loss function Reconstruction was carried out under the guidance of [unspecified entity].

[0087] The specific implementation process is as follows:

[0088] S2.1: Multimodal feature extraction;

[0089] S2.1.1: Shallow feature extraction;

[0090] First, two convolutional layers are used to extract images from the visible light source I. VI Extracting shallow features From infrared source image I IR Extracting shallow features The initial convolutional layer effectively extracts local semantic details. Furthermore, the non-local information contained in these features may be useful in subsequent processing. This process can be represented as:

[0091]

[0092] in, This represents the features extracted from the infrared image at layer i-1. H represents the features extracted from the visible light image at layer i-1. SFE (·) represents a convolution operation, extracting shallow features. and This is further used to extract deeper features from the source image. Shallow features are low-level visual features extracted from the bottom layer of the neural network (closest to the input layer), and typically contain basic information such as edges, colors, and textures.

[0093] S2.1.2: Deep feature extraction;

[0094] Deep feature extraction is performed using Transformer blocks. The structure of a Transformer block is as follows: Figure 3 As shown. Deep features extracted through Transformer blocks can effectively extract dense global features from the input. First, shallow features... Divide the area into D×D local blocks through shape manipulation, and reshape it into... Local window features (local window features refer to dividing the input feature map into several small regions and independently calculating the feature representation within each window. This method is often used in Transformers or attention mechanisms to reduce computational complexity), where This represents the total number of local blocks. Multi-head self-attention (MHSA) is applied to each Transformer block for local window features. Using three weight matrices Let's project the key K, query Q, and retrieve the value V. This process can be represented as:

[0095] {K,Q,V}={F At W K ,F At W Q ,F At W V} (2)

[0096] S2.1.3: First, calculate the dot product between query Q and each key K, then normalize using the Softmax function to generate attention scores. This process can be represented as:

[0097]

[0098] Where Atten(K,Q,V) represents the attention score, d kThe key dimension is represented by P, the learnable parameters related to position encoding are represented by V, the value matrix is ​​represented by (used to store the feature information that needs to be weighted and aggregated, the attention weights will eventually be applied to the value matrix V, and the output features are obtained by weighted summation of the value matrix V through the attention weights), T represents the matrix transpose operator, and k represents the key dimension.

[0099] S2.1.4: Layer normalization processing;

[0100] This invention first employs a multi-head self-attention mechanism (MHSA) to aggregate the output of each Transformer block to obtain multi-directional, multi-scale information. Then, a feedforward network (FFN) composed of a multilayer perceptron (MLP) containing Gaussian error linear units is used to further refine compact features, thereby obtaining the corresponding intermediate features. Both MHSA and MLP employ layer normalization. This process can be represented as:

[0101]

[0102] in, Y represents the intermediate features after multi-head self-attention and residual connections, which serve as the input to the subsequent feedforward network; Y represents the final output features of the current Transformer block, which can be used as the input to the next layer or for object detection tasks; LN represents layer normalization, which normalizes the output of MHSA to stabilize the training process.

[0103] S2.2: Feature fusion;

[0104] The intermediate feature maps obtained from the Transformer block are input into the attention fusion block (GAFB). For example... Figure 4 As shown, the specific processing flow of GAFB is as follows:

[0105] S2.2.1: First, intermediate features Block features are obtained by passing through a 3×3 convolutional layer with a stride of s. The block features P are transformed into a two-dimensional feature matrix [M] through a permutation operation. N×C Used for composition, among which A graph is constructed based on the Euclidean distance of the row vectors (node ​​features) of the two-dimensional feature matrix. The adjacency matrix is ​​generated by calculating the distance between all pairs of nodes and using the K-nearest neighbor method. The mathematical representation of a graph is as follows: Reshape the two-dimensional feature M into a node set. Where the feature vector of node i L represents the dimension of each node's feature (i.e., the length of the feature vector), i = 1, 2, ..., N.

[0106] S2.2.2: Subsequently, a graph attention network (GAT) is applied to obtain a new set of nodes. C represents the output feature vector of node i after processing by the multi-head self-attention mechanism. ' This indicates the dimension of the output features, which is the length of the feature vector of each node after GAT processing.

[0107] S2.2.3: To extract deeper features, a learnable shared linear transformation weight matrix is ​​introduced for each node. And it adopts a shared self-attention mechanism Calculate the feature vector of node j The feature vector of node i Influence coefficient:

[0108]

[0109] Where L0 represents the LeakyReLU activation function, and a represents the number of learnable parameters. A single-layer feedforward network, where LR represents the Leaky ReLU activation function. Influence coefficient e ij Feature vectors of node i only K first-order neighbor nodes calculate. This indicates that node j belongs to the set of first-order neighbor nodes of node i.

[0110] S2.2.4: Normalize the influence coefficients using the Softmax function. This process can be expressed as:

[0111]

[0112] Where, N i The set of neighboring nodes of node i represents the range of neighboring nodes considered when calculating attention. This represents the feature vector of node l.

[0113] S2.2.5: Output features are generated by aggregating neighborhood features using a nonlinear function σ. This process can be represented as:

[0114]

[0115] Where, N i The set of neighboring nodes of node i represents the range of adjacent nodes considered when aggregating features.

[0116] S2.2.6: A multi-head self-attention mechanism is employed to enhance feature extraction capabilities. The output features of Z independent attention heads are concatenated and then averaged. This process can be represented as:

[0117]

[0118] in, W represents the normalized coefficient of the z-th attention head. z This represents the corresponding weight matrix. The final output feature vector of node i after processing by the multi-head self-attention mechanism will... Remodeling The 3D feature map is transformed into output features through a transposed convolutional layer, i.e., deconvolution. After fusing features from various source images, the convolutional layer aggregates local and global information in the channel dimension to generate the fused feature embedding for the input decoder.

[0119] S2.3: Reconstruction;

[0120] After aggregating local information, the fused features are processed by a Transformer-block-based deep feature reconstruction unit. This unit contains a Transformer block and three 3×3 convolutional layers to refine the fused deep features and recover the fused shallow features. By reducing the number of channels through a series of convolutional layers, the fused image is finally reconstructed.

[0121] The performance of the fusion process largely depends on the design of the loss function. In this invention, the loss function needs to simultaneously capture the thermal radiation information of the infrared image and the diverse scene features of discrete targets in the visible light image. Therefore, this invention proposes a loss function that can effectively preserve complementary information of the source images. This loss function is based on structural similarity (SSIM) and gradient information between the source image and the fused image. SSIM ensures that the fused image maintains structural similarity with both the infrared and visible light images, while the gradient information is used to preserve high-frequency details and reduce noise in the fused image. The loss function is defined as follows:

[0122]

[0123] in, Indicates SSIM loss. Let λ represent the total variational (TV) loss function, and let λ represent the hyperparameter controlling relative importance (set to 10). 3 ), I f and I s Let represent the output image and the source image, respectively. The SSIM loss calculation formula is as follows:

[0124]

[0125] The TV loss used to preserve image gradient information is calculated using the following formula:

[0126] T(m,n)=I f (m,n)-I s (m,n) (12)

[0127]

[0128] Where T represents the difference between the source image and the fused image, ||2| represents the l2 norm, and (m,n) represent the horizontal and vertical coordinates of the image pixels, respectively.

[0129] This invention presents a deep learning-based feature extraction and fusion algorithm that extracts feature information from visible light and infrared images separately, and then fuses the features from different modalities using a weighted fusion method. This method can fully utilize the advantages of each modality and overcome the limitations of single-modality data. By constructing a multimodal feature extraction network and combining it with a dynamic fusion strategy, this invention can generate richer and more discriminative feature representations, significantly improving the accuracy and robustness of target detection.

[0130] Step 3: Target detection algorithm;

[0131] (1) Overall architecture;

[0132] The overall architecture of YOLOv7-sea is mainly based on improvements to YOLOv7. The architecture consists of three parts: First, it uses the ELAN backbone network of YOLOv7 for feature map extraction; second, to enhance the network's ability to learn key information, this invention introduces a SimAM attention module, such as... Figure 5 As shown, the SimAM attention module can highlight key target features in shallow networks and suppress irrelevant information, thereby improving the detection performance of the target detection algorithm for small targets. In view of the large number of tiny targets in the SeaDroneSee dataset, this invention adds an additional prediction head to the neck network and head network. Finally, techniques such as test-time enhancement (TTA) and weighted box fusion (WBF) are also used to improve detection accuracy and robustness.

[0133] This invention employs an improved YOLO target detection algorithm, combined with multimodal fusion features, to quickly and accurately detect the target location and category of vessels, meeting the high real-time requirements of maritime vessel target detection tasks. This target detection algorithm significantly improves detection accuracy while maintaining high detection speed. The improved YOLO target detection algorithm, through optimized network structure and fusion strategy, is better adapted to target detection tasks in complex marine environments, exhibiting higher accuracy and real-time performance, particularly when detecting small vessels or vessels at long distances.

[0134] In this invention, the improved YOLO target detection algorithm integrates the SimAM attention module. This module is based on neuroscience theory and can flexibly assign 3D attention weights to feature maps. Without increasing the original network parameters, it can effectively highlight key target features, weaken irrelevant information, and improve the network's detection performance for small targets.

[0135] This invention employs strategies such as data augmentation, test-time augmentation, and weighted bounding box fusion. Data augmentation enhances the model's robustness; Test-Time Alteracupuncture (TTA) improves detection performance during the inference phase by transforming and flipping images at different scales; and Web-Bounds fusion (WBF) improves the model's generalization ability by fusing predictions from multiple models, resulting in superior performance of the fused model on the validation set.

[0136] (2) Additional prediction head;

[0137] Because maritime drone datasets contain a large number of extremely small target instances, traditional three-head detection structures struggle to handle them effectively. Unlike YOLOv7's three prediction heads, this invention employs a four-prediction-head design to enhance the detection capabilities for small swimmers and aquatic equipment, particularly for small targets captured from high altitudes. This four-prediction-head structure makes detection more stable and mitigates the negative impact of drastic changes in target scale. The additional prediction heads are derived from low-level, high-resolution feature maps; while this increases computational and memory overhead, it significantly improves the performance of small target detection.

[0138] (3) SimAM attention module;

[0139] Images captured by drones have a wide field of view but are often affected by sea conditions, making it crucial to effectively focus on key areas. SimAM is a parameter-free attention mechanism that flexibly assigns 3D attention weights to feature maps, thereby enhancing the model's feature extraction capabilities. This SimAM attention module identifies important neurons by measuring the linear separability between neurons; its energy function is defined as:

[0140]

[0141] Where t and x i ω represents the target neuron and other neurons in the same channel, respectively. t and b t Decibel represents the weights and biases of the linear transformation, y represents the vertical coordinates of the bounding box in object detection, λ represents the hyperparameters, and M = HW represents the total number of neurons in a single channel. The minimum energy formula can be obtained analytically as follows:

[0142]

[0143] A smaller energy value indicates a stronger linear separability of neuron t from other neurons. This SimAM attention module is implemented based on the energy function, avoiding heuristic parameter tuning. Experiments show that introducing SimAM into YOLOv7 significantly improves the model's feature extraction capability without increasing network parameters.

[0144] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting maritime vessels from an UAV perspective based on multimodal data fusion, characterized in that, Includes the following steps: S1: Input the acquired visible light image and infrared image into the multimodal feature extraction network to extract shallow features; S2: Use Transformer blocks to extract deep features from shallow features; S3: First, a multi-head self-attention mechanism is used to aggregate the output of each Transformer block to obtain multi-directional, multi-scale information. Then, a feedforward network is used to refine compact features and obtain intermediate features. S4: Use graph attention fusion blocks to perform feature fusion to aggregate local information and obtain fused features; S5: Reconstruct the fusion features to obtain the fused image; S6: Use the improved YOLO target detection algorithm for maritime vessel target detection.

2. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, The extraction process of the shallow features is represented as follows: in, This represents the features extracted from the infrared image at layer i-1. H represents the features extracted from the visible light image at layer i-1. SFE (·) represents the convolution operation. and These represent the shallow features extracted from visible light source images and infrared source images, respectively.

3. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, In step S2, the shallow features are first... Divide the area into D×D local blocks through shape manipulation, and reshape it into... The local window features are used, and a multi-head self-attention mechanism is applied to each Transformer block; for the local window features Using three weight matrices Use the projection key K, query Q, and value V: {K,Q,V}={F At W K ,F At W Q ,F At W V } (2) First, calculate the dot product between query Q and each key K, then normalize using the Softmax function to generate attention scores: Where Atten(K,Q,V) represents the attention score, d k The key dimension is represented by P, the learnable parameter associated with position encoding is represented by V, the value matrix is ​​represented by T, the matrix transpose operator is represented by k, and the key dimension is represented by k.

4. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, The feedforward network consists of a multilayer perceptron containing Gaussian error linear units; both the multi-head self-attention mechanism and the multilayer perceptron employ layer normalization processing. in, Y represents the intermediate features after multi-head self-attention and residual connections, which serve as the input to the subsequent feedforward network. LN represents layer normalization, MHSA represents multi-head self-attention mechanism, FFN represents feedforward network, and K, Q, and V represent the key matrix, query matrix, and value matrix, respectively.

5. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, In step S4, the intermediate feature F In Block features P are obtained through a convolutional layer with stride s; the block features P are transformed into a matrix form of two-dimensional features M through a permutation operation; a graph is constructed based on the Euclidean distances of the row vectors of the two-dimensional feature matrix, i.e., the node features; the distances between all node pairs are calculated, and the adjacency matrix A is generated using the K-nearest neighbor method. adj The mathematical representation of the graph is as follows: Reshape the two-dimensional feature M into a node set M l ; Using a graph attention network to obtain a new node set M' l .

6. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 5, characterized in that, In step S4, a learnable shared linear transformation weight matrix W is introduced for each node, and a shared self-attention mechanism is used to calculate the feature vector of node j. The feature vector of node i Influence coefficient: Where 'a' represents the number of learnable parameters. A single-layer feedforward network, where LR represents the LeakyReLU activation function; the influence coefficients are normalized using the Softmax function: Where, N i The set of neighboring nodes of node i represents the range of neighboring nodes considered when calculating attention. This represents the feature vector of node l.

7. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 6, characterized in that, In step S4, the output features are generated by aggregating neighborhood features using the nonlinear function σ. in, N represents the output feature vector of node i after processing by the multi-head self-attention mechanism. i The set of neighboring nodes of node i represents the range of adjacent nodes considered when aggregating features; A multi-head self-attention mechanism is used to enhance feature extraction capabilities by concatenating the output features of Z independent attention heads and then averaging them. in, W represents the normalized coefficient of the z-th attention head. z This represents the corresponding weight matrix; finally, the output feature vector of node i after processing by the multi-head self-attention mechanism is... Reconstruct the feature map into a 3D feature map, and obtain the output feature F through deconvolution transformation. Out .

8. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, In step S5, an image reconstruction is achieved using a loss function that effectively preserves complementary information of the source image. The loss function is defined as follows: in, Indicates SSIM loss. Let I represent the total variation loss function, λ represent the hyperparameter controlling the relative importance, and I represent the total variation loss function. f and I s Let the output image and source image be represented respectively; then the SSIM loss calculation formula is: The formula for calculating TV loss to preserve image gradient information is: T(m,n)=I f (m,n)-I s (m,n) (12) Where T represents the difference between the source image and the fused image, ||2| represents the l2 norm, and (m,n) represent the horizontal and vertical coordinates of the image pixels, respectively.

9. The method for detecting maritime vessels from a UAV perspective based on multimodal data fusion according to claim 1, characterized in that, In step S6, the improved YOLO object detection algorithm introduces the SimAM attention module and adds an additional prediction head to the neck and head networks, while combining test-time enhancement and weighted bounding box fusion. First, the ELAN backbone network of YOLOv7 is used to extract feature maps. Then, the SimAM attention module is used to assign 3D attention weights to the feature maps to highlight key target features in the shallow network and suppress irrelevant information. Finally, test-time enhancement is used to transform and flip the image at different scales, and weighted bounding box fusion is used to fuse the prediction results of multiple models.

10. A UAV-based maritime vessel target detection system based on multimodal data fusion, characterized in that, This is achieved using a multimodal data fusion framework, including: Data acquisition module: used to collect multimodal data; it uses the optical camera and infrared thermal imager mounted on the UAV to collect visible light and infrared images of ships at sea, respectively. Data preprocessing module: used to preprocess the acquired multimodal data, including data correction, noise removal and data alignment; Feature extraction and fusion module: used to extract feature information from each modality of data separately, and to fuse the features of different modalities of data through a fusion algorithm; Target detection module: used to detect ships at sea based on the fused multimodal feature information and target detection algorithms; Results optimization module: Used to optimize detection results, including target tracking, result correction and false detection elimination.

Images