A method and device for detecting marine targets based on multimodal fusion

Abstract

The present invention relates to a method and device for detecting marine targets based on multimodal fusion, comprising the following steps: obtaining a visible light image and an infrared imaging image of the same marine scene, and synchronously inputting the visible light image and the infrared imaging image into a dual-stream backbone network constructed based on GhostNet; extracting features of the visible light image and the infrared imaging image in the third to fifth layers of the dual-stream backbone network, respectively, and inputting these features into a cross-modal attention feature fusion module for cross-modal feature fusion to obtain fused features; then inputting the fused features into a hybrid encoder composed of an AIFI module and a CCFF module for processing to obtain cross-scale feature interaction and fusion results; and inputting the cross-scale feature interaction and fusion results into a decoder after undergoing uncertainty minimum query selection to complete marine target detection. By fusing visible light and infrared modal data, the present invention can improve the detection success rate of small targets in complex marine conditions.

Description

Technical Field

[0001] The present invention relates to the technical field related to maritime target detection, and in particular to a maritime target detection method and device based on multimodal fusion. Background Art

[0002] In the detection of small targets at sea, mobile platforms such as drones have become the primary detection tool due to their flexibility and wide-area coverage. However, limited computing power and memory capacity of onboard equipment make lightweight network architecture design a challenge. While the fusion of a two-stream convolutional network and DETR offers excellent detection performance, its large number of parameters and high computational complexity make it difficult to directly deploy. Therefore, research on optimization methods for lightweight network architectures is of great significance.

[0003] Existing lightweighting techniques fall into three main categories: post-processing optimization methods, knowledge transfer methods, and efficient architecture design approaches. Post-processing optimization methods, exemplified by network pruning, compress pre-trained models by removing redundant connections or channels, but performance is limited by the upper bound of the original network. Knowledge distillation transfers knowledge from large networks to smaller ones through a "teacher-student" framework, but training is complex and relies on a high-quality teacher model. In contrast, efficient neural architecture design fundamentally reduces computational complexity by redesigning fundamental computing units, offering greater optimization potential.

[0004] In the area of ​​efficient architecture design, researchers have proposed various solutions. The MobileNet series replaces standard convolution with depthwise separable convolution to reduce computational complexity. ShuffleNet introduces channel shuffling, improving feature representation through cross-group information exchange, achieving comparable performance to AlexNet on the ImageNet classification task while maintaining lower computational overhead. To address feature map redundancy, GhostNet proposes a feature generation mechanism. Research has shown that feature maps output by standard convolutional layers contain a large number of similar features. The Ghost module generates "ghost" feature maps through inexpensive linear transformations, which are then concatenated with the original feature maps to form the complete output. Theoretical analysis shows that the Ghost module can reduce computational complexity to 1 / s of standard convolution, where s is the compression ratio. Experiments have demonstrated that GhostNet achieves less computational overhead than MobileNetV3 on the ImageNet classification task while maintaining similar accuracy.

[0005] The related problems existing in the prior art are as follows:

[0006] 1. Simple cross-modal fusion

[0007] Multispectral imaging technology integrates the complementary information of visible light and infrared cameras, providing a richer perception dimension for detection networks, thereby improving the system's reliability, perception capabilities, and environmental adaptability. However, traditional cross-modal fusion methods have significant limitations: pixel-level addition, while simple to implement, ignores the nonlinear relationships between different spectral channels; four-channel input methods, while preserving the original information, fail to effectively exploit intermodal correlation features; and shallow feature fusion struggles to fully extract the complementary characteristics between modalities.

[0008] The main problem with direct concatenation or weighted summation fusion schemes is that the RGB and thermal infrared modalities each carry unique feature representations, which contain both valuable complementary cues and modality-specific noise. Linear combination or cascade operations cannot effectively distinguish and utilize these features, resulting in insufficient extraction of shared features between modalities and unclear representation of modality-specific features. Therefore, more sophisticated fusion strategies are needed to fully exploit the synergistic effects of multispectral data.

[0009] 2. Single network architecture

[0010] Object detection methods based on convolutional neural networks (CNNs) have demonstrated their powerful feature representation capabilities in numerous studies. However, from a neural network perspective, the convolution operation of a CNN can be modeled as a non-fully connected graph structure: each spatial location in the feature map corresponds to a node, and the connections between nodes are limited to the local receptive field of the convolution kernel. While this local connectivity gives CNNs a good spatial inductive bias, it also limits their ability to model long-range dependencies.

[0011] In contrast, the self-attention mechanism in the Transformer architecture can be viewed as a fully connected graph structure, where each node can establish direct connections with all other nodes globally. This global modeling capability enables the Transformer to effectively capture long-range dependencies in images, providing a more powerful representation space for feature fusion of multispectral images. However, directly feeding high-resolution images into the encoder results in computational complexity that grows quadratically with the sequence length, which is often unacceptable in practice.

[0012] Therefore, the CNN+Transformer hybrid architecture provides an efficient solution: CNN is first used to extract local features and reduce spatial resolution, and then the compressed features are input into the Transformer for global relationship modeling. This layered processing strategy retains the advantages of CNN in local feature extraction while fully leveraging the global modeling capabilities of the Transformer, enabling more efficient mining of global contextual information in multispectral images while maintaining an acceptable computational cost.

[0013] 3. Reliance on manually designed components

[0014] The core task of object detection is to predict the bounding box coordinates and category label for each object of interest in an image. Traditional detection methods typically employ indirect collective prediction strategies: based on predefined anchors, region proposals, or window centers, they approximate object localization through proxy regression and classification tasks. The performance of such methods relies heavily on complex post-processing mechanisms: first, a dense anchor distribution or region proposal generation strategy must be designed; second, matching rules between object boxes and anchors must be established; and finally, post-processing steps such as non-maximum suppression (NMS) are required to remove duplicate predictions.

[0015] Detection frameworks based on prior knowledge have inherent design limitations: First, the anchor distribution strategy requires manual tuning based on the statistical characteristics of a specific dataset; second, the matching rules between object boxes and anchors often use heuristic criteria such as the Intersection over Union (IoU) threshold; and finally, post-processing steps such as NMS introduce additional hyperparameters. To overcome these limitations and build a truly end-to-end detection network, the detection process must be fundamentally restructured: abandoning the pre-defined spatial anchor mechanism, eliminating heuristic matching rules, and removing post-processing steps such as NMS.

[0016] 4. Limited computing resources of drone onboard equipment

[0017] For small-target detection missions at sea, drones, with their high maneuverability and wide-area coverage, are ideal platforms. However, their limited onboard computing resources place stringent demands on network lightweighting. Existing lightweighting technologies fall into three main categories: post-processing optimization, knowledge transfer, and efficient architecture design. Post-processing optimization, exemplified by network pruning, compresses the model by removing redundant connections, but its performance is limited by the original network. Knowledge distillation utilizes a "teacher-student" framework to transfer knowledge from large models, but training is complex and relies on a high-quality "teacher" model. Efficient architecture design, by fundamentally reducing computational complexity through the reconstruction of fundamental computing units, presents a promising approach for lightweighting. Summary of the Invention

[0018] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a method and device for detecting marine targets based on multimodal fusion.

[0019] In order to achieve the above object, the present invention adopts the following technical solutions:

[0020] Specifically, a method for detecting maritime targets based on multimodal fusion is proposed, which includes the following:

[0021] Obtain visible light images and infrared imaging images of the same scene at sea, and simultaneously input the visible light images and infrared imaging images into a dual-stream backbone network built based on GhostNet;

[0022] In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module;

[0023] The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results.

[0024] Among them, the AIFI module and CCFF module together form a hybrid encoder;

[0025] The cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection.

[0026] Furthermore, specifically, the cross-modal attention feature fusion module includes a differential enhancement module and a public selection module, and the differential enhancement module and the public selection module are arranged in parallel, wherein,

[0027] The process of the differential enhancement module is as follows: given the intermediate RGB convolution feature map and hot convolution feature maps , where C represents the number of channels, H represents the height, and W represents the width. First, the direct subtraction of the two modalities obtains the differential feature map , then, respectively Perform global average pooling to obtain And global maximum pooling to get ,in and Both are convolution feature maps. and After an hourglass-shaped two-layer shared convolutional layer Get and , secondly, and Add element by element and pass the activation function Sigmoid to get the channel dimension attention map , then multiply the attention map to each input feature map, and then add the residuals to get the enhanced feature map of each modality. Finally, the two enhanced feature maps are summed to get the output of the differential enhancement module ;

[0028] The process of the public selection module is as follows: given an RGB convolution feature map and hot convolution feature maps , first obtain the common modal feature map by direct summation , secondly, After global average pooling, , Then pass through two fully connected networks and get and ; Then the public selection module calculates the RGB feature attention map and thermal feature attention map ; Then, the attention map of RGB features and the attention map of thermal features are multiplied by their inputs respectively; Finally, the output results of the previous step are added together to obtain the output of the common selection module .

[0029] Furthermore, specifically, the CCFF module realizes the progressive fusion of multi-scale features by introducing multiple cascaded fusion blocks. Each fusion block includes two key components: the first key component is two 1×1 convolution layers for feature transformation of the channel dimension; the second key component is N RepBlocks composed of RepConv for feature fusion. When given the feature input of two adjacent scales, the CCFF module first adjusts the channel dimension through the 1×1 convolution of the first key component, then extracts features through the Rep Block of the second key component, and finally realizes feature fusion by element-by-element addition.

[0030] Furthermore, the specific working principle of the uncertainty minimum query selection includes:

[0031] The feature uncertainty U is defined as the difference measure between the prediction distributions of the two tasks of positioning (P) and classification (C). In order to effectively reduce the uncertainty of the query feature, this uncertainty indicator is integrated into the loss function and minimized through a gradient-based optimization method.

[0032] Furthermore, specifically, the cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection, including:

[0033] The cross-scale feature interaction and fusion results are used to select a fixed number of encoder features as the decoder's initial object query through uncertainty minimization query. Finally, the decoder with auxiliary prediction head iteratively optimizes the object query to generate categories and boxes.

[0034] The present invention also proposes a device for detecting marine targets based on multimodal fusion, comprising the following:

[0035] A data acquisition module is used to acquire visible light images and infrared imaging images of the same scene at sea, and synchronously input the visible light images and infrared imaging images into a processing module;

[0036] The processing module is constructed by a dual-stream backbone network based on GhostNet;

[0037] The operation process of the processing module is as follows:

[0038] In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module;

[0039] The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results.

[0040] Among them, the AIFI module and CCFF module together form a hybrid encoder;

[0041] The cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection.

[0042] The beneficial effects of the present invention are:

[0043] The present invention proposes a method and device for detecting marine targets based on multimodal fusion, which has the following beneficial effects:

[0044] 1. Strengthening maritime safety and security capabilities: By integrating visible light and infrared modal data, the detection success rate of small targets (such as floating objects and small ships) can be improved in complex sea conditions (such as haze, nighttime, and high waves), reducing collision accidents or illegal intrusions caused by missed targets, and ensuring maritime navigation safety.

[0045] 2. Improve search and rescue efficiency: In maritime search and rescue missions, this method can quickly locate people who have fallen into the water, lost vessels, or distressed equipment, shortening the golden rescue time, reducing casualties and property losses, and enhancing social emergency response capabilities.

[0046] 3. Support marine environmental protection: timely detection of marine pollutants such as oil and plastic waste, help protect the marine ecosystem, avoid environmental disasters caused by missed detection, and promote the realization of sustainable development goals.

[0047] 4. Reduce shipping costs: Reduce accidents such as ship collisions and equipment damage caused by target detection failures, directly reducing insurance claims and repair costs.

[0048] 5. Promote the development of marine resources: Provide high-precision detection support for industries such as offshore wind power, seabed mineral development, and deep-sea exploration, reduce engineering risks, shorten development cycles, and accelerate the development of the blue economy.

[0049] 6. Breaking through the limitations of a single modality: Combining the high resolution of visible light and infrared thermal radiation perception, and then deeply fusing cross-modal features, effectively addressing the performance shortcomings of a single sensor in complex sea conditions.

[0050] 7. Achieve real-time and lightweight deployment: Abandoning the traditional manually designed network paradigm, the end-to-end network based on efficient architecture design can deploy lightweight inference engines in edge computing platforms such as drones and buoys to achieve low-latency real-time target detection while meeting the strict constraints of edge devices on computing resources and energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other features of the present disclosure will become more apparent through a detailed description of the embodiments shown in conjunction with the accompanying drawings. The same reference numerals in the drawings of the present disclosure represent the same or similar elements. Obviously, the drawings described below are only some embodiments of the present disclosure. It is possible for a person skilled in the art to derive other drawings based on these drawings without inventive effort. In the drawings:

[0052] Figure 1 The figure shows the structure of the cross-modal attention feature fusion module in the present invention;

[0053] Figure 2 FIG2 is a schematic diagram showing the structure of a dual-stream backbone network based on GhostNet involved in the method for detecting small targets at sea based on cross-modal feature fusion of the present invention;

[0054] Figure 3 Shown is a flow chart of the marine target detection method based on multimodal fusion of the present invention. DETAILED DESCRIPTION

[0055] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless there is a conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0056] Traditional neural networks typically use visible light images of targets as input for training and detection. However, this approach has not yet achieved satisfactory results in small target detection tasks at sea. Infrared images are generated by capturing the thermal radiation emitted by objects. Compared to visible light images, they are more stable in adverse environments (such as haze, rain, and snow). Therefore, there is a clear complementarity between infrared and visible light images, making multispectral image detection an effective solution. Unlike traditional detection networks that only support single-modality input, multispectral detection requires simultaneous processing of data from both infrared and visible light modalities, making the dual-stream convolutional neural network architecture an ideal backbone network.

[0057] However, existing multispectral detection methods typically perform only simple pixel-level addition of visible and infrared images, or perform shallow fusion of features from the two modalities. It is crucial for the detection network to efficiently fuse features from both modalities. A convolutional neural network with a two-stream architecture constructs two independent feature extractors, encoding the visible and infrared images into a common feature space. This common feature space is divided into three subspaces: a color-specific feature space, a shared feature space, and a thermal radiation-specific feature space. The network utilizes a feature enhancement mechanism to preserve and enhance the unique characteristics of color and thermal radiation. Specifically, the color-specific feature space primarily captures color and texture information in visible light images, while the thermal-specific feature space focuses on thermal radiation features in infrared images. Furthermore, features in the shared feature space are selected to reduce the influence of redundant features. In this way, the network effectively utilizes both shared and specific information from each sample, thereby improving detection performance.

[0058] The development of computer vision modeling systems demonstrates a clear intergenerational evolution. Convolutional Neural Networks (CNNs), with their unique spatial inductive bias, have gradually established their dominance in vision tasks. The evolution of CNN architectures has followed three optimization paths: first, the continuous expansion of network depth and width, such as VGGNet's deep network construction by stacking standard convolutional layers; second, innovations in connection mechanisms, such as ResNet's introduction of residual connections to alleviate the vanishing gradient problem; and third, improvements to convolutional operations, including dilated convolutions to increase receptive field and deformable convolutions to enhance geometric deformation modeling capabilities. These architectural innovations have enabled CNNs to demonstrate powerful feature representation capabilities in tasks such as image classification, object detection, and semantic segmentation. In natural language processing, the Transformer architecture, with its global self-attention mechanism, has achieved breakthroughs in sequence modeling tasks. This mechanism effectively addresses the gradient decay problem of traditional recurrent neural networks (RNNs) by establishing fully connected, long-range dependencies between input elements. Inspired by this, a wave of research on the Vision Transformer (ViT) has emerged in the computer vision community. ViT uses an image partitioning strategy to divide the input image into 16×16 sequential tiles and obtain token embeddings through linear mapping. However, when directly processing high-resolution images, the computational complexity of the self-attention mechanism is proportional to the square of the sequence length, resulting in a sharp increase in computational effort.

[0059] To address this challenge, researchers have proposed two improved paradigms: one is the hierarchical attention mechanism, such as the SwinTransformer, which achieves hierarchical aggregation of local attention through a shifted window strategy, reducing computational complexity to a linear level while maintaining global modeling capabilities; the other is the CNN-Transformer hybrid architecture, represented by the Detection Transformer (DETR), which innovatively uses a CNN backbone network (such as ResNet) to extract multi-scale features and then uses the Transformer encoder-decoder for relationship modeling. DETR directly predicts target sets through learnable object queries, abandoning the manually designed components such as anchor box generation and non-maximum suppression (NMS) in traditional detection methods, achieving true end-to-end object detection.

[0060] In the task of detecting small targets at sea, the fusion architecture of a two-stream convolutional network and DETR effectively combines the local feature extraction capabilities of CNNs with the global modeling advantages of Transformers. This architecture extracts complementary features from visible and infrared dual-modal branches and leverages the Transformer's self-attention mechanism for deep feature fusion. This significantly improves detection performance for targets with low signal-to-noise ratios, while achieving true end-to-end object detection.

[0061] In terms of network architecture, the advantages of CNN and Transformer are combined to construct a hybrid model with stronger feature representation capabilities. At the feature fusion level, the coordinated processing of visible light and infrared dual-modal data achieves robust perception of complex sea conditions. A lightweight network architecture is designed for UAV platforms, ensuring performance while meeting computing resource constraints. Furthermore, the end-to-end network design simplifies the traditional detection process and improves the system's real-time performance. This method for detecting small targets at sea effectively addresses the unique challenges of the marine environment.

[0062] Based on the above considerations, Example 1, referring to Figure 2 as well as Figure 3 The present invention proposes a method for detecting marine targets based on multimodal fusion, which includes the following steps:

[0063] Obtain visible light images and infrared imaging images of the same scene at sea, and simultaneously input the visible light images and infrared imaging images into a dual-stream backbone network built based on GhostNet;

[0064] In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module;

[0065] The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results.

[0066] Among them, the AIFI module and CCFF module together form a hybrid encoder;

[0067] The cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection.

[0068] Inspired by the way a differential amplifier amplifies differential-mode signals and suppresses common-mode signals, the Cross-Modality Attentive Feature Fusion (CMAFF) module consists of two parts: a differential enhancement module and a common selection module. These two modules are arranged in parallel, making the CMAFF a dual-input, dual-output module.

[0069] Reference Figure 2 As a preferred embodiment of the present invention, specifically, the cross-modal attention feature fusion module includes a differential enhancement module and a public selection module, and the differential enhancement module and the public selection module are arranged in parallel, wherein,

[0070] The differential enhancement module is inspired by the differential mode signal amplified in the differential amplifier circuit, and aims to enhance thermal or visible features through differential modal features. The main idea of ​​the differential enhancement module is to use the channel attention weighting mechanism to enhance features from different modalities. The working process of the differential enhancement module is: given the intermediate RGB convolution feature map and hot convolution feature maps , firstly, the direct subtraction of the two modes obtains the differential feature map , then, respectively Perform global average pooling to obtain And global maximum pooling to get , and After an hourglass-shaped two-layer shared convolutional layer Get and , secondly, and Add element by element and pass the activation function Sigmoid to get the channel dimension attention map , then multiply the attention map to each input feature map, and then add the residuals to get the enhanced feature map of each modality. Finally, the two enhanced feature maps are summed to get the output of the differential enhancement module ;

[0071] The specific formula is as follows:

[0072] (1);

[0073] (2);

[0074] (3);

[0075] (4);

[0076] (5);

[0077] (6);

[0078] (7);

[0079] in represents the global average pooling function, represents the global maximum pooling function, represents the Sigmoid activation function, Represents element-by-element multiplication, where i is the row number of the pixel and j is the column number of the pixel.

[0080] The public selection module adaptively selects bimodal channel features based on the common modal features and recombines them into new features. The public selection module can dynamically select features through SoftMax attention, which is guided by the information in RGB and hot branches. The working process of the public selection module is as follows: given an RGB convolution feature map and hot convolution feature maps , first obtain the common modal feature map by direct summation , secondly, After global average pooling, , Then pass through two fully connected networks and get and ; Then the public selection module calculates the RGB feature attention map and thermal feature attention map ; Then, the attention map of RGB features and the attention map of thermal features are multiplied by their inputs respectively; Finally, the output results of the previous step are added together to obtain the output of the common selection module .

[0081] The specific formula is as follows:

[0082] (8);

[0083] (9);

[0084] (10);

[0085] (11);

[0086] (12);

[0087] (13);

[0088] in represents the global average pooling function, represents the global average pooling function, Represents element-by-element multiplication, where n is the row number of the pixel and m is the column number of the pixel.

[0089] The dual-stream backbone network built based on GhostNet is an end-to-end architecture network with dual-stream GhostNet as the backbone network and based on Transformer;

[0090] As shown in Table 1, the main structure of GhostNet consists of multiple stacked Ghost bottleneck (G-bneck) modules, with the Ghost module serving as the core building block. The network's initial layer uses a standard convolutional layer with 16 kernels, followed by a cascade of multiple Ghost bottleneck modules with increasing channel counts. At the end of the network, a global average pooling layer and convolutional layers convert the feature maps into 1280-dimensional feature vectors for use in downstream tasks.

[0091] Table 1. GhostNet network structure

[0092]

[0093] The Ghost module uses standard convolution Input feature map Compress the number of channels to obtain , and then The features of each channel in the function are applied with a series of “cheap” linear operations to generate s ghost features, and finally these feature maps are concatenated to form a new output feature. The calculation of the Ghost module is expressed as:

[0094] (14);

[0095] (15);

[0096] (16);

[0097] in, , b represents the bias term, yes The qth channel feature map in, Indicates that the lth linear mapping is used to generate The lth ghost feature (Except for the linear mapping at the very end).

[0098] The Ghost bottleneck is a bottleneck structure composed of Ghost modules. Its structural design is similar to the basic residual block of ResNet, both containing convolutional layers and shortcut connections. It primarily consists of two cascaded Ghost modules: the first Ghost module acts as an expansion layer, increasing the channel dimension by increasing the output / input channel ratio; the second Ghost module is responsible for reducing the number of channels to match the shortcut path. For stride = 1, this structure can be applied directly; for stride = 2, a shortcut path is implemented through a downsampling layer, and a depthwise separable convolution with stride = 2 is inserted between the two Ghost modules.

[0099] The Real-Time Detection Transformer (RT-DETR) based on DETR achieves end-to-end detection while reducing computational complexity. (1) An efficient hybrid encoder is designed to improve speed by decoupling intra-scale interactions and cross-scale fusion, thereby quickly processing multi-scale features. (2) A query selection with minimum uncertainty is proposed to provide high-quality initial queries to the decoder, thereby improving accuracy.

[0100] The hybrid encoder proposes two modules: Attention-based Intra-scale Feature Interaction (AIFI) and CNN-based Cross-scale Feature Fusion (CCFF). In the AIFI module, the Transformer-based self-attention operation is applied to high-level features with richer semantic concepts, capturing the connection between conceptual entities, which facilitates the positioning and recognition of objects by subsequent modules. However, due to the lack of semantic concepts and the risk of duplication and confusion in interaction with high-level features, the scale interaction of low-level features is unnecessary. Therefore, AIFI uses a single-scale Transformer encoder only in The scale interaction can be carried out on the network to further reduce the computation cost.

[0101] As a preferred embodiment of the present invention, specifically, an efficient cross-scale feature fusion architecture is designed in the CCFF module. By introducing multiple cascaded fusion blocks, progressive fusion of multi-scale features is achieved. Each fusion block includes two key components: the first key component is two 1×1 convolutional layers for feature transformation in the channel dimension; the second key component is N RepBlocks composed of RepConv for feature fusion. Given feature inputs of two adjacent scales, the CCFF module first adjusts the channel dimension through the 1×1 convolution of the first key component, then extracts features through the RepBlock of the second key component, and finally achieves feature fusion by element-by-element addition.

[0102] The computational expression of the hybrid encoder is:

[0103] (17);

[0104] (18);

[0105] (19);

[0106] in Restore the shape of the one-dimensional feature to Same shape.

[0107] As a preferred embodiment of the present invention, specifically, the working principle of the least uncertainty query selection includes:

[0108] The feature uncertainty U is defined as the difference measure between the prediction distributions of the two tasks of positioning (P) and classification (C). In order to effectively reduce the uncertainty of the query feature, this uncertainty indicator is integrated into the loss function and minimized through a gradient-based optimization method.

[0109] In this preferred embodiment, the query selection based on the DETR architecture model in the past resulted in a considerable uncertainty in the selected features, which led to suboptimal initialization of the decoder and hindered the performance of the detector. The uncertainty-minimizing query selection scheme explicitly constructs and optimizes epistemic uncertainty to model the joint latent variables of the encoder features, thereby providing high-quality queries for the encoder. Specifically, we consider the feature uncertainty Defined as positioning and classification A measure of the difference between the prediction distributions of two tasks. To effectively reduce the uncertainty of query features, this uncertainty indicator is integrated into the loss function and minimized using a gradient-based optimization method. The specific calculation is as follows:

[0110] (20);

[0111] (twenty one);

[0112] in and denote the predicted value and the true value respectively, and , and represent categories and bounding boxes respectively. Represents the encoder characteristics. , , denote the total loss, boundary loss and classification loss respectively.

[0113] As shown in the figure, the network uses a two-stream GhostNet as the backbone network. The hybrid encoder completes the fusion and interaction of bimodal and multi-scale features. Then, the uncertainty minimum query selects a fixed number of encoder features as the decoder's initial object query. Finally, the decoder with an auxiliary prediction head iteratively optimizes the object query to generate categories and boxes.

[0114] Specifically, the overall operation process of the method for detecting small targets at sea based on cross-modal fusion proposed in the present invention is as follows: a dual-stream GhostNet is used as the backbone network to process the dual-modal inputs of RGB visible light and infrared thermal imaging respectively. In the feature extraction stage, each branch first performs shallow feature extraction through a standard 16-channel convolution layer, and then stacks a group of Ghost bottleneck modules in a cascade manner, with the number of channels following a progressive expansion strategy. Each Ghost bottleneck module adopts a Ghost feature generation mechanism: first, the number of input channels is compressed to 1 / s of the original dimension through point-by-point convolution, and then a phantom feature map (Ghost features) is generated using a (s-1) group of layer-by-layer convolutions, and finally, the output of the same dimension as the original feature map is reconstructed through channel splicing. In the cross-modal feature fusion stage, the RGB and infrared dual branches are further embedded in a cross-modal attention feature fusion module (CMAFF) at the output of the third to fifth layers of the cascade. The RGB convolution feature maps and thermal convolution features output from the third to fifth layers are simultaneously input into the differential enhancement module and the common selection module of the CMAFF arranged in parallel. In the differential enhancement module, given the intermediate RGB convolution feature map and hot convolution feature maps , firstly, the direct subtraction of the two modes obtains the differential feature map Then, respectively Perform global average pooling to obtain And global maximum pooling to get , and After an hourglass-shaped two-layer shared convolutional layer Get and Secondly, and Add element by element and pass the activation function Sigmoid to get the channel dimension attention map , then multiply the attention map to each input feature map, and then add the residuals to get the enhanced feature map of each modality. Finally, the two enhanced feature maps are summed to get the output of the differential enhancement module In the public selection module, given the RGB convolution feature map and hot convolution feature maps , first obtain the common modal feature map by direct summation . Secondly, After global average pooling, , Then pass through two fully connected networks and get and The public selection module then calculates the RGB feature attention map and thermal feature attention map Next, the attention map of the RGB feature and the attention map of the thermal feature are multiplied by their inputs respectively. Finally, the output results of the previous step are added together to obtain the output of the common selection module Finally, and Add element by element to get . The third to fifth floors , , Input to the hybrid encoder respectively. Only the fifth layer Intra-scale feature interaction (AIFI) based on attention is used for intra-scale interaction. Specifically, First, it is flattened into a one-dimensional vector, and then linear projection is used to generate query (Query, Q), key (Key, K), and value (Value, V) vectors. These vectors are input to the AIFI module, and the multi-head self-attention mechanism (MSA) is used to capture the global context information. Finally, the output one-dimensional features are restored to the same spatial dimension as the input features to obtain the enhanced features. . Then , , The encoder features are obtained by inputting them into the CNN-based Cross-scale Feature Fusion (CCFF) Then, the uncertainty minimum query selects a fixed number of encoder features as the initial object query for the decoder. Finally, the decoder with the auxiliary prediction head iteratively refines the object query to generate categories and boxes.

[0115] The present invention also proposes a device for detecting marine targets based on multimodal fusion, comprising the following:

[0116] A data acquisition module is used to acquire visible light images and infrared imaging images of the same scene at sea, and synchronously input the visible light images and infrared imaging images into a processing module;

[0117] The processing module is constructed by a dual-stream backbone network based on GhostNet;

[0118] The operation process of the processing module is as follows:

[0119] In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module;

[0120] The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results.

[0121] Among them, the AIFI module and CCFF module together form a hybrid encoder;

[0122] The cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection.

[0123] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules.

[0124] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0125] Although the present invention has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be construed as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively encompassing the intended scope of the invention. In addition, the invention has been described above in terms of embodiments foreseen by the inventors for the purpose of providing a useful description, and those insubstantial modifications of the invention that are not currently foreseen may still represent equivalent modifications of the invention.

[0126] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. As long as the technical effects of the present invention are achieved by the same means, they shall fall within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementation methods may be made.

Claims

1. A method for detecting marine targets based on multimodal fusion, characterized in that: These include: Obtain visible light images and infrared imaging images of the same scene at sea, and simultaneously input the visible light images and infrared imaging images into a dual-stream backbone network built based on GhostNet; In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module; The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results. Among them, the AIFI module and CCFF module together form a hybrid encoder; The cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the marine target detection; Specifically, the cross-modal attention feature fusion module includes a differential enhancement module and a public selection module, which are arranged in parallel. The process of the differential enhancement module is as follows: given the intermediate RGB convolution feature map and hot convolution feature maps , where C represents the number of channels, H represents the height, and W represents the width. First, the direct subtraction of the two modalities obtains the differential feature map , then, respectively Perform global average pooling to obtain And global maximum pooling to get ,in and Both are convolution feature maps. and After a two-layer hourglass-shaped shared convolutional layer Get and , secondly, and Add element by element and pass the activation function Sigmoid to get the channel dimension attention map , then multiply the attention map to each input feature map, and then add the residuals to get the enhanced feature map of each modality; finally, sum the two enhanced feature maps to get the output of the differential enhancement module ; The process of the public selection module is as follows: given an RGB convolution feature map and hot convolution feature maps , first obtain the common modal feature map by direct summation , secondly, After global average pooling, , Then pass through two fully connected networks and get and ; Then the public selection module calculates the RGB feature attention map and thermal feature attention map ; Then, the attention map of RGB features and the attention map of thermal features are multiplied by their inputs respectively; Finally, the output results of the previous step are added together to obtain the output of the common selection module .

2. The method for detecting marine targets based on multimodal fusion according to claim 1, characterized in that: Specifically, the CCFF module realizes the progressive fusion of multi-scale features by introducing multiple cascaded fusion blocks. Each fusion block includes two key components: the first key component is two 1×1 convolutional layers for feature transformation in the channel dimension; the second key component is N RepBlocks composed of RepConv for feature fusion. When given the feature input of two adjacent scales, the CCFF module first adjusts the channel dimension through the 1×1 convolution of the first key component, then extracts features through the Rep Block of the second key component, and finally realizes feature fusion by element-by-element addition.

3. The method for detecting marine targets based on multimodal fusion according to claim 1, wherein: Specifically, the working principle of uncertainty minimum query selection includes: Feature uncertainty U() is defined as the difference measure between the prediction distributions of the two tasks of positioning P() and classification C(). In order to effectively reduce the uncertainty of query features, this uncertainty indicator is integrated into the loss function and minimized through a gradient-based optimization method.

4. The method for detecting marine targets based on multimodal fusion according to claim 3, characterized in that: Specifically, the cross-scale feature interaction and fusion results are input into the decoder after the uncertainty minimum query selection to complete the maritime target detection, including: The cross-scale feature interaction and fusion results are subjected to uncertainty minimum query selection to select a fixed number of encoder features as the initial object query for the decoder; finally, the decoder with auxiliary prediction head iteratively optimizes the object query to generate categories and boxes.

5. A device for detecting marine targets based on multimodal fusion, characterized in that: The method for detecting marine targets based on multimodal fusion according to any one of claims 1 to 4 is applied, wherein the device comprises: the following: A data acquisition module is used to acquire visible light images and infrared imaging images of the same scene at sea, and synchronously input the visible light images and infrared imaging images into a processing module; The processing module is constructed by a dual-stream backbone network based on GhostNet; The operation process of the processing module is as follows: In the third, fourth, and fifth layers of the dual-stream backbone network, features of the visible light image and the infrared imaging image are extracted, respectively, and these features are input into the cross-modal attention feature fusion module; The cross-modal attention feature fusion module performs cross-modal feature fusion to obtain fused features of the third, fourth, and fifth layers. The fused features of the fifth layer are then exchanged and fused within the attention-based intra-scale feature interaction module (AIFI module) to obtain an output result. The output result and the fused features of the third and fourth layers are input into the cross-scale feature fusion module (CCFF module) based on the convolutional neural network to obtain cross-scale feature interaction and fusion results. Among them, the AIFI module and CCFF module together form a hybrid encoder; The cross-scale feature interaction and fusion results are input into the decoder after the minimum uncertainty query selection to complete the maritime target detection.

Images