Underwater image target detection method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of underwater target detection, in particular to an underwater image target detection method and system based on multi-modal fusion. The method comprises the following steps: acquiring an underwater optical image and sonar image pair and a trained underwater multi-mode target detection network; inputting an image pair into the network, and respectively extracting multi-scale features of an optical mode and a sonar mode through a double-branch backbone network; a cross-modal feature offset is calculated by using an attention mechanism and a confidence mask through a multi-modal feature alignment module, and spatial adaptive alignment of acousto-optic features is realized; a multi-head zoom dot product attention and parallel double-gating structure is adopted through a cross-modal feature fusion module, bimodal complementary features are adaptively screened and fused, and target feature representation is enhanced; and the fused features are sent to the detected neck and head to complete target classification and positioning. According to the method, the problems of underwater acousto-optic mode space dislocation and imaging difference are solved, and the precision and robustness of target detection in a complex underwater environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underwater target detection, and particularly relates to a method and system for underwater image target detection based on multi-modal fusion. BACKGROUND

[0002] Underwater target detection is a core technology for autonomous underwater vehicles (AUVs) to realize environmental perception, and its performance directly determines the efficiency and safety of tasks such as ocean exploration, resource exploration, and ecological monitoring. Traditional underwater target detection mainly relies on a single sensing modality, including detection based on optical images and detection based on sonar images. However, the complexity and harsh imaging conditions of underwater environments severely restrict the applicability of single-modality detection technology.

[0003] Specifically, underwater optical cameras can obtain fine features such as color and texture of targets, but due to the effects of light absorption and scattering in seawater medium, there are problems such as limited detection distance, image prone to blue-green color cast and blurring degradation, and performance drops in turbid water or long-distance scenes. Underwater sonar has strong environmental adaptability and can detect target outlines at long distances without being affected by light, but sonar images have low resolution, sparse texture features, and severe background noise interference, making it difficult to distinguish the shape and texture of targets, and the classification accuracy of similar targets is low.

[0004] Multi-modal fusion technology has achieved remarkable results in automatic driving, robot navigation, and other fields by integrating complementary information from different sensing modalities. In underwater scenarios, sonar and optical modalities have natural complementarity: sonar provides long-distance contour positioning capability, and optical cameras provide near-distance detailed information. The fusion of the two can form an advantage complementary mode of "long-distance contour locking and near-distance detailed identification", effectively improving the perception confidence. However, underwater acoustic-optical fusion target detection faces the following core challenges:

[0005] First, data set scarcity restricts technology development. Underwater acoustic-optical paired data acquisition requires professional equipment, which is costly and time-consuming, and manual annotation needs to consider the correspondence of dual-modality targets, which is time-consuming and labor-intensive. Existing public data sets are single in type and do not cover a wide range of target categories such as aquatic organisms, making it difficult to support robustness verification of algorithms.

[0006] Second, spatial misalignment is serious. There are essential differences between sonar and optical sensor imaging mechanisms: optical cameras image in the X-Y plane, and sonar images based on echo intensity in the X-Z plane. Coupled with underwater dynamic disturbances (such as AUV attitude changes and water flow), the spatial dimensions of acoustic-optical image pairs are highly misaligned. Traditional physical registration methods relying on calibration boards are complex to operate and difficult to adapt to complex scenarios such as deep sea exploration.

[0007] Furthermore, the modal differences are significant. The feature distributions of sonar images and optical images differ significantly. Directly transferring land multimodal fusion techniques (such as RGB-D and RGB-T) will lead to low feature fusion efficiency and may even introduce redundant noise, making it difficult to fully leverage the advantages of acoustic-optical complementarity.

[0008] While existing technologies attempt to achieve weak alignment fusion through deformable attention mechanisms, they lack explicit prior information on alignment, resulting in insufficient alignment accuracy in scenarios with severe misalignment. Furthermore, existing fusion methods struggle to adaptively select effective features, easily introducing interference during the fusion process and leading to decreased detection accuracy.

[0009] In summary, there is an urgent need for an underwater multimodal target detection method and system that can adaptively handle spatial misalignment, effectively suppress modal differences, and fully exploit acoustic-optical complementary information, along with the construction of a high-quality benchmark dataset to promote the development of underwater intelligent sensing technology. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention provides an underwater image target detection method and system based on multimodal fusion, aiming to adaptively achieve acousto-optic modal feature alignment and deep fusion, thereby significantly improving the accuracy and robustness of underwater target detection.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] On one hand, embodiments of the present invention provide an underwater image target detection method based on multimodal fusion, the method comprising the following steps:

[0013] S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module.

[0014] S200, The optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps;

[0015] S300: For each corresponding scale of optical feature map and sonar feature map, the multimodal feature alignment module calculates the cross-modal feature offset, and performs spatial alignment correction on the sonar feature map based on the cross-modal feature offset value to obtain the aligned sonar feature map.

[0016] S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module. Cross-modal long-range dependencies are established through a multi-head scaling dot product attention mechanism. A parallel dual-gating structure is used to adaptively filter and fuse dual-modal complementary features, and an enhanced fused feature map is output.

[0017] S500, send the enhanced fusion feature map into the detection neck to perform multi-scale feature aggregation, and then perform target classification and bounding box regression through the detection head to obtain a final underwater target detection result.

[0018] In another aspect, the embodiment of the present application provides an underwater image target detection system based on multi-modal fusion, comprising:

[0019] at least one processor;

[0020] at least one memory for storing at least one program;

[0021] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0022] In another aspect, the embodiment of the present application provides a computer readable storage medium, wherein a processor executable program is stored, and the processor executable program is used to execute the above method when executed by a processor.

[0023] The embodiment of the present application has the following beneficial effects:

[0024] The adaptive feature alignment module (MFAM) proposed in the present application innovatively fuses attention mechanism and confidence mask, dynamically estimates cross-modal spatial offset without relying on physical calibration, generates explicit alignment prior information, effectively solves the spatial misalignment problem of acoustic-optical image pairs in underwater dynamic environment, and the alignment accuracy and robustness are significantly better than those of traditional weak alignment methods.

[0025] The cross-modal feature fusion module (CMFF) designed in the present application introduces multi-head scaled dot product attention and parallel double-gate structure, introduces sonar profile features through additive gating and filters optical noise through suppression gating, realizes adaptive bidirectional feature selection and differential fusion, deeply fuses acoustic-optical complementary information while effectively suppressing redundant noise, and the detection accuracy is improved by 2.02% compared with the baseline method.

[0026] The USOI dataset constructed in the present application fills the data gap in the field of underwater acoustic-optical fusion target detection, covers 9 typical underwater targets and dynamic video data, provides high-quality and standardized data support for algorithm verification and model training, and promotes the technical development in this field.

[0027] Without cross-modal physical pre-alignment, the present application realizes accurate matching and effective complementation of acoustic-optical features through end-to-end adaptive learning mechanism, simplifies the system deployment process, enhances the real-time perception ability of AUV in complex and variable underwater environment, and reaches the industry leading level on USOI and FLIR datasets. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0029] Figure 1 is a flow diagram of the underwater image target detection method based on multi-modal fusion in the embodiments of the present application;

[0030] Figure 2 is a general structure diagram of the underwater multi-modal target detection network (multimodal sonar-optical collaboration net, MSOCNet) in the embodiments of the present application, wherein (a) is the overall architecture, (b) is the structure of the multi-modal feature alignment module (MFAM), and (c) is the structure of the cross-modal feature fusion module (CMFF);

[0031] Figure 3 is a class display diagram in the USOI dataset;

[0032] Figure 4 is a class number statistical diagram of the USOI dataset. DETAILED DESCRIPTION

[0033] The disclosed concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with embodiments and drawings, so as to fully understand the purposes, schemes and effects of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0034] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. When the following description relates to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0035] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0036] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0037] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0038] refer to Figure 1 ,like Figure 1 The image shown is an underwater image target detection method based on multimodal fusion provided by an embodiment of the present invention. The method includes the following steps:

[0039] S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module.

[0040] S200, The optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps;

[0041] S300: For each corresponding scale of optical feature map and sonar feature map, the cross-modal feature offset is calculated by the multimodal feature alignment module, and the sonar feature map is spatially aligned and corrected based on the cross-modal feature offset value to obtain the aligned sonar feature map.

[0042] Specifically, for each corresponding scale of the optical feature map and the sonar feature map, the optical feature map and the sonar feature map are input into a multimodal feature alignment module, a cross-modal feature offset is calculated by jointly modeling the double-modal features, the spatial alignment correction of the sonar feature map is performed based on the cross-modal feature offset, so as to compensate for the spatial misalignment caused by the imaging difference of different modalities, and an aligned sonar feature map is obtained.

[0043] S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module, a cross-modal long-range dependency relationship is established by using a multi-head scaled dot-product attention mechanism, and a parallel double-gated structure is used to adaptively screen and fuse the double-modal complementary features, and an enhanced fusion feature map is output.

[0044] S500, the enhanced fusion feature map is sent to the detection neck to aggregate multi-scale features, and then the detection head is used to complete the classification and bounding box regression of the target, and finally the underwater target detection result is obtained.

[0045] The present application provides a kind of underwater image target detection method and system based on multimodal fusion, cross-modal feature offset is calculated explicitly by multimodal feature alignment module, effectively solve the spatial misalignment problem caused by the imaging mechanism difference and dynamic interference of underwater acoustic-optical image, avoid the defects of complex operation and poor adaptability of traditional physical registration method.Cross-modal feature fusion module innovatively establishes cross-modal long-range dependency relationship by using multi-head scaled dot-product attention mechanism, and adaptively screens and fuses double-modal complementary features by parallel double-gated structure, which can fully exploit the complementarity between the long-distance profile positioning information provided by sonar and the near-distance detail information provided by optical image, significantly suppress the redundant noise caused by modal difference, and improve the quality of fusion features.The present application can effectively cope with the core challenges of underwater data set scarcity, spatial misalignment and modal difference, by integrating the advantages of acoustic-optical modal, an efficient perception mode of "long-distance locking profile, near-distance refinement identification" is formed, thereby significantly improving the precision and robustness of underwater target detection, and providing more reliable technical support for marine exploration, resource exploration and ecological monitoring.

[0046] The core of the embodiment is a underwater multimodal target detection network (Multimodal Sonar-Optical Collaboration Net, MSOCNet) network framework, which realizes the cooperation of acoustic-optical modal information through adaptive alignment and double-gated fusion.

[0047] Data acquisition and network preparation: obtain underwater optical images and corresponding sonar images to form image pairs. Load a pre-trained underwater multimodal target detection network.

[0048] like Figure 2 As shown in (a), our proposed method is a two-branch backbone network specifically tailored for extracting features from underwater optical sonar image pairs. The network comprises:

[0049] Dual-branch backbone network: Optical and sonar features are extracted using ResNet structures respectively;

[0050] Multimodal Feature Alignment Module (MFAM): Enables cross-modal space alignment;

[0051] Cross-modal feature fusion module (CMFF): Enables deep feature fusion and enhancement;

[0052] Neck detection: FPN and PANet structures are used;

[0053] Detect the head: Perform classification and regression tasks;

[0054] This method mainly consists of three stages: unimodal feature extraction, bimodal feature alignment and fusion, and neck detection. Unimodal feature extraction can be expressed by the equation:

[0055] , ;

[0056] in , They represent from the first layer( The feature mapping of the optical and sonar branches of ). , and These represent the height, width, and number of channels of the feature map, respectively. , This represents the input optical and sonar images. The feature extraction functions for the optical and sonar branches are respectively, and their parameters are as follows: and In this invention, we use ResNet as... The function utilizes multi-scale features to capture targets of different sizes.

[0057] Secondly, given the feature map and Due to the significant differences between underwater optical imaging and forward-looking sonar imaging mechanisms, acousto-optic image pairs suffer from spatial and textural inconsistencies. Therefore, in multispectral target detection, cross-mode feature alignment is required first, followed by cross-mode feature fusion to aggregate features from different branches. This can be defined by the following formula:

[0058] ;

[0059] wherein, represents the fusion feature in the layer, represents a feature fusion function with parameters represents a feature alignment function with parameters . Existing research has explored different fusion structures and verified that intermediate fusion performs better than other fusion methods. We use intermediate fusion as the default setting to fuse multi-modal features from the convolutional layer . In the present invention, a cross-modal feature alignment module is proposed to simulate and an adaptive attention feature fusion module is proposed to simulate .

[0060] Finally, the feature maps from are fed into the detector neck for multi-scale feature fusion, and then delivered to the detector head for subsequent classification and regression.

[0061] ;

[0062] wherein, and represent a multi-scale feature aggregation and detection head function with parameters , , are the classification probability and bounding box coordinates of the target, respectively. FPN and PANET are usually used as functions to enhance the semantic expression and positioning ability of features, while takes on the role of classification and bounding box regression with parameters .

[0063] In some embodiments, in S300, the cross-modal feature offset is calculated by the multi-modal feature alignment module, including:

[0064] S310, the optical feature map and the sonar feature map are mapped to a unified query-key-value space via three independent linear projections, outputting an optical feature sequence and a sonar feature sequence;

[0065] Specifically, the optical feature map and the sonar feature map are input into the cross-modal feature fusion module, and the optical feature map and the sonar feature map are respectively transformed by three groups of linear mapping units independent of each other arranged in the cross-modal feature fusion module, and are mapped to a unified query vector space, a key vector space and a value vector space, outputting an optical feature sequence and a sonar feature sequence, wherein the optical feature map is used to generate a query vector, and the sonar feature map is used to generate a key vector and a value vector;

[0066] S320, a token-by-token similarity matrix of the optical query and the sonar key is calculated, a Softmax normalization is performed on the similarity matrix to obtain an alignment weight matrix, and the alignment weight matrix is applied to a value vector generated by the sonar feature to obtain a preliminary aligned sonar feature;

[0067] S330, a difference between the preliminary aligned sonar feature and the original sonar feature sequence is calculated as a difference residual, the difference residual is input into a residual connection input double-layer feedforward network for offset error correction, and multiplied by an adaptive learning scale factor to generate a preliminary alignment feature offset;

[0068] S340, the optical feature map and the sonar feature map are spliced and input into a two-layer fully connected multilayer perceptron for feature refinement, a confidence mask is output, the preliminary alignment feature offset is multiplied by the confidence mask element by element to obtain a cross-modal feature offset.

[0069] Specifically, the optical feature map and the sonar feature map are spliced in the feature dimension to obtain a fusion feature sequence, and the fusion feature sequence is input into a two-layer fully connected multilayer perceptron network arranged in the cross-modal feature fusion module, the fusion feature is refined by nonlinear feature transformation, a confidence mask representing the reliability of the cross-modal feature is output, and the preliminary alignment feature offset is multiplied by the confidence mask element by element to obtain a cross-modal feature offset.

[0070] The cross-modal feature offset obtained by the embodiment is used to accurately adjust the spatial position of the original sonar feature map, effectively compensating for the spatial misalignment problem of the sonar and optical images caused by the difference in imaging principles, and laying an accurate spatial foundation for subsequent feature fusion. Specifically, after obtaining the cross-modal feature offset, it is applied to each feature point of the sonar feature map, and the sonar feature map is geometrically transformed by interpolation and other methods, so that the target area in the sonar feature map can maintain high consistency with the corresponding target area in the optical feature map in spatial position. This fine alignment operation avoids the feature conflict and information interference that may occur when unaligned features are directly fused, significantly improves the spatial consistency of the dual-modal features, and thus ensures that the subsequent fusion module can more effectively extract and utilize the complementary information from the two modalities.

[0071] The multi-modal feature alignment module is introduced as follows:

[0072] Due to the difference in imaging mechanisms between acoustic sensors and optical sensors, there is often a significant spatial geometric misalignment in the images captured by the two in a shared scene. Unlike lidar, which can calculate an accurate projection matrix based on 3D point cloud, acoustic-optical images lack three-dimensional spatial correlation clues, making traditional alignment methods unsuitable. Existing solutions mostly rely on attention mechanisms to dynamically identify cross-channel feature correspondence. To improve the quality of feature alignment, the cross-modal feature alignment module MFAM innovatively combines attention mechanisms and confidence masks, as shown in structure (b). For clarity, this embodiment only illustrates the branch of sonar-enhanced optical image features. In the feature alignment stage, we first map the optical and sonar features to a unified query-key-value space via three independent linear projections, represented by the formula: Figure 2

[0073]

[0074]

[0075] In some embodiments, in S320, the similarity matrix of the optical query and the sonar key is calculated, and the alignment weight matrix is obtained by performing Softmax normalization on the similarity matrix. The alignment weight matrix is applied to the value vector generated by the sonar feature to obtain the preliminary aligned sonar feature, including:

[0076] S321, the optical feature sequence is taken as the query vector Q, and the sonar feature sequence is taken as the key vector K and the value vector V;

[0077] S322, the similarity matrix of the optical query and the sonar key is obtained by performing matrix multiplication operation on the transpose of the query vector Q and the key vector K, and then dividing by a scaling factor;

[0078] S323, the alignment weight matrix is obtained by performing Softmax normalization on the similarity matrix;

[0079] S324, the alignment weight matrix and the value vector V are subjected to matrix multiplication operation to obtain the preliminary aligned sonar feature.

[0080] ​​​​​​​​​​​​​In this embodiment, the attention mechanism is used to capture the fine-grained correspondence between optical features and sonar features, thereby realizing preliminary feature alignment. Specifically, the optical feature sequence is regarded as a query vector Q, and the sonar feature sequence is regarded as a key vector K and a value vector V, respectively. By calculating the matrix multiplication of Q and the transpose of K and dividing by a scaling factor (usually the square root of the dimension of the key vector to avoid gradient vanishing or explosion), a similarity matrix is obtained, which reflects the association strength between each optical feature token and each sonar feature token. After Softmax normalization of the similarity matrix, the alignment weight matrix obtained can effectively highlight the sonar feature regions that are most relevant to the optical features. Multiplying the alignment weight matrix by the value vector V is actually a weighted sum of the sonar features, which makes the preliminary aligned sonar features more semantically and spatially close to the optical features, laying a foundation for subsequent more accurate offset calculation and feature fusion. This attention-based preliminary alignment method can adaptively focus on the key corresponding parts of cross-modal features, effectively alleviating the initial feature misalignment problem caused by the difference in imaging principles.

[0081] In this embodiment, the similarity matrix between optical queries and sonar keys is calculated token by token to construct the association mapping between cross-modal features, and then the similarity matrix is subjected to Softmax normalization to obtain an attention matrix that can represent the inter-modal feature dependency weight. Finally, the attention matrix is applied to the value vector generated by the sonar features, and the sonar features that are spatially aligned with the optical features can be output. The mathematical expression of the above process is as follows:

[0082] ; ;

[0083] wherein, represents the alignment weight matrix for measuring the association strength between inter-modal features, and the superscript T represents the matrix transpose operation, is a scaling factor to alleviate the excessively large similarity calculation value, is the sonar feature after preliminary alignment by the attention mechanism.

[0084] In some embodiments, in S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module, a multi-head scaled dot-product attention mechanism is used to establish a cross-modal long-range dependency relationship, and a parallel double-gated structure is used to adaptively select and fuse the complementary features of the two modalities, and an enhanced fusion feature map is output, including:

[0085] In S410, the optical feature map and the aligned sonar feature map are respectively flattened into a sequence of tokens, and a learnable position embedding is added to each token to obtain a token sequence containing position information. The position embedding is a trainable parameter with the same dimension as the channel number of the token sequence.

[0086] S420, inject the mark sequence into the cross-modal feature offset, encode the space-channel difference between modalities into the attention calculation process to generate multi-head attention input features;

[0087] S430, perform a multi-head scaled dot product attention operation on the multi-head attention input features to build long-range dependencies between cross-modal features and generate attention features containing cross-modal correlation information; the number of heads of the multi-head scaled dot product attention operation is h, and the dimension of a single attention head is d, satisfying hd=C, where C is the number of channels of the mark sequence;

[0088] S440, filter and fuse the attention features using a parallel double-gated structure to obtain fused cross-modal features;

[0089] S450, sequentially perform residual connection, Dropout processing, and feedforward network operation on the fused cross-modal features, and then re-project the feature vector to the original space through nonlinear transformation and add it to the input mark sequence to output an enhanced fused feature map.

[0090] To further optimize the alignment accuracy, the difference between the preliminary aligned features and the original sonar features is input into a double-layer feedforward network through residual connection to fine-tune the offset error. The corrected output is multiplied by a scale factor learned adaptively during network training to finally generate a preliminary offset for precise alignment. The mathematical expression of the above process is as follows:

[0091] , ;

[0092] wherein, represents the difference between the aligned sonar features and the original sonar features, is a scale factor learned by the network, represents a double-layer linear feedforward network, is a preliminary alignment feature offset, is a Batch, is tokens, is Channels. To filter unreliable offset information, the optical features and the sonar features are concatenated and input into two layers of fully connected multilayer perceptron (MLP) for feature refinement, and a confidence mask with matching dimensions is output to quantify the reliability of the offset. Finally, the preliminary offset is multiplied element by element with the confidence mask to obtain a precise cross-modal feature offset, which provides an explicit feature alignment prior for the subsequent cross-modal fusion module. The mathematical expression of the above operation is as follows:

[0093] , ;

[0094] wherein, represents a confidence mask for screening reliable offset information, is a two-layer fully connected multilayer perceptron for feature refinement, represents an element-wise multiplication operation, is a cross-modal feature offset obtained after final alignment by filtering the confidence mask.

[0095] The cross-modal feature fusion module is introduced as follows:

[0096] Unlike pixel-intensive prediction tasks such as semantic segmentation and salient object detection, the target detection task focuses more on the feature representation of the target object. Therefore, the core of improving the performance of the multi-modal target detection network lies in efficiently fusing the target features of different modalities. Although explicit alignment information between different modalities is obtained after processing by the feature alignment module, due to the differences in the sound and light imaging mechanisms, the features of the same target under two modalities still have significant differences. If the features of the two modalities are directly concatenated or element-added, redundant noise is easily introduced, which may even affect the inference efficiency and detection accuracy of the detector. To solve this problem, the present application designs a cross-modal cross-feature fusion module CMFF, so that the single modality can efficiently learn more complementary information from the auxiliary modality, effectively excavate the complementary correlation between the optical and sonar modalities, and balance the detection efficiency and accuracy. The structure is as shown in Figure 2 (c).

[0097] Given the input feature map , first, the two types of feature maps are respectively flattened into a sequence of tokens, and a learnable position embedding is added to each token. This is a trainable parameter with a dimension of , which is used to encode the spatial position correlation between different tokens. After this processing, the token sequence containing position information can be obtained as the input of the CMFF module. On this basis, the offset estimated by the cross-modal alignment module is further injected, and the spatial-channel difference between modalities is encoded into the attention calculation process.

[0098] In some embodiments, in S420, the token sequence is injected into the cross-modal feature offset, the spatial-channel difference between modalities is encoded into the attention calculation process, and a multi-head attention input feature is generated, including:

[0099] S421, the token sequence containing position information is mapped to a query matrix, a key matrix and a value matrix through a linear layer;

[0100] S422, the cross-modal feature offset is adjusted in dimension so as to match the spatial dimension of the query matrix and the key matrix;

[0101] S423, the adjusted offset is added element-by-element to the query matrix and the key matrix respectively to obtain the query matrix and key matrix incorporating spatial-channel difference information.

[0102] This embodiment effectively integrates explicit spatial alignment prior information, such as cross-modal feature offsets, into the core computation of the attention mechanism. By pre-considering the spatial misalignment between the two modal features, the attention mechanism is guided to more accurately locate the correspondence between cross-modal features, avoiding attention weight allocation deviations caused by spatial-channel differences between modalities. This provides a more accurate input feature foundation for constructing long-range cross-modal dependencies in subsequent multi-head scaling dot product attention operations.

[0103] The specific formula is as follows:

[0104] ;

[0105] ;

[0106] in, , , These are the learnable parameter matrices for query, key, and value space mappings, respectively. , , , These are the multi-head query matrix, multi-head key matrix, and multi-head value matrix, respectively, after being split by the Reshape operation. This indicates the cross-modal feature offset. Dimensionally reshaped according to the number of multi-head attention heads h, so as to align with the multi-head key matrix. Matching the spatial dimensions so as to subsequently... Element-wise addition is performed to encode the spatial-channel difference information between modes into the key matrix. This means reshaping the cross-modal feature offset O according to the number of multi-head attention heads h, so that it is aligned with the multi-head value matrix. Matching the spatial dimensions, and thus with Element-wise addition is performed to synchronously encode the spatial-channel differences between modalities into the value matrix. This method ensures that in subsequent multi-head scaling dot product attention calculations, the similarity calculation between queries and keys / values ​​fully considers the inherent spatial misalignment and channel characteristic differences between modalities, thereby more accurately capturing the correspondence between cross-modal features. , These represent the multihead key matrix and multihead value matrix, respectively, after being reshaped by the cross-modal feature offset O and incorporating intermodal spatial-channel difference information.

[0107] In some embodiments, in S430, the multi-head attention input features are subjected to a multi-head scaled dot-product attention operation to generate attention features containing cross-modal correlation information, and a parallel double-gating structure is used to screen and fuse the attention features to obtain fused cross-modal features, including:

[0108] S431, the query matrix and the key matrix are subjected to scaled dot-product attention calculation to obtain an attention matrix;

[0109] S432, the attention matrix is multiplied by the value matrix to obtain the output features of a single attention head, and the output features of all attention heads are spliced in the channel dimension to generate attention features containing cross-modal correlation information;

[0110] S433, after the optical feature sequence and the fused features are spliced in the channel dimension, they are respectively input into an additive gate and an inhibitory gate; wherein the additive gate and the inhibitory gate are both realized by two fully connected layers combined with Gaussian error linear unit activation function for nonlinear transformation, and the Sigmoid activation function is used to generate the additive gate activation output and the inhibitory gate activation output;

[0111] S434, the additive gate activation output is multiplied element by element with the attention features to obtain sonar complementary features screened and enhanced by the additive gate; the inhibitory gate activation output is negated and multiplied element by element with the optical feature sequence to obtain optical features denoised by the inhibitory gate;

[0112] S435, the sonar complementary features and the optical features are added element by element to obtain the fused cross-modal features.

[0113] The embodiment performs multi-head scaled dot-product attention to build long-range dependency relationship between cross-modal features and fully excavate the correlation information of heterogeneous modalities. To avoid the problem that a single modality feature is submerged in one-way fusion, a parallel double-gating structure is designed. The additive gate introduces the complementary features of the sonar modality through adaptive weight adjustment to strengthen the target key information. The inhibitory gate specifically filters the potential noise in the optical modality. The two work together to achieve fine-grained enhancement and boundary information protection of the target features. The mathematical expression of the above process is as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] wherein, The attention matrix calculated for multi-head attention, used to measure the correlation strength between different modal feature labels; The dimension of a single attention head; The fusion feature obtained by concatenating the features output by the multi-head attention; The activation output of the additive gate, representing the Sigmoid activation function, The optical feature sequence and the fusion feature are concatenated in the channel dimension; The activation output of the inhibition gate, The parameter matrix of the additive gate, The parameter matrix of the inhibition gate, used to implement nonlinear transformation, expressed as , , The weight matrices of the two fully connected layers, respectively, , The corresponding bias terms, and GELU is the Gaussian Error Linear Unit activation function; The operation of concatenating the features output by each head of the multi-head attention in the channel dimension.

[0119] In some embodiments, in S440, the fused cross-modal features are sequentially subjected to residual connection, Dropout processing and feedforward network operation, and then the feature vector is re-projected to the original space through nonlinear transformation and added to the input label sequence to output an enhanced fusion feature map, including:

[0120] S441, the activation output of the additive gate is multiplied element by element with the fusion feature, while the activation output of the inhibition gate is multiplied element by element with the optical feature sequence, and the result of the former is subtracted from the result of the latter, then the optical feature sequence is added and Dropout operation is performed to obtain the intermediate enhanced optical feature;

[0121] S442, the intermediate enhanced optical feature is mapped to the original space through the output weight matrix, and then added to the original optical feature sequence after being multiplied by the learnable weight parameter, to obtain the feature vector after projection and residual fusion;

[0122] S443, the feature vector after projection and residual fusion is subjected to feedforward network operation, and the operation result is added to the feature vector after being multiplied by the learnable weight parameter, to output an enhanced fusion feature map.

[0123] After residual connection and Dropout feedforward network processing, the enhanced optical feature is obtained, and then the vector is re-projected to the original space through nonlinear transformation The sequence is reprojected back to the original space and added to the input sequence via a residual connection, as shown in the following formula:

[0124] ;

[0125] ;

[0126] ;

[0127] in, This is an intermediate enhanced optical feature obtained through the combined effect of additive gating and suppression gating. This indicates element-wise multiplication, and Drop(·) is the Dropout operation used to prevent overfitting of the model. The feature vector is obtained after projection and residual fusion, and α and β are learnable weight parameters used to dynamically balance the contribution ratio of enhanced features and original optical features. This is the output weight matrix before the FFN layer. , used to map feature dimensions back to the original space; This is the enhanced fused feature map after processing by a feedforward network (FFN) and then reconnecting with residuals. γ and δ are learnable weight parameters used to adjust the fusion ratio between the feedforward network output features and the residual features. α, β, γ, and δ are initialized to 1 during training. FFN(·) represents the feedforward network operation, which further enhances the expressive power of the features through two layers of linear transformation and nonlinear activation functions.

[0128] This embodiment achieves precise screening and dynamic fusion of cross-modal features through a parallel dual-gating structure. Additive gating learns the nonlinear mapping relationship between the optical feature sequence and the concatenated fused features, generating activation weights in the 0-1 range to adaptively enhance complementary features beneficial for target detection in the sonar mode, such as target contour information that sonar signals may retain in low-light or complex water environments. Suppression gating, on the other hand, targets redundant information in the optical features, such as water scattering noise and bubble interference, by generating a suppression mask through inversion, selectively denoising the optical features while preserving their clear texture details and color information. The two gating mechanisms work synergistically, fully utilizing the complementary information of the optical and sonar modes to effectively suppress noise interference. The final result is a fused cross-modal feature that contains key multimodal information while effectively suppressing noise interference, significantly improving the discriminativeness and robustness of the target feature representation and providing high-quality feature input for subsequent target detection tasks.

[0129] Dataset creation:

[0130] Currently, there is a lack of specialized datasets in the field of underwater acoustic-optical fusion target detection, which greatly limits the development and performance verification of this direction. Therefore, this study constructs a benchmark dataset to provide standardized data support for autonomous underwater vehicle multi-modal target detection tasks.

[0131] In the data acquisition stage, our underwater emergency rescue ROV platform is configured as follows: the AUV is equipped with a BlueView M900 two-dimensional forward-looking sonar on the top, which has a 130° ultra-wide field of view, a 6.25 cm high-resolution feature, and a maximum detection range of 100 m, which can efficiently obtain the acoustic feature information of underwater targets; the platform is equipped with a zed2 underwater binocular optical camera below, which supports 1280×800 and 640×400 resolution output, and can capture underwater optical images in real time at a frame rate of 60 fps, ensuring the clarity and timeliness of visual information. All data were collected in Sanya, China. Before collection, the sonar and camera were strictly calibrated in space and parameters. During the collection process, the ROV was controlled to take pictures of the target area from different angles and distances, while the acoustic and optical images of the same scene were stored synchronously, ensuring the precise alignment of the two types of data in the time dimension and laying a foundation for data consistency for subsequent fusion tasks.

[0132] To meet the application requirements of AUV in fish analysis, underwater exploration and other key practical tasks, our dataset has comprehensive coverage of targets, including 9 typical underwater targets: cage, frame, hook, anchor, tire, rov, plastic bucket, fish and oil drums, covering two core categories of underwater organisms and stationary objects, as shown in Figure 3 It is worth noting that, to expand the application scenarios of the dataset, we additionally conducted continuous frame video shooting for fish and underwater vehicle dynamic targets, and the obtained video data can be directly used for algorithm training and performance evaluation in the field of underwater target tracking.

[0133] In the data post-processing stage, we removed redundant images of repeated scenes, and checked the time synchronization accuracy of acoustic and optical data, finally selected 4000 valid acoustic-optical image pairs. Our dataset not only enriches the data resources of acoustic-optical fusion target detection, but also is the first underwater multi-modal target detection and tracking benchmark dataset that covers underwater organisms and stationary objects and contains dynamic target continuous video data, providing rich and reliable data support for underwater intelligent perception related research.

[0134] Experiments and results:

[0135] Our method is based on the PyTorch 2.6.0 framework, and the experimental environment is deployed on an Ubuntu 20.04 server with a CPU R9-7900X, 64G memory, and an NVIDIA RTX 4090D 24G GPU, which can meet the computing power requirements during model training. In the training stage, the batch size is set to 25 epochs, the optimizer uses SGD, the initial learning rate is 1.0x10-2, the momentum parameter is 0.937, and a weight decay factor of 0.0005 is introduced to suppress overfitting. The learning rate decay strategy uses cosine annealing. The image input size is uniformly adjusted to 640x640, and mosaic data augmentation and random flip operations are used to improve the model's adaptability to different scene features.

[0136] To fully verify the detection performance of the model, we selected the representative FLIR dataset based on the USOI dataset for testing. The following will detail the sample size, division method, and target categories of the two datasets:

[0137] (1) USOI dataset: After labeling the 4000 image pairs, we assigned the images to the training set, validation set, and test set in a 7:2:1 ratio, resulting in 2800 training sets, 800 validation sets, and 400 test sets. The distribution of different types of targets in the dataset is shown in Figure 4 .

[0138] (2) FLIR dataset: FLIR is a challenging multispectral object detection dataset that includes both daytime and nighttime scenes. We selected the FLIR-aligned version, which has 5142 aligned multispectral image pairs, and divided them into training, validation, and test sets in a 7:2:1 ratio, resulting in 3599 training sets, 1029 validation sets, and 514 test sets. It contains four types of objects, namely people, cars, dogs, and bicycles.

[0139] During the experiment, we used the mean average precision (mAP) with an IoU of 0.5 to evaluate the performance of the multi-modal object detector. Specifically, we examined the average precision (AP) for each individual class and the overall mAP for all classes. AP is derived from the recall-precision (P-R) curve in object detection.

[0140] Sonar-enhanced optical experiments:

[0141] Objective comparison: In the quantitative analysis experiment, we select the multimodal feature fusion network designed for other scenes and the current mainstream single modal detector for comparison. In order to compare different multimodal feature fusion methods, we do not use the decoder of these methods in the comparison experiment, and only apply the multimodal feature fusion technology proposed by them to the target detection task. Specifically, we select CMX, CMOT and MFUR suitable for RGB-D multimodal tasks, and select EAEFNet, DEYOLO, BDLFusion and Crossfuse suitable for RGB-T tasks for realizing feature fusion of acoustic image and optical image, and use YOLOv12 and YOLOv13 to carry out performance evaluation in the optical single modal detection task. In order to ensure the fairness of comparison, all multimodal fusion methods are realized on the unified benchmark detection network ICAFusion. The network includes three core parts: two backbone networks for extracting features of different modalities, a neck for multi-scale feature fusion, and a detection head for optimizing detection results.

[0142] We use the way of enhancing optical features with sonar feature information to carry out target detection, and the objective index evaluation experiment results of each comparison method are shown in Table 1. From the results in the table, it can be obviously seen that the multimodal fusion methods represented by CMX and EAEFNet rely on the aligned image in the cross-modal feature fusion. Although these methods perform better than the benchmark network ICAFusion in fusing underwater acoustic and optical modal features, the inherent heterogeneity and serious misalignment of acoustic and optical images will hinder the capture of the positional dependence relationship between modalities in the fusion process and introduce redundant features. In contrast, the method proposed in the application automatically identifies the feature dependence relationship between modalities through the MFAM module, then integrates the alignment feature offset, and realizes the deep fusion of acoustic and optical complementary features through the CMFF module. Finally, the detection performance of MSOCNet in most categories is better than that of other methods, and the average precision mean index is 2.02% higher than that of the benchmark method, and the average detection precision of the method is improved by 1.08% compared with the second-ranked Crossfuse. In addition, compared with the two single modal benchmark models YOLOv12 and YOLOv13, MSOCNet improves the average precision by 2.73% and 3.09% respectively by effectively fusing sonar modal features, breaking the information limitation of a single modality, realizing the precise information cooperation of sonar and optical images, and thus achieving more excellent detection results.

[0143] Table 1: Comparison results of target detection methods based on enhanced optical features on USOI dataset

[0144]

[0145] The results of Table 1 show that the mAP of MSOCNet in the task of enhancing the detection of optical features reaches 95.64%, which is 2.02% higher than the benchmark ICAFusion and 2.73% higher than YOLOv2. The AP of fish and rov, which are difficult to detect, reaches 89.23% and 88.74% respectively, which is significantly better than the comparison method.

[0146] Optical enhanced sonar experiment:

[0147] Objective comparison: In the quantitative analysis of the optical enhanced sonar experiment, we also compared the detection results of the multi-modal feature fusion methods CMOT, DEYOLO, MFUR, CMX, Crossfuse, BDLFusion, EAEFNet and the single-modal models YOLOv2, YOLOv3 in the sonar image task based on the unified ICAFusion detection framework. The test results are shown in Table 2.

[0148] As can be seen from Table 2, the two multi-modal fusion methods represented by EAEFNet and CMX still maintain a high mAP in the sonar dominant scene, reaching 84.34% and 84.58% respectively, but their feature interaction is based on the ideal assumption of quasi-synchronization and alignment. When the sonar image has severe speckle, projection distortion or spatial offset with the optical image, this kind of method often misjudges the optical edge as the real target, resulting in missed detection and false alarm of small or high-reflectivity targets such as fish and oil drums. In contrast, our method first mines the local structure consistency of sonar-optical through the MFAM module, and then makes the optical complementary features deeply fused into the sonar features through the CMFF module. Thanks to this, the average precision of MSOCNet in the Oildrums category is improved by about 10% compared with the baseline model, and the overall mAP reaches 85.82%, which is 1.24 percentage points higher than the sub-optimal CMX, verifying that the introduction of optical information enhancement in the sonar main mode can still suppress heterogeneous noise and achieve more robust target detection. In addition, the single-modal model still lags behind the multi-modal method as a whole. YOLOv2 and YOLOv3 are close to the benchmark in the regular categories, but they significantly degrade in the performance of fish, which has strong modality complementarity. The average precision of YOLOv2 and YOLOv3 is reduced by 4.25% and 7.41% respectively compared with the benchmark, which clearly reveals the information deficiency problem of the single-modal model in the complex noise environment, and highlights the technical advantage of MSOCNet in effectively making up for the deficiency of a single modality and improving the robustness of detection through cross-modal information cooperation.

[0149] Table 2: Comparison results of target detection methods based on enhanced sonar features on USOI dataset

[0150]

[0151] Ablation experiment:

[0152] To verify the effectiveness of the multi-modal feature alignment module MFAM and the cross-modal feature fusion module CMFF, the present application takes the USOI dataset as an experimental carrier, and selects ICAFusion as a benchmark detection framework to carry out an ablation experiment. During the experiment, by gradually introducing the modules to be verified, comparing the average precision mean and parameter quantity of the model under different module combinations, the performance of each module on the model target detection and the influence of the model complexity are quantitatively and qualitatively analyzed. The experimental data is presented through the quantitative indicators in Table 3, and both of them jointly prove the progressive optimization value of the modules from the intuitive effect and the numerical dimension.

[0153] From the quantitative results in Table 3, the performance improvement shows a clear progressive relationship. When using the ICAFusion benchmark framework only, the model mAP is 93.62%. After introducing the MFAM module alone, due to the reduction of cross-modal alignment error, the mAP is improved by 1.06 percentage points compared with the benchmark. When the CMFF module is introduced alone, the mAP is further improved to 95.13% due to the preliminary relief of feature redundancy interference. When MFAM and CMFF form a synergy - MFAM provides accurate cross-modal alignment prior, and CMFF efficiently aggregates complementary features on this basis, the model mAP finally reaches 95.64%, which is improved by 2.02 percentage points compared with the benchmark framework, and the performance gain is significantly better than the effect of introducing a single module. It is worth noting that the parameter quantity data in Table 3 also shows that even if MSOCNet enables the feature alignment and fusion modules at the same time, the parameter quantity is still much lower than that of the ICAFusion benchmark model, which realizes the optimization of model efficiency while improving the detection accuracy.

[0154] In summary, the MFAM module can effectively solve the spatial misalignment problem of underwater acoustic-optical modal, and lay a precise alignment foundation for subsequent feature fusion. The CMFF module can filter redundant noise and strengthen the complementary information mining between modal. The combination of the two is the key to the performance leap of the target detection based on underwater acoustic-optical information synergy, and also has good performance in parameter quantity control, further proving the scientificity and effectiveness of the module design proposed by the present application.

[0155] Table 3: Comparison results and parameter quantity analysis of ablation experiment

[0156]

[0157] The objective evaluation index experimental results of MSOCNet on the FLIR dataset are shown in Table 4. As can be seen from the table, MSOCNet is superior to the baseline ICAFusion method, and the average detection accuracy is improved by 1.23%. Even compared with the DEYOLO model specially designed for RGB-T modal feature fusion, the proposed MSOCNet can still improve the average detection accuracy by 0.26%. Notably, the detection accuracy advantage of MSOCNet is particularly prominent on the Dog category, which further embodies the positive role of MSOCNet in the fusion of aligned multi-modal land-based images.

[0158] Table 4: Objective evaluation index experimental results of MSOCNet on the FLIR dataset

[0159]

[0160] Conclusion: The present application aims at the target detection challenge in complex underwater environment, deeply studies the target detection paradigm based on the cooperation of sonar-optical modal information, and proposes a multi-modal underwater image target detection network MSOCNet which can efficiently utilize the complementary information of acoustic-optical images. In order to solve the spatial dimension misalignment problem of acoustic and optical modal images and the significant difference in imaging principle, a self-adaptive alignment-cross-modal fusion double-layer architecture is used to realize the deep integration of multi-modal features. The cross-modal feature alignment module MFAM combines attention mechanism and confidence mask, dynamically estimates the spatial offset of cross-modal features and generates explicit alignment prior information, realizing the adaptive self-calibration of acoustic and optical features. The cross-modal feature fusion module CMFF introduces multi-head scaling dot product attention and parallel double-gate structure, which not only adaptively selects the effective target area in acoustic and optical images, but also differentially fuses the selected features to suppress redundant noise and invalid background information. MFAM and CMFF cooperatively form a progressive feature integration process of coarse alignment-fine fusion, which can efficiently exploit the complementary value of acoustic and optical modalities without physical pre-registration operation. In addition, we independently collect the first high-quality acoustic-optical image benchmark dataset USOI which covers 9 typical underwater objects, contains continuous video of dynamic targets and is strictly time-synchronized. A large number of experiments on USOI and FLIR are carried out to verify the effectiveness and cross-domain robustness of MSOCNet, thereby providing a new technical approach and data basis for high-precision target detection tasks in complex underwater environments.

[0161] The embodiment of the present application also provides a multi-modal fusion based underwater image target detection system, comprising:

[0162] at least one processor;

[0163] at least one memory for storing at least one program;

[0164] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0165] The contents in the above method embodiments are applicable to the present embodiment, the present embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments, which will not be described herein again.

[0166] The present embodiment further provides an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0167] It can be understood that the contents in the above method embodiments are applicable to the present device embodiment, the present device embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0168] The present embodiment further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0169] It can be understood that the contents in the above method embodiments are applicable to the present storage medium embodiment, the present storage medium embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0170] The present embodiment further provides a computer program product, which includes a computer program or computer instructions, the computer program or computer instructions are stored in a memory, a processor of a computer device reads the computer program or computer instructions from the memory, and the processor executes the computer program or computer instructions to make the computer device execute the above method.

[0171] It can be understood that the contents in the above method embodiments are applicable to the present program product embodiment, the present program product embodiment specifically implements the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0172] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include memory that is remotely located with respect to the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] Those of ordinary skill in the art will appreciate that all or certain steps in the methods disclosed above, systems can be implemented as software, firmware, hardware, and appropriate combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media.

[0174] The apparatus embodiments described above are merely illustrative, and units described as separate components can or can not be physically separate, i.e., can be located in one place or can be distributed over multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

Claims

1. A method for underwater image target detection based on multimodal fusion, characterized in that, The method includes the following steps: S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module. S200, The optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps; S300: For each corresponding scale of optical feature map and sonar feature map, the cross-modal feature offset is calculated by the multimodal feature alignment module, and the sonar feature map is spatially aligned and corrected based on the cross-modal feature offset value to obtain the aligned sonar feature map. S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module. Cross-modal long-range dependencies are established through a multi-head scaling dot product attention mechanism. A parallel dual-gating structure is used to adaptively filter and fuse dual-modal complementary features, and an enhanced fused feature map is output. S500, the enhanced fusion feature map is sent to the detection neck for multi-scale feature aggregation, and then the target classification and bounding box regression are completed through the detection head to obtain the final underwater target detection result.

2. The method according to claim 1, characterized in that, In S300, the calculation of cross-modal feature offset through the multimodal feature alignment module includes: S310, the optical feature map and sonar feature map are mapped to a unified query-key-value space through three sets of independent linear projections, and the optical feature sequence and sonar feature sequence are output. S320: Calculate the token-by-token similarity matrix between the optical query and the sonar key. Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix. Apply the alignment weight matrix to the value vector generated by the sonar features to obtain the sonar features after preliminary alignment. S330, calculate the difference between the initially aligned sonar features and the original sonar feature sequence as the difference residual, input the difference residual into a two-layer feedforward network through the residual connection for offset error correction, and multiply it by the adaptive learning scale factor to generate the initial aligned feature offset. S340, the optical feature map and the sonar feature map are stitched together and then input into a two-layer fully connected multilayer perceptron for feature refinement. A confidence mask is output, and the preliminary aligned feature offset is multiplied element by element with the confidence mask to obtain the cross-modal feature offset.

3. The method according to claim 2, characterized in that, In S320, the token-by-token similarity matrix between the optical query and the sonar key is calculated. This similarity matrix is ​​then subjected to Softmax normalization to obtain an alignment weight matrix. This alignment weight matrix is ​​applied to the value vector generated from the sonar features to obtain the preliminarily aligned sonar features, including: S321, the optical feature sequence is used as the query vector Q, and the sonar feature sequence is used as the key vector K and the value vector V; S322, by performing matrix multiplication on the transpose of the query vector Q and the key vector K, and then dividing by the scaling factor, the token-by-token similarity matrix between the optical query and the sonar key is obtained. S323, Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix; S324, Perform matrix multiplication on the alignment weight matrix and the value vector V to obtain the sonar features after preliminary alignment.

4. The method according to claim 1, characterized in that, In S400, the process of inputting the optical feature map and the aligned sonar feature map into the cross-modal feature fusion module, establishing cross-modal long-range dependencies through a multi-head scaling dot product attention mechanism, and adaptively filtering and fusing bimodal complementary features using a parallel dual-gating structure to output an enhanced fused feature map includes: S410, the optical feature map and the aligned sonar feature map are flattened into a sequence of labels, and a learnable position embedding is added to each label to obtain a label sequence containing position information; the position embedding is a trainable parameter with the same dimension as the number of channels in the label sequence; S420, the labeled sequence is injected with cross-modal feature offsets to encode the spatial-channel differences between modalities into the attention calculation process, generating multi-head attention input features; S430, perform a multi-head scaling dot product attention operation on the multi-head attention input features to generate attention features containing cross-modal association information. Use a parallel dual-gating structure to filter and fuse the attention features to obtain fused cross-modal features. The number of heads in the multi-head scaling dot product attention operation is h, and the dimension of a single attention head is d, satisfying hd=C, where C is the number of channels in the labeled sequence. S440, the fused cross-modal features are sequentially subjected to residual connection, Dropout processing and feedforward network operation, and then the feature vector is reprojected onto the original space through nonlinear transformation and added to the input label sequence to output an enhanced fused feature map.

5. The method according to claim 4, characterized in that, In S420, injecting the labeled sequence into the cross-modal feature offset, encoding the spatial-channel differences between modalities into the attention calculation process, and generating multi-head attention input features includes: S421, the labeled sequence containing location information is mapped into a query matrix, a key matrix, and a value matrix through a linear layer; S422, adjust the dimensions of the cross-modal feature offsets to match the spatial dimensions of the query matrix and the key matrix; S423, the adjusted offset is added element-by-element to the query matrix and the key matrix respectively to obtain the multi-head key matrix and multi-head value matrix incorporating spatial-channel difference information.

6. The method according to claim 4, characterized in that, In S430, the multi-head scaled dot product attention operation is performed on the multi-head attention input features to generate attention features containing cross-modal association information. A parallel dual-gating structure is then used to filter and fuse the attention features to obtain fused cross-modal features, including: S431, Scaling dot product attention calculation is performed on the query matrix and key matrix that incorporate spatial-channel difference information to obtain the attention matrix; S432, multiply the attention matrix with the value matrix that incorporates spatial-channel difference information to obtain the output features of a single attention head, and concatenate the output features of all attention heads in the channel dimension to generate attention features containing cross-modal correlation information; S433: The optical feature sequence and the fused feature are concatenated in the channel dimension and then input into the additive gating and the suppression gating, respectively. The additive gating and the suppression gating are both implemented through two fully connected layers combined with the Gaussian error linear unit activation function to achieve nonlinear transformation, and the additive gating activation output and the suppression gating activation output are generated by the Sigmoid activation function. S434: The additive gated activation output is multiplied element-wise with the attention features to obtain the sonar complementary features enhanced by additive gated filtering; the suppression gated activation output is inverted and multiplied element-wise with the optical feature sequence to obtain the optical features after suppression gated noise reduction. S435, the sonar complementary features and optical features are added element by element to obtain the fused cross-modal features.

7. The method according to claim 4, characterized in that, In S440, the fused cross-modal features are sequentially subjected to residual connections, Dropout processing, and feedforward network operations. Then, the feature vectors are reprojected onto the original space through a nonlinear transformation and added to the input label sequence to output an enhanced fused feature map, including: S441 uses the activation output of additive gating to multiply the fused features element-wise, and simultaneously uses the activation output of suppression gating to multiply the optical feature sequence element-wise. The former result is subtracted from the latter result, and then added to the optical feature sequence before performing a Dropout operation to obtain the intermediate enhanced optical features. S442, the intermediate enhanced optical feature is mapped to the original space through the output weight matrix, and then multiplied by the original optical feature sequence by the learnable weight parameters and added to obtain the feature vector after projection and residual fusion; S443, perform feedforward network operation on the feature vector after projection and residual fusion, multiply the operation result by the feature vector by the learnable weight parameters and add them together to output the enhanced fused feature map.

8. An underwater image target detection system based on multimodal fusion, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images