A Method and System for Underwater Image Target Detection Based on Multimodal Fusion

Abstract

This invention relates to the field of underwater target detection technology, specifically to an underwater image target detection method and system based on multimodal fusion. The method includes: acquiring underwater optical and sonar image pairs and a trained underwater multimodal target detection network; inputting the image pairs into the network, extracting multi-scale features of the optical and sonar modes respectively through a dual-branch backbone network; calculating cross-modal feature offsets using an attention mechanism and confidence mask through a multimodal feature alignment module to achieve spatial adaptive alignment of acoustic and optical features; adaptively filtering and fusing complementary features of the two modes through a cross-modal feature fusion module using multi-head scaling dot product attention and a parallel dual-gating structure to enhance target feature representation; and sending the fused features to the detection neck and head to complete target classification and localization. This invention solves the problems of spatial misalignment and imaging differences in underwater acoustic and optical modes, improving the accuracy and robustness of target detection in complex underwater environments.

Description

Technical Field

[0001] This invention relates to the field of underwater target detection technology, and specifically to an underwater image target detection method and system based on multimodal fusion. Background Technology

[0002] Underwater target detection is a core technology for autonomous underwater vehicles (AUVs) to achieve environmental awareness, and its performance directly determines the efficiency and safety of tasks such as marine exploration, resource exploration, and ecological monitoring. Traditional underwater target detection mainly relies on a single sensing modality, including detection based on optical images and detection based on sonar images. However, the complexity of the underwater environment and the harsh imaging conditions severely limit the applicability of single-modality detection technology.

[0003] Specifically, while underwater optical cameras can acquire fine features such as the color and texture of targets, they suffer from limited detection range, image blurring and degradation due to the light absorption and scattering effects of seawater. Their performance drops sharply in turbid waters or at long distances. Underwater sonar, on the other hand, possesses strong environmental adaptability, can detect target outlines over long distances unaffected by lighting conditions, but suffers from low image resolution, sparse texture features, and severe background noise interference, making it difficult to distinguish the shape, texture, and other fine attributes of targets, resulting in low accuracy in classifying similar targets.

[0004] Multimodal fusion technology, by integrating complementary information from different perception modalities, has achieved significant results in fields such as autonomous driving and robot navigation. In underwater scenarios, sonar and optical modalities are naturally complementary: sonar provides long-range contour localization capabilities, while optical cameras provide close-range detail information. Their fusion can form a complementary model of "long-range contour locking and close-range detailed recognition," effectively improving perception confidence. However, underwater acoustic-optical fusion target detection faces the following core challenges:

[0005] First, the scarcity of datasets hinders technological development. Underwater acoustic-optical pairing data acquisition requires specialized equipment, is costly and time-consuming, and manual annotation requires considering the correspondence between bimodal targets, which is also time-consuming and labor-intensive. Existing publicly available datasets are of limited types and do not cover a wide range of target categories such as aquatic organisms, making it difficult to support the robustness verification of algorithms.

[0006] Secondly, the spatial misalignment problem is severe. There are fundamental differences in the imaging mechanisms of sonar and optical sensors: optical cameras image in the XY plane, while sonar images in the XZ plane based on echo intensity. Coupled with underwater dynamic disturbances (such as AUV attitude changes and water flow), this leads to a high degree of spatial misalignment between the acoustic and optical image pairs. Traditional physical registration methods relying on calibration plates are complex to operate and difficult to adapt to complex scenarios such as deep-sea exploration.

[0007] Furthermore, the modal differences are significant. The feature distributions of sonar images and optical images differ significantly. Directly transferring land multimodal fusion techniques (such as RGB-D and RGB-T) will lead to low feature fusion efficiency and may even introduce redundant noise, making it difficult to fully leverage the advantages of acoustic-optical complementarity.

[0008] While existing technologies attempt to achieve weak alignment fusion through deformable attention mechanisms, they lack explicit prior information on alignment, resulting in insufficient alignment accuracy in scenarios with severe misalignment. Furthermore, existing fusion methods struggle to adaptively select effective features, easily introducing interference during the fusion process and leading to decreased detection accuracy.

[0009] In summary, there is an urgent need for an underwater multimodal target detection method and system that can adaptively handle spatial misalignment, effectively suppress modal differences, and fully exploit acoustic-optical complementary information, along with the construction of a high-quality benchmark dataset to promote the development of underwater intelligent sensing technology. Summary of the Invention

[0010] To address the aforementioned technical problems, this invention provides an underwater image target detection method and system based on multimodal fusion, aiming to adaptively achieve acousto-optic modal feature alignment and deep fusion, thereby significantly improving the accuracy and robustness of underwater target detection.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] On one hand, embodiments of the present invention provide an underwater image target detection method based on multimodal fusion, the method comprising the following steps:

[0013] S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module.

[0014] S200, the optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps;

[0015] S300: For each corresponding scale of optical feature map and sonar feature map, the multimodal feature alignment module calculates the cross-modal feature offset, and performs spatial alignment correction on the sonar feature map based on the cross-modal feature offset value to obtain the aligned sonar feature map.

[0016] S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module. Cross-modal long-range dependencies are established through a multi-head scaling dot product attention mechanism. A parallel dual-gating structure is used to adaptively filter and fuse dual-modal complementary features, and an enhanced fused feature map is output.

[0017] S500, the enhanced fusion feature map is sent to the detection neck for multi-scale feature aggregation, and then the target classification and bounding box regression are completed through the detection head to obtain the final underwater target detection result.

[0018] On the other hand, embodiments of the present invention provide an underwater image target detection system based on multimodal fusion, comprising:

[0019] At least one processor;

[0020] At least one memory for storing at least one program;

[0021] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0022] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the above-described method.

[0023] The embodiments of the present invention have the following beneficial effects:

[0024] The adaptive feature alignment module (MFAM) proposed in this invention innovatively integrates attention mechanism and confidence mask, which can dynamically estimate cross-modal spatial offset without relying on physical calibration and generate explicit alignment prior information. It effectively solves the spatial misalignment problem of acoustic-optical image pairs in underwater dynamic environments, and the alignment accuracy and robustness are significantly better than traditional weak alignment methods.

[0025] The cross-modal feature fusion module (CMFF) designed in this invention introduces multi-head scaling dot product attention and parallel dual-gating structure. It introduces sonar contour features through additive gating and suppresses optical noise through gating, thereby achieving adaptive bidirectional feature selection and differential fusion. While deeply fusing complementary acoustic and optical information, it effectively suppresses redundant noise, and improves the detection accuracy by 2.02% compared with the benchmark method.

[0026] The USOI dataset constructed in this invention fills the data gap in the field of underwater acoustic-optical fusion target detection, covering nine typical underwater targets and dynamic video data. It provides high-quality and standardized data support for algorithm verification and model training, and promotes the technological development in this field.

[0027] This invention achieves accurate matching and effective complementarity of acoustic and optical features through an end-to-end adaptive learning mechanism without requiring cross-modal physical pre-registration. This simplifies the system deployment process, enhances the real-time perception capability of AUVs in complex and variable underwater environments, and achieves industry-leading levels on both the USOI and FLIR datasets. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the underwater image target detection method based on multimodal fusion in an embodiment of the present invention.

[0030] Figure 2 This is a diagram of the overall structure of the underwater multimodal target detection network (MSOCNet) in an embodiment of the present invention, where (a) is the overall architecture, (b) is the structure of the multimodal feature alignment module (MFAM), and (c) is the structure of the crossmodal feature fusion module (CMFF).

[0031] Figure 3 This is a category display image from the USOI dataset;

[0032] Figure 4 This is a statistical chart showing the number of categories in the USOI dataset. Detailed Implementation

[0033] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0035] It is understood that the terms "first," "second," etc., used in this invention may be used to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of embodiments of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to determination," or "in the event of a determination."

[0036] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0037] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this invention is for descriptive purposes only and is not intended to limit the invention.

[0038] refer to Figure 1 ,like Figure 1 The image shown is an underwater image target detection method based on multimodal fusion provided by an embodiment of the present invention. The method includes the following steps:

[0039] S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module.

[0040] S200, the optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps;

[0041] S300: For each corresponding scale of optical feature map and sonar feature map, the multimodal feature alignment module calculates the cross-modal feature offset, and performs spatial alignment correction on the sonar feature map based on the cross-modal feature offset value to obtain the aligned sonar feature map.

[0042] Specifically, for each corresponding scale of optical feature map and sonar feature map, the optical feature map and sonar feature map are input into the multimodal feature alignment module. By jointly modeling the dual-modal features, the cross-modal feature offset that characterizes the spatial difference between the optical features and sonar features is calculated. Based on the cross-modal feature offset, the sonar feature map is spatially aligned and corrected to compensate for the spatial misalignment caused by the differences in imaging of different modalities, and the aligned sonar feature map is obtained.

[0043] S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module. Cross-modal long-range dependencies are established through a multi-head scaling dot product attention mechanism. A parallel dual-gating structure is used to adaptively filter and fuse dual-modal complementary features, and an enhanced fused feature map is output.

[0044] S500, the enhanced fusion feature map is sent to the detection neck for multi-scale feature aggregation, and then the target classification and bounding box regression are completed through the detection head to obtain the final underwater target detection result.

[0045] This invention provides an underwater image target detection method and system based on multimodal fusion. By explicitly calculating cross-modal feature offsets through a multimodal feature alignment module, it effectively solves the spatial misalignment problem caused by differences in imaging mechanisms and dynamic interference in underwater acoustic-optic images, avoiding the drawbacks of complex operation and poor adaptability of traditional physical registration methods. The cross-modal feature fusion module innovatively employs a multi-head scaling dot product attention mechanism to establish long-range cross-modal dependencies. Through a parallel dual-gating structure, it adaptively filters and fuses complementary features from both modes, fully leveraging the complementarity between long-range contour localization information provided by sonar and short-range detail information provided by optical images. This significantly suppresses redundant noise caused by modal differences and improves the quality of fused features. This invention effectively addresses core challenges such as scarce underwater datasets, severe spatial misalignment, and significant modal differences. By integrating the advantages of acoustic-optic modal information, it forms an efficient perception mode of "long-range contour locking and short-range detailed recognition," thereby significantly improving the accuracy and robustness of underwater target detection and providing more reliable technical support for tasks such as marine exploration, resource exploration, and ecological monitoring.

[0046] The core of this embodiment lies in the underwater multimodal target detection network (Multimodal Sonar-Optical Collaboration Net, MSOCNet) network framework, which achieves acoustic-optical modal information coordination through adaptive alignment and dual-gating fusion.

[0047] Data Acquisition and Network Preparation: Acquire underwater optical images and corresponding sonar images to form image pairs. Load a pre-trained underwater multimodal target detection network.

[0048] like Figure 2 As shown in (a), our proposed method is a two-branch backbone network specifically tailored for extracting features from underwater optical sonar image pairs. The network comprises:

[0049] Dual-branch backbone network: Optical and sonar features are extracted using ResNet structures respectively;

[0050] Multimodal Feature Alignment Module (MFAM): Enables cross-modal space alignment;

[0051] Cross-modal feature fusion module (CMFF): Enables deep feature fusion and enhancement;

[0052] Neck detection: FPN and PANet structures are used;

[0053] Detect the head: Perform classification and regression tasks;

[0054] This method mainly consists of three stages: unimodal feature extraction, bimodal feature alignment and fusion, and neck detection. Unimodal feature extraction can be expressed by the equation:

[0055] , ;

[0056] in , They represent from the first layer( The feature mapping of the optical and sonar branches of ). , and These represent the height, width, and number of channels of the feature map, respectively. , This represents the input optical and sonar images. The feature extraction functions for the optical and sonar branches are respectively, and their parameters are as follows: and In this invention, we use ResNet as... The function utilizes multi-scale features to capture targets of different sizes.

[0057] Secondly, given the feature map and Due to the significant differences between underwater optical imaging and forward-looking sonar imaging mechanisms, acousto-optic image pairs suffer from spatial and textural inconsistencies. Therefore, in multispectral target detection, cross-mode feature alignment is required first, followed by cross-mode feature fusion to aggregate features from different branches. This can be defined by the following formula:

[0058] ;

[0059] in, Indicates the first Fusion features in layers Indicates having parameters Feature fusion function, Indicates having parameters The feature alignment function. Existing research has explored different fusion structures and verified that intermediate fusion outperforms other fusion methods. We use intermediate fusion as the default setting, with fusion originating from convolutional layers. The multimodal characteristics of the feature. In this invention, a cross-modal feature alignment module is proposed to simulate this. And an adaptive attention feature fusion module to simulate .

[0060] Finally, from The feature maps are fed into the detector neck for multi-scale feature fusion, and then delivered to the detector head for subsequent classification and regression.

[0061] ;

[0062] in, and Indicates having parameters Multi-scale feature aggregation and detection head function; , These represent the classification probability of the target and the bounding box coordinates, respectively. FPN and PANET are commonly used as... Functions are used to enhance the semantic representation and localization capabilities of features, while Then it undertakes classification and parameterization. The enclosing box returns to its role.

[0063] In some embodiments, S300, calculating the cross-modal feature offset through the multimodal feature alignment module includes:

[0064] S310, the optical feature map and sonar feature map are mapped to a unified query-key-value space through three sets of independent linear projections, and the optical feature sequence and sonar feature sequence are output.

[0065] Specifically, the optical feature map and the sonar feature map are input into the cross-modal feature fusion module. The optical feature map and the sonar feature map are transformed by three sets of independent linear mapping units set in the cross-modal feature fusion module, and mapped to a unified query vector space, key vector space and value vector space. The optical feature sequence and the sonar feature sequence are output. The optical feature map is used to generate the query vector, and the sonar feature map is used to generate the key vector and the value vector.

[0066] S320: Calculate the token-by-token similarity matrix between the optical query and the sonar key. Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix. Apply the alignment weight matrix to the value vector generated by the sonar features to obtain the sonar features after preliminary alignment.

[0067] S330, calculate the difference between the initially aligned sonar features and the original sonar feature sequence as the difference residual, input the difference residual into a two-layer feedforward network through the residual connection for offset error correction, and multiply it by the adaptive learning scale factor to generate the initial aligned feature offset.

[0068] S340, the optical feature map and the sonar feature map are stitched together and then input into a two-layer fully connected multilayer perceptron for feature refinement. A confidence mask is output, and the preliminary aligned feature offset is multiplied element by element with the confidence mask to obtain the cross-modal feature offset.

[0069] Specifically, the optical feature map and the sonar feature map are concatenated along the feature dimension to obtain a fused feature sequence. The fused feature sequence is then input into a two-layer fully connected multilayer perceptron network set in the cross-modal feature fusion module. The fused features are refined through nonlinear feature transformation, and a confidence mask representing the reliability of the cross-modal features is output. The initial alignment feature offset is multiplied element-wise with the confidence mask to obtain the cross-modal feature offset.

[0070] This embodiment uses calculated cross-modal feature offsets to precisely adjust the spatial position of the original sonar feature map, effectively compensating for the spatial misalignment of the acousto-optic images caused by differences in imaging principles, thus laying an accurate spatial foundation for subsequent feature fusion. Specifically, after obtaining the cross-modal feature offsets, they are applied to each feature point of the sonar feature map. Geometric transformations are performed on the sonar feature map using interpolation and other methods, ensuring that the target region in the sonar feature map maintains a high degree of spatial consistency with the corresponding target region in the optical feature map. This refined alignment operation avoids feature conflicts and information interference that may occur when unaligned features are directly fused, significantly improving the spatial consistency of the dual-modal features. This ensures that the subsequent fusion module can more effectively extract and utilize complementary information from the two modalities.

[0071] The multimodal feature alignment module is described below:

[0072] Due to the differences in imaging mechanisms between acoustic and optical sensors, images captured by them in shared scenes often exhibit significant spatial geometric misalignment. Unlike LiDAR, which can achieve precise alignment using 3D point cloud computing projection matrices, acousto-optic images lack three-dimensional spatial correlation cues, making traditional alignment methods difficult to apply. Existing solutions mostly rely solely on attention mechanisms to dynamically identify cross-channel feature correspondences. To improve the quality of feature alignment, the proposed cross-modal feature alignment module MFAM innovatively integrates an attention mechanism and a confidence mask, with the structure as follows: Figure 2 As shown in (b). For clarity, this embodiment only illustrates the branch of sonar-enhanced optical image features. In the feature alignment stage, we first map the optical features and sonar features to a unified query-key-value space via three sets of independent linear projections, expressed by the formula:

[0073] , , ;

[0074] in, , These represent the input optical and sonar characteristics, respectively. , , These correspond to the height, width, and number of channels of the feature map, respectively. , , This is the parameter matrix that can be adaptively learned during network training.

[0075] In some embodiments, in S320, the step of calculating the token-by-token similarity matrix between the optical query and the sonar key, performing Softmax normalization on the similarity matrix to obtain an alignment weight matrix, and applying the alignment weight matrix to the value vector generated by the sonar features to obtain the preliminarily aligned sonar features includes:

[0076] S321, the optical feature sequence is used as the query vector Q, and the sonar feature sequence is used as the key vector K and the value vector V;

[0077] S322, by performing matrix multiplication on the transpose of the query vector Q and the key vector K, and then dividing by the scaling factor, the token-by-token similarity matrix between the optical query and the sonar key is obtained.

[0078] S323, Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix;

[0079] S324, Perform matrix multiplication on the alignment weight matrix and the value vector V to obtain the sonar features after preliminary alignment.

[0080] In this embodiment, an attention mechanism is used to capture the fine-grained correspondence between optical and sonar features, thereby achieving preliminary feature alignment. Specifically, the optical feature sequence is regarded as a query vector Q, and the sonar feature sequence is regarded as a key vector K and a value vector V, respectively. By calculating the matrix multiplication of the transpose of Q and K and dividing by a scaling factor (usually the square root of the key vector dimension to avoid gradient vanishing or exploding), the resulting similarity matrix reflects the correlation strength between each optical feature token and each sonar feature token. After performing Softmax normalization on this similarity matrix, the resulting alignment weight matrix can effectively highlight the sonar feature regions most relevant to the optical features. Multiplying this alignment weight matrix by the value vector V is actually a weighted summation of the sonar features, making the pre-aligned sonar features semantically and spatially closer to the optical features, laying the foundation for subsequent more accurate offset calculation and feature fusion. This attention-based preliminary alignment method can adaptively focus on the key corresponding parts of cross-modal features, effectively alleviating the problem of initial feature misalignment caused by differences in imaging principles.

[0081] This embodiment calculates the token-by-token similarity matrix between the optical query and the sonar key to construct the association mapping between cross-modal features. Then, the similarity matrix is ​​subjected to Softmax normalization to obtain an attention matrix that characterizes the inter-modal feature dependency weights. Finally, this attention matrix is ​​applied to the value vector generated by the sonar features to output sonar features aligned with the optical feature space. The mathematical expression of the above process is as follows:

[0082] ; ;

[0083] in, This represents the alignment weight matrix used to measure the strength of feature associations between modalities. The superscript T indicates the matrix transpose operation. To alleviate the excessively large scaling factor in similarity calculations, These are the sonar features after initial alignment via the attention mechanism.

[0084] In some embodiments, S400, the step of inputting the optical feature map and the aligned sonar feature map into the cross-modal feature fusion module, establishing cross-modal long-range dependencies through a multi-head scaling dot product attention mechanism, and adaptively filtering and fusing bimodal complementary features using a parallel dual-gating structure to output an enhanced fused feature map includes:

[0085] S410, the optical feature map and the aligned sonar feature map are flattened into a sequence of labels, and a learnable position embedding is added to each label to obtain a label sequence containing position information; the position embedding is a trainable parameter with the same dimension as the number of channels in the label sequence;

[0086] S420, the labeled sequence is injected with cross-modal feature offsets to encode the spatial-channel differences between modalities into the attention calculation process, generating multi-head attention input features;

[0087] S430, Perform a multi-head scaling dot product attention operation on the multi-head attention input features to construct long-range dependencies between cross-modal features and generate attention features containing cross-modal association information; The number of heads in the multi-head scaling dot product attention operation is h, and the dimension of a single attention head is d, satisfying hd=C, where C is the number of channels in the label sequence;

[0088] S440, a parallel dual-gating structure is used to filter and fuse the attention features to obtain fused cross-modal features;

[0089] S450, the fused cross-modal features are sequentially subjected to residual connection, Dropout processing and feedforward network operation, and then the feature vector is reprojected onto the original space through nonlinear transformation and added to the input label sequence to output an enhanced fused feature map.

[0090] To further optimize alignment accuracy, this embodiment inputs the difference between the initially aligned features and the original sonar features into a two-layer feedforward network via residual connections to finely correct the offset error. The corrected output is then multiplied by the scaling factor learned adaptively during network training to finally generate the initial offset for precise alignment. The mathematical expression for the above process is as follows:

[0091] , ;

[0092] in, This represents the residual difference between the aligned sonar features and the original sonar features. Scaling factor for network learning This represents a two-layer linear feedforward network. For initial alignment of feature offsets, It's a batch. It's tokens. It refers to Channels. To filter out unreliable offset information, optical and sonar features are concatenated and then input into a two-layer fully connected multilayer perceptron (MLP) for feature refinement. The output is a dimension-matched confidence mask used to quantify the reliability of the offset. Finally, the initial offset is multiplied element-wise by this confidence mask to obtain accurate cross-modal feature offsets, providing explicit feature alignment priors for subsequent cross-modal fusion modules. The mathematical expression of the above operations is as follows:

[0093] , ;

[0094] in, This represents a confidence mask used to filter reliable offset information. To achieve a two-layer fully connected multilayer perceptron with refined features, This indicates element-wise multiplication. It is the cross-modal feature offset obtained after final alignment after confidence mask filtering.

[0095] The cross-modal feature fusion module is described below:

[0096] Unlike pixel-intensive prediction tasks such as semantic segmentation and salient object detection, object detection focuses more on the feature representation of the target. Therefore, the core of improving the performance of multimodal object detection networks lies in efficiently fusing target features from different modalities. Although explicit alignment information between different modalities is obtained after processing by the feature alignment module, the feature representation of the same target still differs significantly between the two modalities due to the differences in acousto-optic imaging mechanisms. Directly concatenating or adding elements to the features of the two modalities can easily introduce redundant noise, which in turn restricts the inference efficiency and detection accuracy of the detector. To solve this problem, this invention designs a cross-modal feature fusion module (CMFF), which enables a single modality to efficiently learn more complementary information from the auxiliary modality, effectively mining the complementary correlation between optical and sonar modalities, balancing detection efficiency and accuracy. The structure is as follows: Figure 2 As shown in (c).

[0097] Given input feature map First, the two types of feature maps are flattened into sequence-like labels, and a learnable position embedding is added to each label. This is a dimensional... The trainable parameters are used to encode the spatial relationships between different tags. After this processing, a tag sequence containing location information is obtained. This serves as input to the CMFF module. Furthermore, offsets explicitly estimated by the cross-modal alignment module are injected, encoding the spatial-channel differences between modalities into the attention computation process.

[0098] In some embodiments, S420, injecting the labeled sequence into cross-modal feature offsets, encoding inter-modal spatial-channel differences into the attention calculation process, and generating multi-head attention input features includes:

[0099] S421, the labeled sequence containing location information is mapped into a query matrix, a key matrix, and a value matrix through a linear layer;

[0100] S422, adjust the dimensions of the cross-modal feature offsets to match the spatial dimensions of the query matrix and the key matrix;

[0101] S423, the adjusted offset is added element-by-element to the query matrix and the key matrix respectively to obtain the query matrix and key matrix incorporating spatial-channel difference information.

[0102] This embodiment effectively integrates explicit spatial alignment prior information, such as cross-modal feature offsets, into the core computation of the attention mechanism. By pre-considering the spatial misalignment between the two modal features, the attention mechanism is guided to more accurately locate the correspondence between cross-modal features, avoiding attention weight allocation deviations caused by spatial-channel differences between modalities. This provides a more accurate input feature foundation for constructing long-range cross-modal dependencies in subsequent multi-head scaling dot product attention operations.

[0103] The specific formula is as follows:

[0104] ;

[0105] ;

[0106] in, , , These are the learnable parameter matrices for query, key, and value space mappings, respectively. , , , These are the multi-head query matrix, multi-head key matrix, and multi-head value matrix, respectively, after being split by the Reshape operation. This indicates the cross-modal feature offset. Dimensionally reshaped according to the number of multi-head attention heads h, so as to align with the multi-head key matrix. Matching the spatial dimensions so as to subsequently... Element-wise addition is performed to encode the spatial-channel difference information between modes into the key matrix. This means reshaping the cross-modal feature offset O according to the number of multi-head attention heads h, so that it is aligned with the multi-head value matrix. Matching the spatial dimensions, and thus with Element-wise addition is performed to synchronously encode the spatial-channel differences between modalities into the value matrix. This method ensures that in subsequent multi-head scaling dot product attention calculations, the similarity calculation between queries and keys / values ​​fully considers the inherent spatial misalignment and channel characteristic differences between modalities, thereby more accurately capturing the correspondence between cross-modal features. , These represent the multihead key matrix and multihead value matrix, respectively, after being reshaped by the cross-modal feature offset O and incorporating intermodal spatial-channel difference information.

[0107] In some embodiments, in S430, performing a multi-head scaled dot product attention operation on the multi-head attention input features to generate attention features containing cross-modal association information, and using a parallel dual-gating structure to filter and fuse the attention features to obtain fused cross-modal features, includes:

[0108] S431, Scaling dot product attention calculation is performed on the query matrix and key matrix that incorporate spatial-channel difference information to obtain the attention matrix;

[0109] S432, multiply the attention matrix with the value matrix that incorporates spatial-channel difference information to obtain the output features of a single attention head, and concatenate the output features of all attention heads in the channel dimension to generate attention features containing cross-modal correlation information;

[0110] S433: The optical feature sequence and the fused feature are concatenated in the channel dimension and then input into the additive gating and the suppression gating, respectively. The additive gating and the suppression gating are both implemented through two fully connected layers combined with the Gaussian error linear unit activation function to achieve nonlinear transformation, and the additive gating activation output and the suppression gating activation output are generated by the Sigmoid activation function.

[0111] S434: The additive gated activation output is multiplied element-wise with the attention features to obtain the sonar complementary features enhanced by additive gated filtering; the suppression gated activation output is inverted and multiplied element-wise with the optical feature sequence to obtain the optical features after suppression gated noise reduction.

[0112] S435, the sonar complementary features and optical features are added element by element to obtain the fused cross-modal features.

[0113] This embodiment employs multi-head scaling dot product attention to construct long-range dependencies between cross-modal features, fully mining the correlation information of heterogeneous modalities. To avoid the problem of single-modal features being overwhelmed in unidirectional fusion, the module is designed with a parallel dual-gating structure. Additive gating introduces complementary features from the sonar modality through adaptive weight adjustment, enhancing key target information. Suppression gating specifically filters out potential noise from the optical modality. The two work synergistically to achieve fine-grained enhancement of target features and protection of boundary information. The mathematical expression of the above process is as follows:

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] in, The attention matrix calculated for multi-head attention is used to measure the correlation strength between feature labels of different modalities; The dimension of a single attention head; This represents the fused feature obtained by concatenating the features output by multi-head attention; For the activation output of additive gating, Represents the Sigmoid activation function. Indicates the optical feature sequence With fusion features Perform splicing along the channel dimension; To suppress the activation output of the gating, The parameter matrix is ​​for additive gating. The parameter matrix for suppression gating is used to implement the nonlinear transformation, and its expression is: , , These are the weight matrices for the two fully connected layers. , For the corresponding bias term, GELU is the activation function of the Gaussian error linear unit; This indicates the operation of splicing the features of each head output from the multi-head attention in the channel dimension.

[0119] In some embodiments, in S440, the step of sequentially performing residual connection, Dropout processing, and feedforward network operation on the fused cross-modal features, and then reprojecting the feature vector back to the original space through nonlinear transformation and adding it to the input label sequence to output an enhanced fused feature map includes:

[0120] S441 uses the activation output of additive gating to multiply the fused features element-wise, and simultaneously uses the activation output of suppression gating to multiply the optical feature sequence element-wise. The former result is subtracted from the latter result, and then added to the optical feature sequence before performing a Dropout operation to obtain the intermediate enhanced optical features.

[0121] S442, the intermediate enhanced optical feature is mapped to the original space through the output weight matrix, and then multiplied by the original optical feature sequence by the learnable weight parameters and added to obtain the feature vector after projection and residual fusion;

[0122] S443, perform feedforward network operation on the feature vector after projection and residual fusion, multiply the operation result by the feature vector by the learnable weight parameters and add them together to output the enhanced fused feature map.

[0123] After processing with residual connections and a Dropout feedforward network, the enhanced optical features are obtained. Then, the vector is transformed through nonlinear transformation. The sequence is reprojected back to the original space and added to the input sequence via a residual connection, as shown in the following formula:

[0124] ;

[0125] ;

[0126] ;

[0127] in, This is an intermediate enhanced optical feature obtained through the combined effect of additive gating and suppression gating. This indicates element-wise multiplication, and Drop(·) is the Dropout operation used to prevent overfitting of the model. The feature vector is obtained after projection and residual fusion, and α and β are learnable weight parameters used to dynamically balance the contribution ratio of enhanced features and original optical features. This is the output weight matrix before the FFN layer. , used to map feature dimensions back to the original space; This is the enhanced fused feature map after processing by a feedforward network (FFN) and then reconnecting with residuals. γ and δ are learnable weight parameters used to adjust the fusion ratio between the feedforward network output features and the residual features. α, β, γ, and δ are initialized to 1 during training. FFN(·) represents the feedforward network operation, which further enhances the expressive power of the features through two layers of linear transformation and nonlinear activation functions.

[0128] This embodiment achieves precise screening and dynamic fusion of cross-modal features through a parallel dual-gating structure. Additive gating learns the nonlinear mapping relationship between the optical feature sequence and the concatenated fused features, generating activation weights in the 0-1 range to adaptively enhance complementary features beneficial for target detection in the sonar mode, such as target contour information that sonar signals may retain in low-light or complex water environments. Suppression gating, on the other hand, targets redundant information in the optical features, such as water scattering noise and bubble interference, by generating a suppression mask through inversion, selectively denoising the optical features while preserving their clear texture details and color information. The two gating mechanisms work synergistically, fully utilizing the complementary information of the optical and sonar modes to effectively suppress noise interference. The final result is a fused cross-modal feature that contains key multimodal information while effectively suppressing noise interference, significantly improving the discriminativeness and robustness of the target feature representation and providing high-quality feature input for subsequent target detection tasks.

[0129] Dataset creation:

[0130] Currently, there is a lack of dedicated datasets in the field of underwater acoustic-optical fusion target detection, which greatly limits the technological development and performance verification in this area. Therefore, this study constructs a benchmark dataset to provide standardized data support for multimodal target detection tasks of autonomous underwater vehicles.

[0131] In the data acquisition phase, our underwater emergency rescue ROV platform is equipped with the following hardware: a BlueView M900 2D forward-looking sonar mounted on top of the AUV, featuring a 130° ultra-wide field of view, a high resolution of 6.25cm, and a maximum detection range of 100m, efficiently acquiring acoustic feature information of underwater targets; and a ZED2 underwater binocular optical camera mounted below the platform, supporting 1280×800 and 640×400 resolution outputs, and capable of capturing underwater optical images at a real-time frame rate of 60fps, ensuring the clarity and timeliness of visual information. All data was collected on-site in Sanya, China. Before collection, the sonar and camera underwent rigorous spatial and parameter calibration. During the collection process, the ROV was controlled to capture images of the target area from different angles and distances, while simultaneously storing acoustic and optical images of the same scene, ensuring precise alignment of the two types of data in the time dimension, laying a foundation for data consistency for subsequent fusion tasks.

[0132] To meet the application needs of AUVs in key practical tasks such as fish swarm analysis and underwater exploration, our dataset is comprehensive in target coverage, containing nine typical underwater targets: cages, frames, hooks, anchors, tires, underwater vehicles (rov), plastic buckets, fish, and oil drums. It covers two core categories: underwater organisms and stationary objects. Figure 3 As shown. It is worth noting that, in order to expand the application scenarios of the dataset, we additionally conducted continuous frame video shooting for two types of dynamic targets: fish and underwater vehicles. The obtained video data can be directly used for algorithm training and performance evaluation in the field of underwater target tracking.

[0133] In the post-processing stage, we removed redundant images from repetitive scenes and performed secondary verification of the time synchronization accuracy of the audio-visual data, ultimately selecting 4,000 valid audio-visual image pairs. Our dataset not only enriches the data resource library in the field of audio-visual fusion target detection, but is also the first benchmark dataset for underwater multimodal target detection and tracking that simultaneously covers underwater organisms and stationary objects, and includes continuous video data of dynamic targets. It can provide rich and reliable data support for research related to underwater intelligent sensing.

[0134] Experiments and Results:

[0135] Our method is built on the PyTorch 2.6.0 framework. The experimental environment is deployed on an Ubuntu 20.04 server with a hardware configuration of CPU R9-7900X, 64GB RAM, and NVIDIA RTX 4090D 24G GPU, which meets the computational requirements during model training. During training, the batch size is set to 25 epochs, the optimizer is SGD, the initial learning rate is 1.0×10⁻², the momentum parameter is 0.937, and a weight decay factor of 0.0005 is introduced to suppress overfitting. Cosine annealing is used as the learning rate decay strategy. The image input size is uniformly adjusted to 640×640, and mosaic data augmentation and random flipping operations are used to improve the model's adaptability to different scene features.

[0136] To fully validate the model's detection performance, we selected the representative FLIR dataset for testing, building upon the USOI dataset. The following details the sample size, partitioning method, and target categories of the two datasets:

[0137] (1) USOI Dataset: After labeling these 4000 image pairs, we allocated the images to the training, validation, and test sets in a 7:2:1 ratio, resulting in 2800 training images, 800 validation images, and 400 test images. The distribution of different target types in the dataset is as follows: Figure 4 As shown.

[0138] (2) FLIR Dataset: FLIR is a challenging multispectral object detection dataset, including both daytime and nighttime scenes. We selected the aligned version of FLIR, which contains 5142 aligned multispectral image pairs. These were divided into training, validation, and test sets in a 7:2:1 ratio, resulting in 3599 training images, 1029 validation images, and 514 test images. It contains four object classes: people, cars, dogs, and bicycles.

[0139] In our experiments, we used the mean precision (mAP) metric with an IoU of 0.5 to evaluate the performance of the multimodal object detector. Specifically, we examined the mean precision (AP) for each individual class and the overall mAP for all classes. AP is derived from the recall-precision (P–R) curve in object detection.

[0140] Sonar-enhanced optics experiment:

[0141] Objective Comparison: In the quantitative analysis experiments, we chose to compare our methods with multimodal feature fusion networks designed for other scenarios and current mainstream single-modal detectors. To ensure a fair comparison of different multimodal feature fusion methods, we did not use the decoders of these methods during the comparative experiments; instead, we applied their proposed multimodal feature fusion techniques to the target detection task. Specifically, we selected CMX, CMOT, and MFUR, suitable for RGB-D multimodal tasks, and EAEFNet, DEYOLO, BDLFusion, and Crossfuse, suitable for RGB-T tasks, to achieve feature fusion of acoustic and optical images. We also used YOLOv12 and YOLOv13 for performance evaluation in optical single-modal detection tasks. To ensure the fairness of the comparison, all multimodal fusion methods were implemented on a unified benchmark detection network, ICAFusion. This network consists of three core parts: two backbone networks for extracting different modal features, a neck for multi-scale feature fusion, and a detection head for optimizing the detection results.

[0142] We used sonar feature information to enhance optical features for target detection, and the objective evaluation results of each comparative method are shown in Table 1. The results clearly show that multimodal fusion methods, represented by CMX and EAEFNet, rely on aligned images for cross-modal feature fusion. Although these methods outperform the benchmark network ICAFSion in fusing underwater acoustic and optical modal features, the inherent heterogeneity and severe misalignment of acousto-optic images hinder the capture of intermodal positional dependencies during the fusion process and introduce redundant features. In contrast, the method proposed in this invention automatically identifies intermodal feature dependencies through the MFAM module, then integrates the aligned feature offsets, and achieves deep fusion of acoustic and optical complementary features through the CMFF module. Ultimately, MSOCNet outperforms other methods in detection performance across most categories, with an average accuracy 2.02% higher than the benchmark method and an average detection accuracy 1.08% higher than the second-ranked Crossfuse. Furthermore, compared to the two single-modal benchmark models YOLOv12 and YOLOv13, MSOCNet improves the average accuracy by 2.73% and 3.09% respectively by effectively fusing sonar modal features, breaking the information limitations of a single modality, realizing accurate information synergy between sonar and optical images, and thus achieving better detection results.

[0143] Table 1: Comparison Results of Target Detection Methods Based on Enhanced Optical Features on the USOI Dataset

[0144]

[0145] Table 1 shows that MSOCNet achieves an mAP of 95.64% in the enhanced optical feature detection task, which is 2.02% higher than the benchmark ICAFusion and 2.73% higher than YOLOv12l. In difficult detection categories such as fish and underwater vehicles (rov), the APs reach 89.23% and 88.74% respectively, significantly outperforming the comparison methods.

[0146] Optical Enhanced Sonar Experiment:

[0147] Objective comparison: In the quantitative analysis of the optically enhanced sonar experiment, based on the unified ICAFSion detection framework, we also compared the detection results of multimodal feature fusion methods CMOT, DEYOLO, MFUR, CMX, Crossfuse, BDLFusion, EAEFNet, and single-modal models YOLOv12 and YOLOv13 in the sonar image task. The experimental results are shown in Table 2.

[0148] As shown in Table 2, the two multimodal fusion methods, represented by EAEFNet and CMX, still maintain high mAP in sonar-dominated scenarios, reaching 84.34% and 84.58% respectively. However, their feature interactions are based on the ideal assumption of quasi-synchronous and aligned features. When sonar images exhibit severe speckle, projection distortion, or spatial offset from optical images, these methods often misclassify optical edges as real targets, leading to missed detections and false alarms for weak or highly reflective targets such as fish and oil drums. In contrast, our method first mines the consistency of sonar-optical local structures through the MFAM module, and then uses the CMFF module to deeply fuse optical complementary features into sonar features. Thanks to this, MSOCNet improves the average accuracy of the Oildrums category by about 10% compared to the baseline model, driving the overall mAP to 85.82%, which is 1.24 percentage points higher than the second-best CMX. This verifies that introducing optical information enhancement in the sonar dominant mode can still suppress heterogeneous noise and achieve more robust target detection. Furthermore, single-modal models still lag behind multimodal methods overall. YOLOv12 and YOLOv13 are close to the benchmark in general categories, but they significantly degrade in targets with strong modal complementarity, such as fish, with average accuracy decreasing by 4.25% and 7.41% respectively compared to the benchmark. This phenomenon clearly reveals the information loss problem of single-modal models in complex noisy environments, and further highlights the technical advantage of MSOCNet in effectively compensating for the shortcomings of single modality and improving detection robustness through cross-modal information collaboration.

[0149] Table 2: Comparison Results of Target Detection Methods Based on Enhanced Sonar Features on the USOI Dataset

[0150]

[0151] Ablation experiment:

[0152] To verify the effectiveness of the Multimodal Feature Alignment (MFAM) module and the Cross-modal Feature Fusion (CMFF) module, this invention uses the USOI dataset as the experimental platform and selects ICAFSion as the benchmark detection framework for ablation experiments. During the experiments, by progressively introducing the modules to be verified and comparing the average accuracy and parameter count of the model under different module combinations, the impact of each module on the model's target detection performance and model complexity was quantitatively and qualitatively analyzed. The experimental data are presented through the quantitative indicators in Table 3, which, from both intuitive effects and numerical dimensions, jointly demonstrate the value of progressive module optimization.

[0153] As shown in Table 3, the performance improvement exhibits a clear progressive relationship. Using only the ICAFSion benchmark framework, the model's mAP is 93.62%. Introducing the MFAM module alone improves the mAP by 1.06 percentage points compared to the benchmark due to reduced cross-modal alignment errors. Introducing the CMFF module alone further improves the mAP to 95.13% thanks to the initial mitigation of feature redundancy interference. When MFAM and CMFF work synergistically—MFAM provides accurate cross-modal alignment priors, and CMFF efficiently aggregates complementary features—the model's mAP ultimately reaches 95.64%, an improvement of 2.02 percentage points compared to the benchmark framework, and the performance gain is significantly better than the effect of introducing a single module. It is worth noting that the parameter data in Table 3 also shows that even when MSOCNet simultaneously enables feature alignment and fusion modules, its parameter count is still much lower than the ICAFSion benchmark model, achieving model efficiency optimization while improving detection accuracy.

[0154] In summary, the MFAM module effectively solves the spatial misalignment problem of underwater acousto-optic modes, laying a precise alignment foundation for subsequent feature fusion. The CMFF module can specifically filter redundant noise and enhance the mining of complementary information between modes. The combination of the two is the key to achieving a performance leap in target detection based on underwater acousto-optic information synergy, while also demonstrating good performance in parameter control, further proving the scientific nature and effectiveness of the module design proposed in this invention.

[0155] Table 3: Comparison Results and Parameter Analysis of Ablation Experiments

[0156]

[0157] Table 4 shows the experimental results of MSOCNet on the FLIR dataset based on objective evaluation metrics. As can be seen from the table, MSOCNet outperforms the baseline ICAFusion method, improving the average detection accuracy by 1.23%. Even compared to the DEYOLO model, which is specifically designed for RGB-T modal feature fusion, the proposed MSOCNet still improves the average detection accuracy by 0.26%. Notably, MSOCNet's detection accuracy advantage is particularly prominent in the Dog category, further demonstrating the positive role of MSOCNet in fusing and aligning multimodal land-based images.

[0158] Table 4: Experimental Results of MSOCNet on the FLIR Dataset in Objective Evaluation Metrics

[0159]

[0160] Conclusion: This invention addresses the challenges of target detection in complex underwater environments by deeply exploring a target detection paradigm based on sonar-optical modal information collaboration. It proposes a multimodal underwater image target detection network, MSOCNet, that efficiently utilizes the complementary information of acoustic and optical images. To address the spatial misalignment between acoustic and optical modal images and the significant differences in their imaging principles, a two-layer architecture of adaptive alignment and cross-modal fusion is used to achieve deep integration of multimodal features. The cross-modal feature alignment module (MFAM) combines an attention mechanism with a confidence mask to dynamically estimate the spatial offset of cross-modal features and generate explicit alignment prior information, achieving adaptive self-calibration of acoustic and optical features. The cross-modal feature fusion module (CMFF) introduces multi-head scaling dot product attention and a parallel dual-gating structure, adaptively selecting effective target regions in acoustic and optical images and performing differential fusion on the selected features to suppress redundant noise and invalid background information. MFAM and CMFF work together to form a progressive feature integration process of coarse alignment and fine fusion, efficiently mining the complementary value of acoustic and optical modalities without the need for physical pre-registration. In addition, we independently collected the first high-quality acoustic-optical image benchmark dataset USOI, which covers nine typical underwater objects, includes continuous video of dynamic targets, and has strict time synchronization. We also conducted extensive experiments on USOI and FLIR to verify the effectiveness and cross-domain robustness of MSOCNet, thus providing a new technical approach and data foundation for high-precision target detection tasks in complex aquatic environments.

[0161] This invention also provides an underwater image target detection system based on multimodal fusion, comprising:

[0162] At least one processor;

[0163] At least one memory for storing at least one program;

[0164] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0165] The content of the above method embodiments is applicable to this embodiment. The specific functions implemented in this embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. Therefore, they will not be repeated here.

[0166] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0167] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0169] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0170] This invention also provides a computer program product, including a computer program or computer instructions, which are stored in a memory. A processor of a computer device reads the computer program or computer instructions from the memory and executes the computer program or computer instructions, causing the computer device to perform the above-described method.

[0171] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0172] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A method for underwater image target detection based on multimodal fusion, characterized in that, The method includes the following steps: S100: Acquire underwater optical images, sonar image pairs, and a trained underwater multimodal target detection network. The underwater multimodal target detection network includes a dual-branch backbone network, a multimodal feature alignment module, a cross-modal feature fusion module, and a neck and head detection module. S200, The optical image and sonar image are respectively input into the dual-branch backbone network to extract multi-scale optical feature maps and sonar feature maps; S300: For each corresponding scale of optical feature map and sonar feature map, the multimodal feature alignment module calculates the cross-modal feature offset, and performs spatial alignment correction on the sonar feature map based on the cross-modal feature offset to obtain the aligned sonar feature map. S400, the optical feature map and the aligned sonar feature map are input into the cross-modal feature fusion module. Cross-modal long-range dependencies are established through a multi-head scaling dot product attention mechanism. A parallel dual-gating structure is used to adaptively filter and fuse dual-modal complementary features, and an enhanced fused feature map is output. S500, the enhanced fusion feature map is sent to the detection neck for multi-scale feature aggregation, and then the target classification and bounding box regression are completed through the detection head to obtain the final underwater target detection result; In S300, the calculation of cross-modal feature offset through the multimodal feature alignment module includes: S310, the optical feature map and sonar feature map are mapped to a unified query-key-value space through three sets of independent linear projections, and the optical feature sequence and sonar feature sequence are output. S320: Calculate the token-by-token similarity matrix between the optical query and the sonar key. Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix. Apply the alignment weight matrix to the value vector generated by the sonar features to obtain the sonar features after preliminary alignment. S330, calculate the difference between the initially aligned sonar features and the original sonar feature sequence as the difference residual, input the difference residual into a two-layer feedforward network through the residual connection for offset error correction, and multiply it by the adaptive learning scale factor to generate the initial aligned feature offset. S340, the optical feature map and the sonar feature map are stitched together and then input into a two-layer fully connected multilayer perceptron for feature refinement. A confidence mask is output, and the preliminary aligned feature offset is multiplied element by element with the confidence mask to obtain the cross-modal feature offset.

2. The method according to claim 1, characterized in that, In S320, the token-by-token similarity matrix between the optical query and the sonar key is calculated. This similarity matrix is ​​then subjected to Softmax normalization to obtain an alignment weight matrix. This alignment weight matrix is ​​applied to the value vector generated from the sonar features to obtain the preliminarily aligned sonar features, including: S321, the optical feature sequence is used as the query vector Q, and the sonar feature sequence is used as the key vector K and the value vector V; S322, by performing matrix multiplication on the transpose of the query vector Q and the key vector K, and then dividing by the scaling factor, the token-by-token similarity matrix between the optical query and the sonar key is obtained. S323, Perform Softmax normalization on the similarity matrix to obtain the alignment weight matrix; S324, Perform matrix multiplication on the alignment weight matrix and the value vector V to obtain the sonar features after preliminary alignment.

3. The method according to claim 1, characterized in that, In S400, the process of inputting the optical feature map and the aligned sonar feature map into the cross-modal feature fusion module, establishing cross-modal long-range dependencies through a multi-head scaling dot product attention mechanism, and adaptively filtering and fusing bimodal complementary features using a parallel dual-gating structure to output an enhanced fused feature map includes: S410, the optical feature map and the aligned sonar feature map are flattened into a sequence of labels, and a learnable position embedding is added to each label to obtain a label sequence containing position information; the position embedding is a trainable parameter with the same dimension as the number of channels in the label sequence; S420, the labeled sequence is injected with cross-modal feature offsets to encode the spatial-channel differences between modalities into the attention calculation process, generating multi-head attention input features; S430, perform a multi-head scaling dot product attention operation on the multi-head attention input features to generate attention features containing cross-modal association information. Use a parallel dual-gating structure to filter and fuse the attention features to obtain fused cross-modal features. The number of heads in the multi-head scaling dot product attention operation is h, and the dimension of a single attention head is d, satisfying hd=C, where C is the number of channels in the labeled sequence. S440, the fused cross-modal features are sequentially subjected to residual connection, Dropout processing and feedforward network operation, and then the feature vector is reprojected onto the original space through nonlinear transformation and added to the input label sequence to output an enhanced fused feature map.

4. The method according to claim 3, characterized in that, In S420, injecting the labeled sequence into the cross-modal feature offset, encoding the spatial-channel differences between modalities into the attention calculation process, and generating multi-head attention input features includes: S421, the labeled sequence containing location information is mapped into a query matrix, a key matrix, and a value matrix through a linear layer; S422, adjust the dimensions of the cross-modal feature offsets to match the spatial dimensions of the query matrix and the key matrix; S423, the adjusted offset is added element-by-element to the query matrix and the key matrix respectively to obtain the multi-head key matrix and multi-head value matrix incorporating spatial-channel difference information.

5. The method according to claim 3, characterized in that, In S430, the multi-head scaled dot product attention operation is performed on the multi-head attention input features to generate attention features containing cross-modal association information. A parallel dual-gating structure is then used to filter and fuse the attention features to obtain fused cross-modal features, including: S431, Scaling dot product attention calculation is performed on the query matrix and key matrix that incorporate spatial-channel difference information to obtain the attention matrix; S432, multiply the attention matrix with the value matrix that incorporates spatial-channel difference information to obtain the output features of a single attention head, and concatenate the output features of all attention heads in the channel dimension to generate attention features containing cross-modal correlation information; S433: The optical feature sequence and the fused feature are concatenated in the channel dimension and then input into the additive gating and the suppression gating, respectively. The additive gating and the suppression gating are both implemented through two fully connected layers combined with the Gaussian error linear unit activation function to achieve nonlinear transformation, and the additive gating activation output and the suppression gating activation output are generated by the Sigmoid activation function. S434: The additive gated activation output is multiplied element-wise with the attention features to obtain the sonar complementary features enhanced by additive gated filtering; the suppression gated activation output is inverted and multiplied element-wise with the optical feature sequence to obtain the optical features after suppression gated noise reduction. S435, the sonar complementary features and optical features are added element by element to obtain the fused cross-modal features.

6. The method according to claim 3, characterized in that, In S440, the fused cross-modal features are sequentially subjected to residual connections, Dropout processing, and feedforward network operations. Then, the feature vectors are reprojected onto the original space through a nonlinear transformation and added to the input label sequence to output an enhanced fused feature map, including: S441 uses the activation output of additive gating to multiply the fused features element-wise, and simultaneously uses the activation output of suppression gating to multiply the optical feature sequence element-wise. The former result is subtracted from the latter result, and then added to the optical feature sequence before performing a Dropout operation to obtain the intermediate enhanced optical features. S442, the intermediate enhanced optical feature is mapped to the original space through the output weight matrix, and then multiplied by the original optical feature sequence by the learnable weight parameters and added to obtain the feature vector after projection and residual fusion; S443, perform feedforward network operation on the feature vector after projection and residual fusion, multiply the operation result by the feature vector by the learnable weight parameters and add them together to output the enhanced fused feature map.

7. An underwater image target detection system based on multimodal fusion, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images