An underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention

Abstract

The application discloses an underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention, and comprises the following steps: acquiring underwater optical image data and acoustic image data; constructing a double-flow feature extraction network to extract low-level general features and high-level semantic features of the optical image and the acoustic image; constructing a global query space attention module and embedding the double-flow feature extraction network to fuse the low-level general features of the optical image and the low-level general features of the acoustic image and obtain a fused general feature map; introducing a cross-modal contrast learning module during training to map the high-level semantic features of the optical image and the acoustic image to a shared embedding space, calculate a contrast loss, and drive semantic alignment between the modes; performing global aggregation processing on the high-level semantic features of the optical image and the acoustic image and the fused general feature map to generate a global feature vector; and inputting the global feature vector into a multi-label classification head to output a classification prediction result of an underwater target.

Description

Technical Field

[0001] This invention belongs to the field of underwater target detection and recognition technology, and relates to an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention. Background Technology

[0002] Accurate underwater target classification is a crucial foundation for marine resource assessment, underwater infrastructure inspection, and autonomous underwater vehicle (AUV) navigation. Currently, underwater perception primarily relies on optical cameras and acoustic sensors. However, single-modal sensing has inherent limitations in complex underwater environments: optical images are rich in texture information but are susceptible to interference from water turbidity, light absorption, and scattering, leading to feature blurring and decreased classification performance; acoustic images, thanks to the strong penetrating power of sound waves, possess good environmental robustness, but their imaging mechanism results in sparse semantic information and missing textures, making it difficult to distinguish objects with similar appearances.

[0003] Fusing optical and acoustic information is an essential choice for improving classification accuracy. However, existing multimodal fusion methods face two major challenges when processing underwater optical-acoustic data: first, the representation gap—due to the vastly different imaging mechanisms, optical and acoustic images cannot achieve pixel-level physical alignment like other multimodal fusion methods such as optical-depth, resulting in poor direct fusion performance; second, asymmetric information degradation—dynamic changes in the underwater environment often lead to a sharp decline in the quality of a particular modality. If the fusion strategy cannot perceive the reliability of the modality, noise from low-quality modalities will contaminate the features of high-quality modalities, leading to overall performance degradation. Therefore, a method that can effectively align heterogeneous modal semantics and achieve robust fusion is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, the purpose of this invention is to provide an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention.

[0005] This invention provides an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, comprising:

[0006] Step 1: Acquire time-synchronized and spatially registered underwater optical and acoustic image data to form the original dataset and divide it into training and test sets;

[0007] Step 2: Construct a dual-stream feature extraction network that includes optical feature extraction branches and acoustic feature extraction branches. Input the training set into the dual-stream feature extraction network to extract low-level general features and high-level semantic features of optical images and acoustic images, respectively.

[0008] Step 3: Construct a global query spatial attention module and embed it into a two-stream feature extraction network. Use the global query spatial attention module to fuse the low-level general features of the optical image and the low-level general features of the acoustic image to obtain the fused general feature map.

[0009] Step 4: During the training phase, a cross-modal contrastive learning module is introduced to map the high-level semantic features of optical images and acoustic images to a shared embedding space. The contrastive loss function is then used to drive semantic alignment between modalities.

[0010] Step 5: Perform global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map output by the global query space attention module to generate a global feature vector.

[0011] Step 6: Input the global feature vector into the multi-label classification head and output the classification prediction results of underwater targets.

[0012] The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, as proposed in this invention, has the following beneficial effects:

[0013] The method of this invention can utilize global contextual information to guide local feature fusion, effectively addressing the asymmetric information degradation problem commonly encountered in underwater environments. When the quality of one modality is compromised, the model can be calibrated based on global information from another modality, avoiding the introduction of noise. Simultaneously, the introduced cross-modal contrastive learning mechanism effectively bridges the semantic gap between optical and acoustic images due to their different imaging mechanisms, significantly improving feature alignment quality without increasing inference computational overhead. Attached Figure Description

[0014] Figure 1 This is a flowchart of an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, according to the present invention.

[0015] Figure 2 This is a schematic diagram of the dual-stream feature extraction network of the present invention;

[0016] Figure 3 This is a schematic diagram of the global query space attention module of the present invention;

[0017] Figure 4 This is a schematic diagram of the cross-modal contrastive learning module of the present invention;

[0018] Figure 5 This is a diagram showing the confusion matrix results of an embodiment of the present invention;

[0019] Figure 6 This is the precision-recall result curve of an embodiment of the present invention. Detailed Implementation

[0020] like Figure 1 As shown, the present invention provides an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, comprising:

[0021] Step 1: Acquire time-synchronized and spatially registered underwater optical and acoustic image data to form the original dataset, which is then divided into training and test sets, specifically:

[0022] Step 1.1: Use an underwater robot platform equipped with a high-definition camera and forward-looking sonar to simultaneously collect underwater optical image data and acoustic image data to construct the original dataset.

[0023] Step 1.2: Analyze the raw data and extract the time-synchronized optical and acoustic frames.

[0024] Step 1.3: Adjust each frame of image to a uniform size and perform normalization processing.

[0025] Step 1.4: Construct training and testing sets containing multiple types of underwater targets from the preprocessed raw data.

[0026] Step 2: Construct a two-stream feature extraction network that includes optical and acoustic feature extraction branches. Input the training set into the two-stream feature extraction network to extract low-level general features and high-level semantic features from the optical and acoustic images, respectively. The network structure is as follows: Figure 2 As shown.

[0027] The optical feature extraction branch and the acoustic feature extraction branch are two parallel branches with non-shared weights; both parallel branches adopt a hybrid architecture combining convolutional neural networks and Transformers to balance local details and global feature extraction. The hybrid architecture includes convolutional layers and self-attention layers.

[0028] The optical feature extraction branch includes a low-level general optical feature extraction module and a high-level semantic optical feature extraction module; the acoustic feature extraction branch includes a low-level general acoustic feature extraction module and a high-level semantic acoustic feature extraction module. Feature maps of different resolutions are extracted respectively.

[0029] Step 3: Construct a global query space attention module and embed a two-stream feature extraction network, such as... Figure 3 As shown, the low-level general features of the optical image and the acoustic image are fused using a global query spatial attention module to obtain a fused general feature map. The specific processing steps of the global query spatial attention module include:

[0030] Step 3.1: Based on the imaging quality of the optical image data and the acoustic image data, dynamically select the low-level general features of the high-quality modality as the main modality features, and the low-level general features of the other modality as the auxiliary modality features.

[0031] Step 3.2: Input the main modality features into a 1×1 convolutional layer to obtain the query vector Q, and pass the auxiliary modality features through a 1×1 convolutional layer to obtain the key vector K and the value vector V.

[0032] Step 3.3: Perform adaptive average pooling on the query vector Q and key vector K respectively, compressing their spatial dimensions to 1×1, to obtain the main modality global context vector. and auxiliary modal global context vector .

[0033] Step 3.3: Calculate the dot product of the global context vector of the primary modality and the global context vector of the secondary modality, and obtain the global attention score using the Softmax function:

[0034]

[0035] Where S is the global attention score. This represents the number of feature channels.

[0036] Step 3.4: Perform matrix multiplication between the global attention score and the value vector V that retains complete spatial information to obtain the weighted context information:

[0037]

[0038] in, This represents a broadcast multiplication of a scalar with respect to a tensor.

[0039] Step 3.5: After processing the weighted context information through a convolutional layer, add it to the original main modality features through residual connections to output the fused general feature map.

[0040] Step 4: Introduce a cross-modal contrastive learning module into the two-stream feature extraction network, such as... Figure 4 As shown, high-level semantic features of optical and acoustic images are mapped to a shared embedding space, and semantic alignment between modalities is driven by calculating a contrastive loss function, specifically:

[0041] Step 4.1: Using a nonlinear projection head, the high-level semantic features of the optical image and the high-level semantic features of the acoustic image are mapped to a shared embedding space of the same dimension to obtain normalized optical feature vectors and acoustic feature vectors.

[0042] Step 4.2: Calculate the cosine similarity between the high-level semantic features of each optical image and the high-level semantic features of each acoustic image within the training batch, and construct a similarity matrix.

[0043] Step 4.3: Construct the InfoNCE loss function to update the network parameters by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.

[0044] The one-way InfoNCE loss function, which uses optical features as anchors and acoustic features as positive samples, is denoted as... :

[0045]

[0046] The one-way InfoNCE loss function, which uses acoustic features as anchors and optical features as positive samples, is denoted as... :

[0047]

[0048] Where N represents the total number of samples in the batch, For the normalized i-th optical eigenvector, The normalized k-th optical eigenvector, Let i be the normalized acoustic feature vector. The normalized k-th acoustic feature vector; and The i-th pair of positive samples is formed; all other 2(N-1) samples in the batch are negative samples. τ represents the cosine similarity between u and v; τ is a temperature coefficient used to control the smoothness of the distribution. For indicator functions, when The value is 1 if it is true, and 0 otherwise.

[0049] The final cross-modal contrast loss is defined as the average of the losses in the two directions, and the calculation formula is as follows:

[0050]

[0051] in, This represents the cross-modal contrast loss.

[0052] Step 5: Perform global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map output by the global query space attention module to generate a global feature vector, specifically:

[0053] Step 5.1: Perform global average pooling on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map to obtain feature vectors at multiple scales.

[0054] Step 5.2: Concatenate feature vectors from multiple scales to form a comprehensive feature representation; perform dimensionality reduction and information fusion on the concatenated vectors through a fully connected layer to generate the final global feature vector.

[0055] Step 6: Input the global feature vector into the multi-label classification head and output the classification prediction results of underwater targets.

[0056] The total loss function of the network consists of a weighted classification loss and a cross-modal contrastive loss. The parameters of the two-stream feature extraction network are updated through backpropagation, forcing the network to align optical and acoustic features in the deep semantic space. The formula for calculating the total loss function of the network is as follows:

[0057]

[0058] in, This is the sum of the binary cross-entropy losses of the optical feature extraction branch and the acoustic feature extraction branch. To determine the loss weights, set larger values ​​in the early stages of training to emphasize feature alignment, and decrease the values ​​in the later stages of training to emphasize the classification task.

[0059] As a preferred embodiment of the present invention, the optimizer may be AdamW. Loss weights A phased dynamic adjustment strategy is adopted: larger weights are set in the early stages of training (e.g., ...). =0.05) to force feature alignment; reduce weights in the later stages of training (e.g. =0.001) to focus on refined learning for classification tasks.

[0060] In the inference and classification stage, the contrastive learning branch is removed, and the test set paired with optical and acoustic images collected in the embodiment is input into the dual-stream feature extraction network. The features fused by the global query space attention module and the high-level semantic features are respectively subjected to global average pooling, concatenated and generated through a fully connected layer to generate the final global feature vector, which is then input into the multi-label classification head to output the confidence of the category.

[0061] The test set included nine types of targets: underwater unmanned vehicles, bicycles, unicycles, oil drums, pots, radiators, chairs, tires, and remotely operated vehicles. The experimental results are as follows: Figure 5 and 6 As shown: (1) Confusion matrix result diagram (corresponding to the attached diagram) Figure 5 (2) Precision-Recall Curve Analysis (corresponding to Appendix) Figure 6The precision values ​​for the vast majority of categories exceeded 0.95, and the curve remained almost horizontal within the recall range of 0 to 0.8, indicating that the model ensured no missed detections while producing very few false positives.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for underwater optical-acoustic image fusion and classification based on cross-modal contrastive learning and global attention, characterized in that, include: Step 1: Acquire time-synchronized and spatially registered underwater optical and acoustic image data to form the original dataset and divide it into training and test sets; Step 2: Construct a dual-stream feature extraction network that includes optical feature extraction branches and acoustic feature extraction branches. Input the training set into the dual-stream feature extraction network to extract low-level general features and high-level semantic features of optical images and acoustic images, respectively. Step 3: Construct a global query spatial attention module and embed it into a two-stream feature extraction network. Use the global query spatial attention module to fuse the low-level general features of the optical image and the low-level general features of the acoustic image to obtain the fused general feature map. Step 4: During the training phase, a cross-modal contrastive learning module is introduced to map the high-level semantic features of optical images and acoustic images to a shared embedding space. The contrastive loss function is then used to drive semantic alignment between modalities. Step 5: Perform global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map output by the global query space attention module to generate a global feature vector. Step 6: Input the global feature vector into the multi-label classification head and output the classification prediction results of underwater targets; The specific processing steps of the global query space attention module in step 3 include: Step 3.1: Based on the imaging quality of the optical image data and the acoustic image data, dynamically select the low-level general features of the high-quality modality as the main modality features, and the low-level general features of the other modality as the auxiliary modality features; Step 3.2: Input the main modality features into a 1×1 convolutional layer to obtain the query vector Q, and pass the auxiliary modality features through a 1×1 convolutional layer to obtain the key vector K and the value vector V; Step 3.3: Perform adaptive average pooling on the query vector Q and key vector K respectively, compressing their spatial dimensions to 1×1, to obtain the main modality global context vector. and auxiliary modal global context vector ; Step 3.4: Calculate the dot product of the global context vector of the primary modality and the global context vector of the secondary modality, and obtain the global attention score by using the Softmax function; Where S is the global attention score. The number of feature channels; Step 3.5: Perform matrix multiplication between the global attention score and the value vector V that retains complete spatial information to obtain the weighted context information; in, This represents a scalar-tensor broadcast multiplication; Step 3.6: After processing the weighted context information through a convolutional layer, add it to the original main modality features through residual connections to output the fused general feature map.

2. The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention as described in claim 1, characterized in that, Step 1 specifically involves: Step 1.1: Simultaneously acquire underwater optical and acoustic image data using an underwater robot platform equipped with a high-definition camera and forward-looking sonar to construct the original dataset; Step 1.2: Parse the raw data and extract the time-synchronized optical and acoustic frames; Step 1.3: Adjust each frame of image to a uniform size and perform normalization processing; Step 1.4: Construct training and testing sets containing multiple types of underwater targets from the preprocessed raw data.

3. The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention as described in claim 1, characterized in that, In step 2, the optical feature extraction branch and the acoustic feature extraction branch are two parallel branches with non-shared weights; both parallel branches adopt a hybrid architecture combining convolutional neural networks and Transformers, which includes convolutional layers and self-attention layers. The optical feature extraction branch includes a low-level general optical feature extraction module and a high-level optical semantic feature extraction module; the acoustic feature extraction branch includes a low-level general acoustic feature extraction module and a high-level acoustic semantic feature extraction module.

4. The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention as described in claim 1, characterized in that, Step 4 specifically involves: Step 4.1: Using a nonlinear projection head, the high-level semantic features of the optical image and the high-level semantic features of the acoustic image are mapped to a shared embedding space of the same dimension to obtain normalized optical feature vectors and acoustic feature vectors. Step 4.2: Calculate the cosine similarity between the high-level semantic features of each optical image and the high-level semantic features of each acoustic image within the training batch, and construct a similarity matrix; Step 4.3: Construct the InfoNCE loss function to update the network parameters by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs; The one-way InfoNCE loss function, which uses optical features as anchors and acoustic features as positive samples, is denoted as... : The one-way InfoNCE loss function, which uses acoustic features as anchors and optical features as positive samples, is denoted as... : Where N represents the total number of samples in the batch, For the normalized i-th optical eigenvector, The normalized k-th optical eigenvector, Let i be the normalized acoustic feature vector. The k-th acoustic feature vector is normalized. and The i-th pair of positive samples is formed; all other 2(N-1) samples in the batch are negative samples. τ represents the cosine similarity between u and v; τ is a temperature coefficient used to control the smoothness of the distribution. For indicator functions, when The value is 1 if it is true, and 0 otherwise. The final cross-modal contrast loss is defined as the average of the losses in the two directions, and the calculation formula is as follows: in, This represents the cross-modal contrast loss.

5. The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention as described in claim 1, characterized in that, Step 5 specifically involves: Step 5.1: Perform global average pooling on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map to obtain feature vectors at multiple scales. Step 5.2: Concatenate feature vectors from multiple scales to form a comprehensive feature representation; perform dimensionality reduction and information fusion on the concatenated vectors through a fully connected layer to generate the final global feature vector.

6. The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention as described in claim 4, is characterized in that... The total loss function of the network consists of a weighted classification loss and a cross-modal contrastive loss, and is calculated as follows: in, This is the sum of the binary cross-entropy losses of the optical feature extraction branch and the acoustic feature extraction branch. The hyperparameters for balancing the weights.

Images