Underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention

Abstract

The invention discloses an underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention. The method comprises the following steps: acquiring underwater optical image data and acoustic image data; constructing a double-flow feature extraction network, and extracting low-level general features and high-level semantic features of the optical image and the acoustic image; constructing a global query space attention module, embedding the global query space attention module into a double-flow feature extraction network, and fusing the low-level general features of the optical image and the low-level general features of the acoustic image to obtain a fused general feature map; during training, a cross-modal comparison learning module is introduced, high-level semantic features of an optical image and an acoustic image are mapped to a shared embedding space, and semantic alignment between loss driving modes is calculated and compared; performing global aggregation processing on the high-level semantic features of the optical image and the acoustic image and the fused general feature map to generate a global feature vector; and inputting the global feature vector into a multi-label classification head, and outputting a classification prediction result of the underwater target.

Description

Technical Field

[0001] This invention belongs to the field of underwater target detection and recognition technology, and relates to an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention. Background Technology

[0002] Accurate underwater target classification is a crucial foundation for marine resource assessment, underwater infrastructure inspection, and autonomous underwater vehicle (AUV) navigation. Currently, underwater perception primarily relies on optical cameras and acoustic sensors. However, single-modal sensing has inherent limitations in complex underwater environments: optical images are rich in texture information but are susceptible to interference from water turbidity, light absorption, and scattering, leading to feature blurring and decreased classification performance; acoustic images, thanks to the strong penetrating power of sound waves, possess good environmental robustness, but their imaging mechanism results in sparse semantic information and missing textures, making it difficult to distinguish objects with similar appearances.

[0003] Fusing optical and acoustic information is an essential choice for improving classification accuracy. However, existing multimodal fusion methods face two major challenges when processing underwater optical-acoustic data: first, the representation gap—due to the vastly different imaging mechanisms, optical and acoustic images cannot achieve pixel-level physical alignment like other multimodal fusion methods such as optical-depth, resulting in poor direct fusion performance; second, asymmetric information degradation—dynamic changes in the underwater environment often lead to a sharp decline in the quality of a particular modality. If the fusion strategy cannot perceive the reliability of the modality, noise from low-quality modalities will contaminate the features of high-quality modalities, leading to overall performance degradation. Therefore, a method that can effectively align heterogeneous modal semantics and achieve robust fusion is urgently needed. Summary of the Invention

[0004] To address the aforementioned technical problems, the purpose of this invention is to provide an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention.

[0005] This invention provides an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, comprising:

[0006] Step 1: Acquire time-synchronized and spatially registered underwater optical and acoustic image data to form the original dataset and divide it into training and test sets;

[0007] Step 2: Construct a dual-stream feature extraction network that includes optical feature extraction branches and acoustic feature extraction branches. Input the training set into the dual-stream feature extraction network to extract low-level general features and high-level semantic features of optical images and acoustic images, respectively.

[0008] Step 3: Construct a global query spatial attention module and embed it into a dual-stream feature extraction network. Use the global query spatial attention module to fuse the low-level general features of the optical image and the low-level general features of the acoustic image to obtain the fused general feature map.

[0009] Step 4: During the training phase, a cross-modal contrastive learning module is introduced to map the high-level semantic features of optical images and acoustic images to a shared embedding space. Semantic alignment between modalities is driven by calculating a contrastive loss function.

[0010] Step 5: Perform global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map output by the global query space attention module to generate a global feature vector.

[0011] Step 6: Input the global feature vector into the multi-label classification head and output the classification prediction results of underwater targets.

[0012] The underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, as proposed in this invention, has the following beneficial effects:

[0013] The method of this invention can utilize global contextual information to guide local feature fusion, effectively addressing the asymmetric information degradation problem commonly encountered in underwater environments. When the quality of one modality is compromised, the model can be calibrated based on global information from another modality, avoiding the introduction of noise. Simultaneously, the introduced cross-modal contrastive learning mechanism effectively bridges the semantic gap between optical and acoustic images due to their different imaging mechanisms, significantly improving feature alignment quality without increasing inference computational overhead. Attached Figure Description

[0014] Figure 1 This is a flowchart of an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, according to the present invention.

[0015] Figure 2 This is a schematic diagram of the dual-stream feature extraction network of the present invention;

[0016] Figure 3 This is a schematic diagram of the global query space attention module of the present invention;

[0017] Figure 4 This is a schematic diagram of the cross-modal contrastive learning module of the present invention;

[0018] Figure 5 This is a diagram showing the confusion matrix results of an embodiment of the present invention;

[0019] Figure 6 This is the precision-recall result curve of an embodiment of the present invention. Detailed Implementation

[0020] like Figure 1 As shown, the present invention provides an underwater optical-acoustic image fusion and classification method based on cross-modal contrastive learning and global attention, comprising:

[0021] Step 1: Acquire time-synchronized and spatially registered underwater optical and acoustic image data to form the original dataset, which is then divided into training and test sets, specifically:

[0022] Step 1.1: Use an underwater robot platform equipped with a high-definition camera and forward-looking sonar to simultaneously collect underwater optical image data and acoustic image data to construct the original dataset.

[0023] Step 1.2: Analyze the raw data and extract the time-synchronized optical and acoustic frames.

[0024] Step 1.3: Adjust each frame of image to a uniform size and perform normalization processing.

[0025] Step 1.4: Construct training and testing sets containing multiple types of underwater targets from the preprocessed raw data.

[0026] Step 2: Construct a two-stream feature extraction network that includes optical and acoustic feature extraction branches. Input the training set into the two-stream feature extraction network to extract low-level general features and high-level semantic features from the optical and acoustic images, respectively. The network structure is as follows: Figure 2 As shown.

[0027] The optical feature extraction branch and the acoustic feature extraction branch are two parallel branches with non-shared weights; both parallel branches adopt a hybrid architecture combining convolutional neural networks and Transformers to balance local details and global feature extraction. The hybrid architecture includes convolutional layers and self-attention layers.

[0028] The optical feature extraction branch includes a low-level general optical feature extraction module and a high-level semantic optical feature extraction module; the acoustic feature extraction branch includes a low-level general acoustic feature extraction module and a high-level semantic acoustic feature extraction module. Feature maps of different resolutions are extracted respectively.

[0029] Step 3: Construct a global query space attention module and embed a two-stream feature extraction network, such as... Figure 3 As shown, the low-level general features of the optical image and the acoustic image are fused using a global query spatial attention module to obtain a fused general feature map. The specific processing steps of the global query spatial attention module include:

[0030] Step 3.1: Based on the imaging quality of the optical image data and the acoustic image data, dynamically select the low-level general features of the high-quality modality as the main modality features, and the low-level general features of the other modality as the auxiliary modality features.

[0031] Step 3.2: Input the main modality features into a 1×1 convolutional layer to obtain the query vector Q, and pass the auxiliary modality features through a 1×1 convolutional layer to obtain the key vector K and the value vector V.

[0032] Step 3.3: Perform adaptive average pooling on the query vector Q and key vector K respectively, compressing their spatial dimensions to 1×1, to obtain the main modality global context vector. and auxiliary modal global context vector .

[0033] Step 3.3: Calculate the dot product of the global context vector of the primary modality and the global context vector of the secondary modality, and obtain the global attention score using the Softmax function:

[0034]

[0035] Where S is the global attention score. This represents the number of feature channels.

[0036] Step 3.4: Perform matrix multiplication between the global attention score and the value vector V that retains complete spatial information to obtain the weighted context information:

[0037]

[0038] in, This represents a broadcast multiplication of a scalar with respect to a tensor.

[0039] Step 3.5: After processing the weighted context information through a convolutional layer, add it to the original main modality features through residual connections to output the fused general feature map.

[0040] Step 4: Introduce a cross-modal contrastive learning module into the two-stream feature extraction network, such as... Figure 4 As shown, high-level semantic features of optical and acoustic images are mapped to a shared embedding space, and semantic alignment between modalities is driven by calculating a contrastive loss function, specifically:

[0041] Step 4.1: Using a nonlinear projection head, the high-level semantic features of the optical image and the high-level semantic features of the acoustic image are mapped to a shared embedding space of the same dimension to obtain normalized optical feature vectors and acoustic feature vectors.

[0042] Step 4.2: Calculate the cosine similarity between the high-level semantic features of each optical image and the high-level semantic features of each acoustic image within the training batch, and construct a similarity matrix.

[0043] Step 4.3: Construct the InfoNCE loss function to update the network parameters by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs.

[0044] The one-way InfoNCE loss function, which uses optical features as anchors and acoustic features as positive samples, is denoted as... :

[0045]

[0046] The one-way InfoNCE loss function, which uses acoustic features as anchors and optical features as positive samples, is denoted as... :

[0047]

[0048] Where N represents the total number of samples in the batch, For the normalized i-th optical eigenvector, The normalized k-th optical eigenvector, Let i be the normalized acoustic feature vector. The k-th acoustic feature vector is normalized. and The i-th pair of positive samples is formed; all other 2(N-1) samples in the batch are negative samples. τ represents the cosine similarity between u and v; τ is a temperature coefficient used to control the smoothness of the distribution. For indicator functions, when The value is 1 if it is true, and 0 otherwise.

[0049] The final cross-modal contrast loss is defined as the average of the losses in the two directions, and the calculation formula is as follows:

[0050]

[0051] in, This represents the cross-modal contrast loss.

[0052] Step 5: Perform global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map output by the global query space attention module to generate a global feature vector, specifically:

[0053] Step 5.1: Perform global average pooling on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image, and the fused general feature map to obtain feature vectors at multiple scales.

[0054] Step 5.2: Concatenate feature vectors from multiple scales to form a comprehensive feature representation; perform dimensionality reduction and information fusion on the concatenated vectors through a fully connected layer to generate the final global feature vector.

[0055] Step 6: Input the global feature vector into the multi-label classification head and output the classification prediction results of underwater targets.

[0056] The total loss function of the network consists of a weighted classification loss and a cross-modal contrastive loss. The parameters of the two-stream feature extraction network are updated through backpropagation, forcing the network to align optical and acoustic features in the deep semantic space. The formula for calculating the total loss function of the network is as follows:

[0057]

[0058] in, This is the sum of the binary cross-entropy losses of the optical feature extraction branch and the acoustic feature extraction branch. To determine the loss weights, set larger values ​​in the early stages of training to emphasize feature alignment, and decrease the values ​​in the later stages of training to emphasize the classification task.

[0059] As a preferred embodiment of the present invention, the optimizer may be AdamW. Loss weights A phased dynamic adjustment strategy is adopted: larger weights are set in the early stages of training (e.g., ...). =0.05) to force feature alignment; reduce weights in the later stages of training (e.g. =0.001) to focus on refined learning for classification tasks.

[0060] In the inference and classification stage, the contrastive learning branch is removed, and the test set paired with optical and acoustic images collected in the embodiment is input into the dual-stream feature extraction network. The features fused by the global query space attention module and the high-level semantic features are respectively subjected to global average pooling, concatenated and generated through a fully connected layer to generate the final global feature vector, which is then input into the multi-label classification head to output the confidence of the category.

[0061] The test set included nine types of targets: underwater unmanned vehicles, bicycles, unicycles, oil drums, pots, radiators, chairs, tires, and remotely operated vehicles. The experimental results are as follows: Figure 5 and 6 As shown: (1) Confusion matrix result diagram (corresponding to the attached diagram) Figure 5 (2) Precision-Recall Curve Analysis (corresponding to Appendix) Figure 6The precision values ​​for the vast majority of categories exceeded 0.95, and the curve remained almost horizontal within the recall range of 0 to 0.8, indicating that the model ensured no missed detections while producing very few false positives.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention, characterized in that, The application relates to a method for underwater multi-target classification based on cross-modal contrast learning, comprising the following steps: Step 1: obtaining time-synchronized and space-registered underwater optical image data and acoustic image data to form an original data set and dividing the original data set into a training set and a test set; Step 2: constructing a double-flow feature extraction network comprising an optical feature extraction branch and an acoustic feature extraction branch, inputting the training set into the double-flow feature extraction network, and extracting low-level general features and high-level semantic features of the optical image and the acoustic image respectively; Step 3: constructing a global query space attention module and embedding the double-flow feature extraction network, fusing the low-level general features of the optical image and the low-level general features of the acoustic image by using the global query space attention module, and obtaining fused general feature maps; Step 4: introducing a cross-modal contrast learning module in the training stage, mapping the high-level semantic features of the optical image and the high-level semantic features of the acoustic image to a shared embedding space, and driving semantic alignment between the two modes by calculating a contrast loss function; Step 5: performing global feature aggregation processing on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image and the fused general feature maps output by the global query space attention module to generate a global feature vector; Step 6: inputting the global feature vector into a multi-label classification head to output a classification prediction result of the underwater target.

2. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 1, characterized in that, The step 1 specifically comprises the following steps: Step 1.1: synchronously collecting underwater optical image data and acoustic image data by using an underwater robot platform equipped with a high-definition camera and a forward-looking sonar to construct an original data set; Step 1.2: analyzing the original data to extract time-synchronized optical frames and acoustic frames; Step 1.3: adjusting each frame of image to a uniform size and performing normalization processing; Step 1.4: constructing the preprocessed original data into a training set and a test set containing multiple types of underwater targets.

3. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 1, characterized in that, The optical feature extraction branch and the acoustic feature extraction branch in the step 2 are two parallel branches with unshared weights; both the two parallel branches adopt a hybrid architecture combining a convolutional neural network and a Transformer, and the hybrid architecture comprises a convolutional layer and a self-attention layer; The optical feature extraction branch comprises a low-level optical general feature extraction module and a high-level optical semantic feature extraction module; and the acoustic feature extraction branch comprises a low-level acoustic general feature extraction module and a high-level acoustic semantic feature extraction module.

4. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 1, characterized in that, The specific processing process of the global query space attention module in the step 3 comprises the following steps: Step 3.1: dynamically selecting low-level general features of a good quality modality as main modality features and low-level general features of another modality as auxiliary modality features according to the imaging quality of the optical image data and the acoustic image data; Step 3.2: inputting the main modality features into a 1x1 convolutional layer to obtain a query vector Q, and inputting the auxiliary modality features into a 1x1 convolutional layer to obtain a key vector K and a value vector V; Step 3.3: Adaptive average pooling is performed on the query vector Q and the key vector K respectively, compressing their spatial dimensions to 1x1, resulting in the main modality global context vector and the auxiliary modality global context vector ; Step 3.3: calculating the dot product of the main modality global context vector and the auxiliary modality global context vector, and calculating a global attention score by using a Softmax function; wherein S is a global attention score, is the number of feature channels; Step 3.4: performing matrix multiplication on the global attention score and the value vector V which retains complete space information to obtain weighted context information. wherein, denotes the broadcast multiplication of a scalar by a tensor; Step 3.5: After the weighted context information is processed by the convolution layer, it is added to the original main modal feature through a residual connection, and the fused general feature map is output.

5. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 1, characterized in that, The step 4 is specifically: Step 4.1: The high-level semantic features of the optical image and the high-level semantic features of the acoustic image are respectively mapped to the same dimension shared embedding space by using a nonlinear projection head, to obtain normalized optical feature vectors and acoustic feature vectors; Step 4.2: The cosine similarity between the high-level semantic features of each optical image and the high-level semantic features of each acoustic image in the training batch is calculated to construct a similarity matrix; Step 4.3: An InfoNCE loss function is constructed, and the network parameters are updated by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs; A one-way InfoNCE loss function with optical features as anchor points and acoustic features as positive samples is denoted as : The one-way InfoNCE loss function with acoustic features as anchor points and optical features as positive samples is denoted as : where N denotes the total number of samples in the batch, normalized i-th optical feature vector, normalized k-th optical feature vector, normalized i-th acoustic feature vector, normalized k-th acoustic feature vector; and constitute the i-th positive sample; all other 2(N-1) samples in the batch are negative samples; denotes the cosine similarity between u and v; τ is a temperature coefficient used to control the degree of smoothing of the distribution; is an indicator function that takes the value 1 when and 0 otherwise. The final cross-modal contrastive loss is defined as the average of the loss in two directions, and the calculation formula is as follows: wherein, denotes the cross-modal contrastive loss.

6. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 1, characterized in that, The step 5 is specifically: Step 5.1: Global average pooling is performed on the high-level semantic features of the optical image, the high-level semantic features of the acoustic image and the fused general feature map respectively to obtain feature vectors of multiple scales; Step 5.2: The feature vectors of multiple scales are spliced to form a comprehensive feature representation; the spliced vectors are reduced in dimension and information is fused by a fully connected layer to generate a final global feature vector.

7. The underwater optical-acoustic image fusion and classification method based on cross-modal contrast learning and global attention according to claim 5, characterized in that, The total loss function of the network is composed of a weighted classification loss and a cross-modal contrastive loss, and the calculation formula is as follows: wherein, is the sum of the binary cross-entropy loss of the optical feature extraction branch and the acoustic feature extraction branch, is a hyperparameter for the balancing weight.

Images