Marine unmanned equipment target detection method based on multi-modal representation learning

Abstract

The application discloses a kind of marine unmanned equipment target detection methods based on multi-modal representation learning, belong to artificial intelligence, marine high-end equipment and so on field, comprising: S1: image data, acoustic data and numerical data are respectively preprocessed, and through space-time alignment, generate after synchronization multi-modal data;S2: based on the multi-modal data after synchronization, extract multi-modal feature vector, through projection head mapping to low-dimensional embedding space, based on contrast learning loss and sequence mask reconstruction loss, large-scale pre-training is carried out, and pre-training multi-modal model is generated;S3: based on pre-training multi-modal model, freeze the parameters of encoder, design prompt vector and multi-modal data are spliced, and through self-regularization constraint, parameter efficient fine-tuning is carried out, and classification model is generated;S4: based on pruning and quantization operation, the classification model is compressed, and deployed to the edge computing unit of marine unmanned equipment.The normal target detection capability of marine unmanned equipment is improved.

Description

Technical Field

[0001] This invention relates to fields such as artificial intelligence and high-end marine equipment, and in particular to a target detection method for unmanned marine equipment based on multimodal representation learning. Background Technology

[0002] Marine unmanned equipment is an intelligent surface platform that relies on autonomous control systems, remote control, or a combination of both to navigate and execute missions. Equipped with acoustic, optical, electronic, and magnetic payloads, it can routinely acquire multi-source data and perform real-time dynamic target perception in the open ocean. Acoustic signals are the only effective carrier for underwater information transmission. Currently, the mainstream target detection method is acoustic detection. Marine unmanned equipment is equipped with towed active or passive sonar to receive sensor information such as passive target radiated noise or active target echoes, extract target features, and identify target types.

[0003] However, the performance of acoustic target detection is significantly affected by the stability of the marine environment, and its adaptability to noise interference and complex environments is limited. In dynamically changing marine environments, factors such as strong background noise, spectral aliasing, and weakened target features can significantly reduce the reliability of acoustic detection. Furthermore, the single information source of acoustic detection creates a bottleneck when processing the diversity and complexity of target features, especially in environments with strong interference or multiple targets, easily leading to false detections and missed detections. To address the inherent limitations of single-modal acoustic detection, multi-source information fusion sensing and detection technology is increasingly becoming a research hotspot. This method combines multiple (similar or dissimilar) information sources according to a specific standard in space or time to obtain a consistent interpretation or description of the target object, improving the perception capability of marine unmanned equipment and other unmanned systems for high-value targets. However, in real-world scenarios, due to data heterogeneity, task gaps and inter-domain differences, and limited edge computing resources, existing target detection methods still have significant shortcomings in representation modeling and cross-modal feature extraction, few-shot / zero-shot learning, and multi-scenario, multi-task generalization, seriously affecting the accuracy and reliability of target detection for marine unmanned equipment.

[0004] Therefore, it is necessary to design a target detection method for marine unmanned equipment based on multimodal representation learning to overcome the shortcomings of acoustic single-modal target detection. Summary of the Invention

[0005] This invention provides a target detection method for marine unmanned equipment based on multimodal representation learning to solve the above-mentioned problems in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A target detection method for marine unmanned equipment based on multimodal representation learning includes:

[0008] S1: Preprocess image data, acoustic data, and numerical data separately, and generate synchronized multimodal data through spatiotemporal alignment;

[0009] S2: Based on synchronized multimodal data, multimodal feature vectors are extracted through visual encoder, acoustic encoder and text encoder, mapped to low-dimensional embedding space through projection head, and pre-trained multimodal model is generated by large-scale pre-training based on contrastive learning loss and sequence mask reconstruction loss.

[0010] S3: Based on a pre-trained multimodal model, the encoder parameters are frozen, and cue vectors are designed and concatenated with multimodal data. The parameters are then finely tuned efficiently through self-regularization constraints to generate a classification model.

[0011] S4: The classification model is compressed based on pruning and quantization operations and deployed to the edge computing unit of marine unmanned equipment.

[0012] Furthermore, S1 includes:

[0013] S11: Perform denoising, sharpening, enhancement, cropping, scaling and rotation on the image data to generate preprocessed image data;

[0014] S12: Filter and denoise the acoustic data, and perform spectral conversion to generate preprocessed acoustic data;

[0015] S13: Clean, normalize, and standardize the numerical data, embed text templates to form text representations, and generate preprocessed text data;

[0016] S14: Based on the preprocessed image data, preprocessed acoustic data, and preprocessed text data, spatial and temporal alignment is achieved using interpolation, resampling, and coordinate transformation to generate synchronized multimodal data.

[0017] Furthermore, S2 includes:

[0018] S21: Convert the synchronized multimodal data into a sequence format, perform random masking on the sequence, and generate encoded sequence data through linear mapping and positional encoding.

[0019] S22: Extract features from the encoded sequence data using a visual encoder, an acoustic encoder, and a text encoder based on the Transformer self-attention mechanism, respectively, to generate corresponding image feature vectors, acoustic feature vectors, and text feature vectors;

[0020] S23: The image feature vector, acoustic feature vector, and text feature vector are mapped to a low-dimensional embedding space of the same dimension by the projection head to generate a low-dimensional feature vector.

[0021] S24: Based on low-dimensional feature vectors, perform bidirectional cross-modal contrastive learning of image-text and text-image, as well as bidirectional cross-modal contrastive learning of acoustic signal-text and text-acoustic signal, and calculate the contrastive learning loss function;

[0022] S25: Based on low-dimensional feature vectors, the sequence mask data is reconstructed using a decoder, and the sequence mask reconstruction loss function is calculated.

[0023] S26: Based on the contrastive learning loss function and the sequence mask reconstruction loss function, the relative contribution of the two loss functions is adjusted by hyperparameters, and optimized by gradient backpropagation to generate a pre-trained multimodal model.

[0024] Furthermore, S3 includes:

[0025] S31: Freeze the encoder parameters of the pre-trained multimodal model, remove the projector head and decoder, and connect the frozen encoder to a parameter-trainable classifier.

[0026] S32: Design learnable soft cue vectors for visual and acoustic modalities respectively, and design hard cue vectors based on natural language templates for text modalities.

[0027] S33: Concatenate the multimodal data with the corresponding cue vector to form a new input. Only fine-tune the cue vector and the output layer parameters of the classifier, while keeping the other parameters frozen.

[0028] S34: Based on the class probability distribution obtained by adding prompts and the class probability distribution obtained by not adding prompts, a self-regularization constraint is applied using Kullback-Leibler divergence, and the model is trained using the cross-entropy loss function to generate a classification model.

[0029] Furthermore, S4 includes:

[0030] S41: Perform attention head pruning operation in the Transformer multi-head attention mechanism on the classification model to remove redundant attention heads with low contribution and generate the pruned model.

[0031] S42: Quantize the weight parameters in the pruned model, converting high-precision floating-point numbers into low-precision integers to generate a compressed model;

[0032] S43: Deploy a compressed model on the edge computing unit of the marine unmanned equipment, and load the encoder of the corresponding mode for target detection according to the actual application requirements.

[0033] Furthermore, the formula for calculating the contrastive learning loss function in S24 is as follows:

[0034] ;

[0035] in, The total number of samples within the batch. and They are Matching pairs of the same label and matching pairs of different labels, To control the smoothness of similarity using temperature parameters, This represents the cosine similarity.

[0036] Furthermore, the formula for calculating the sequence mask reconstruction loss function in S25 is as follows:

[0037] ;

[0038] in, Represents the original sequence data. This represents the sequence data reconstructed by the decoder based on the mask data.

[0039] Furthermore, the formula for calculating the self-regularization constraint in S34 is as follows:

[0040] ;

[0041] in, The first one obtained after adding the hint vector The class probability distribution of each sample The first one obtained when no hint vector was added The probability distribution of the categories of each sample.

[0042] Furthermore, the masking strategy employs local region masking and blurring masking in the visual modality; random character masking, random word masking, and random sentence masking in the text modality; and frequency masking, time masking, and noise masking in the acoustic modality.

[0043] The decoder consists of an upsampling layer, a deconvolution layer, and a fully connected layer or a Transformer decoding module. It restores the feature vectors in the low-dimensional embedding space to the original sequence dimension through layer-by-layer feature reconstruction, thereby realizing data recovery at the sequence mask location.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] This invention overcomes the limitations of single-type data by collaboratively processing multi-sensor data from marine unmanned equipment to achieve multi-source information fusion and integrated perception. Utilizing the data complementarity and cross-modal retrieval capabilities between sensors, it enhances the system's small-sample learning ability in situations of missing or imbalanced marine data, further optimizing data advantages and improving detection accuracy and understanding capabilities. This invention improves the system's versatility through multimodal representation learning and efficient parameter fine-tuning. Based on pre-training, it enables transfer learning to downstream tasks, achieving customized applications in different scenarios. Typical applications include acoustic surveys of marine mammals, surface vessel detection and obstacle avoidance, and plankton abundance estimation. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0047] Figure 1 This is a flowchart of a target detection method for marine unmanned equipment based on multimodal representation learning, as described in an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of the multimodal representation learning pre-training process in an embodiment of the present invention;

[0049] Figure 3 This is a flowchart illustrating the efficient parameter fine-tuning process in an embodiment of the present invention. Detailed Implementation

[0050] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0051] The embodiments of the present invention provide, as follows Figure 1 As shown, a target detection method for marine unmanned equipment based on multimodal representation learning includes:

[0052] S1: Preprocess image data, acoustic data, and numerical data separately, and generate synchronized multimodal data through spatiotemporal alignment;

[0053] S2: Based on synchronized multimodal data, multimodal feature vectors are extracted through visual encoder, acoustic encoder and text encoder, mapped to low-dimensional embedding space through projection head, and pre-trained multimodal model is generated by large-scale pre-training based on contrastive learning loss and sequence mask reconstruction loss.

[0054] S3: Based on a pre-trained multimodal model, the encoder parameters are frozen, and cue vectors are designed and concatenated with multimodal data. The parameters are then finely tuned efficiently through self-regularization constraints to generate a classification model.

[0055] S4: The classification model is compressed based on pruning and quantization operations and deployed to the edge computing unit of marine unmanned equipment.

[0056] The following is a detailed description with reference to specific embodiments.

[0057] This invention provides a target detection method for marine unmanned equipment based on multimodal representation learning. The overall technical approach of this invention includes three core stages: Multimodal pre-training: Based on sequence masking and contrastive learning, large-scale representation learning is performed on multi-sensor data (images, acoustic signals, and numerical signals) of marine unmanned equipment to construct a cross-modal aligned low-dimensional embedding space. Efficient parameter fine-tuning: Through cue learning (soft and hard cues) and self-regularization methods, the parameters of the pre-trained model are frozen, and only the cue vector and classifier are fine-tuned to achieve efficient transfer to downstream tasks. Model compression and deployment: Pruning and quantization techniques are used to compress the model size and accelerate inference, ultimately enabling customized deployment of multi-scenario target detection on edge devices.

[0058] Specifically, the following steps are included:

[0059] S1: Preprocess image data, acoustic data, and numerical data separately, and generate synchronized multimodal data through spatiotemporal alignment.

[0060] Specifically, step S1 includes the following sub-steps:

[0061] S11: Perform denoising, sharpening, enhancement, cropping, scaling, and rotation on the image data to generate preprocessed image data.

[0062] Specifically, the marine unmanned equipment acquires image data using a three-axis vision sensor. The acquired images undergo denoising to remove noise interference generated during acquisition, sharpening to enhance image edge information, and enhancement operations to improve image contrast and readability. Based on subsequent processing requirements, the images are cropped to extract regions of interest, scaled to standardize image size, and rotated to correct image orientation, ultimately generating preprocessed image data.

[0063] S12: Filter and denoise the acoustic data, and perform spectral conversion to generate preprocessed acoustic data.

[0064] Specifically, the marine unmanned equipment acquires acoustic data using a towed hydrophone. The towed hydrophone is connected to an underwater tractor via a rubber cable and is responsible for collecting underwater acoustic signals. The acquired acoustic signals are filtered to remove out-of-band interference and noise-reducing to extract useful acoustic features. The acoustic signals undergo spectral conversion, successively converting them into time-frequency representations such as LOFAR spectrum (low-frequency analysis recording spectrum), DEMON spectrum (demodulation envelope spectrum), and Mel-frequency cepstral coefficients (MFCCs), and then stitched together along the channel dimension to generate preprocessed acoustic data.

[0065] S13: Clean, normalize, and standardize the numerical data, embed text templates to form text representations, and generate preprocessed text data.

[0066] Specifically, marine unmanned equipment can be equipped with environmental parameter sensors such as wave meters and weather stations to acquire numerical data, depending on actual needs. The collected numerical data undergoes cleaning to remove outliers, and normalization and standardization processes are performed to eliminate noise, enabling comparison and analysis on a uniform scale. The cleaned numerical data, along with category information, is then embedded into a pre-designed fixed text template to form a text representation.

[0067] In this embodiment, the design examples of the text template are shown in Table 1:

[0068] Table 1. Text Template Design Examples:

[0069] .

[0070] The numerical data and category information are converted into a natural language text representation using the text template described above, generating preprocessed text data.

[0071] S14: Based on the preprocessed image data, preprocessed acoustic data, and preprocessed text data, spatial and temporal alignment is achieved using interpolation, resampling, and coordinate transformation to generate synchronized multimodal data.

[0072] Specifically, preprocessed image, acoustic, and text data acquired from different sensors undergo temporal alignment and spatial processing. For temporal alignment, data from different sensors is time-aligned based on their timestamps using interpolation methods to ensure temporal consistency. Spatially, data from different sensors is spatially aligned based on their spatial location information using coordinate transformation and spatial interpolation methods to ensure spatial consistency. Through this spatiotemporal alignment processing, the fusion and integrated perception of multi-sensor, multi-source data from marine unmanned equipment are achieved, generating synchronized multimodal data.

[0073] S2: Based on synchronized multimodal data, multimodal feature vectors are extracted using a visual encoder, acoustic encoder, and text encoder, and then mapped to a low-dimensional embedding space using a projection head, such as... Figure 2 As shown, a pre-trained multimodal model is generated by large-scale pre-training based on contrastive learning loss and sequence mask reconstruction loss.

[0074] Step S2 includes the following sub-steps:

[0075] S21: Convert the synchronized multimodal data into a sequence format, perform random masking on the sequence, and generate encoded sequence data through linear mapping and positional encoding.

[0076] Specifically, based on the input requirements of the Transformer model, the image, acoustic, and text data in the synchronized multimodal data are converted into sequence formats suitable for Transformer processing. The converted sequences are then subjected to random masking, employing different masking strategies to partially mask the sequence data of different modalities.

[0077] In this embodiment, local region masking and blurring masking strategies are employed for the visual modality. Local region masking divides the image into several regions, then randomly selects some of these regions and sets them to black, white, or other fixed background colors to achieve a masking effect. Blurring masking involves randomly selecting some regions and scrambling the pixel positions within them, allowing the model to learn texture and structural information when reconstructing the image.

[0078] For text modalities, random character masking, random word masking, and random sentence masking strategies are employed. Random character masking involves randomly selecting a subset of characters from the text and replacing them with a special mask marker [MASK]. Random word masking involves randomly selecting a subset of words from the text and replacing them with a special mask marker. Random sentence masking involves randomly selecting a subset of sentences from the text and masking them, or using [CLC] to mask category information in the text, thereby enhancing the model's understanding of text context and facilitating its learning of long text sequences and grammar.

[0079] Frequency masking, time masking, and noise masking strategies are employed for acoustic modes. Frequency masking adds a mean mask to the frequency dimension of the spectrum. Time masking adds a mean mask to the time dimension of the spectrum. Noise masking adds random noise to certain segments of the acoustic signal to reduce the signal-to-noise ratio, thus achieving the effect of masking. By adding masks, the model's ability to model the frequency and temporal information of acoustic data is improved.

[0080] Each sequence block in the masked sequence data is linearly mapped to a fixed-dimensional vector representation, and positional encoding is added to preserve the positional information of the sequence. The resulting encoded sequence data is then used as the input to the encoder.

[0081] S22: Extract features from the encoded sequence data using a visual encoder, an acoustic encoder, and a text encoder based on the Transformer self-attention mechanism, respectively, to generate corresponding image feature vectors, acoustic feature vectors, and text feature vectors.

[0082] Specifically, a visual encoder, an acoustic encoder, and a text encoder are designed based on the Transformer model. The visual encoder processes image sequence data, the acoustic encoder processes acoustic spectrum sequence data, and the text encoder processes text sequence data. All three encoders employ a multi-head self-attention mechanism based on the Transformer structure for feature extraction.

[0083] The visual encoder receives the encoded sequence data corresponding to the image, learns the dependencies within the image sequence through the self-attention mechanism in the multi-layer Transformer structure, extracts local and global features of the image, and generates an image feature vector.

[0084] The acoustic encoder receives the encoded sequence data corresponding to the acoustic spectrum, learns the time-frequency features of the acoustic sequence through the self-attention mechanism in the multi-layer Transformer structure, and generates an acoustic feature vector.

[0085] The text encoder receives the encoded sequence data corresponding to the text, learns the semantic features and contextual information of the text sequence through the self-attention mechanism in the multi-layer Transformer structure, and generates a text feature vector.

[0086] S23: The image feature vector, acoustic feature vector, and text feature vector are mapped to a low-dimensional embedding space of the same dimension by the projection head to generate a low-dimensional feature vector.

[0087] Specifically, corresponding projection heads are configured for encoders of different modalities. Each projection head consists of fully connected layers, used to map the high-dimensional feature vectors of different modalities to a dimensionless array. The image feature vectors output by the visual encoder are mapped to the low-dimensional embedding space through a visual projector, generating low-dimensional image feature vectors. Similarly, the acoustic feature vectors output by the acoustic encoder are mapped to the low-dimensional embedding space through an acoustic projector, generating low-dimensional acoustic feature vectors. Likewise, the text feature vectors output by the text encoder are mapped to the low-dimensional embedding space through a text projector, generating low-dimensional text feature vectors. This mapping by the projector ensures that feature vectors from different modalities are comparable within the same low-dimensional embedding space, laying the foundation for subsequent cross-modal contrastive learning.

[0088] S24: Based on low-dimensional feature vectors, perform bidirectional cross-modal contrastive learning between image and text and between text and image, as well as bidirectional cross-modal contrastive learning between acoustic signal and text and between text and acoustic signal, and calculate the contrastive learning loss function.

[0089] Specifically, contrastive learning is used in the low-dimensional embedding space to align cross-modal features. For both image and text modalities, image-text cross-modal contrastive learning is performed, matching low-dimensional image feature vectors with low-dimensional text feature vectors. This ensures that image-text pairs of the same category attract each other, while image-text pairs of different categories repel each other. Simultaneously, text-image cross-modal contrastive learning is performed, matching low-dimensional text feature vectors with low-dimensional image feature vectors, achieving bidirectional contrastive learning.

[0090] For both acoustic and text modalities, a cross-modal contrastive learning process is performed between acoustic signals and text, matching low-dimensional acoustic feature vectors with low-dimensional text feature vectors. Simultaneously, a cross-modal contrastive learning process between text and acoustic signals is performed, matching low-dimensional text feature vectors with low-dimensional acoustic feature vectors, thus achieving bidirectional contrastive learning.

[0091] The formula for calculating the contrastive learning loss function is:

[0092] ;

[0093] in, The total number of samples within the batch. Indicates the first A low-dimensional feature vector of a modality for each sample. and They are The low-dimensional feature vectors of another modality corresponding to the same label matching pairs (positive sample pairs) and different label matching pairs (negative sample pairs), To control the smoothness of similarity using temperature parameters, Represents cosine similarity. This is the index for iterating through all samples in the batch.

[0094] In this embodiment, temperature parameter The choice depends on the specific application scenario data; generally, it is chosen as... ,in The temperature parameter represents the dimension of the low-dimensional embedding space. A larger temperature parameter can make the training process smoother, which is suitable for situations with high data noise or weak correlation between modalities. A smaller temperature parameter can increase the similarity difference between positive and negative samples, further improving the model's data representation ability.

[0095] By employing a bidirectional contrastive learning strategy, we can enhance cross-modal semantic collaboration capabilities, capture complementary information between different modalities, ensure semantic consistency among modalities, resolve the differences between modalities, and improve the model's ability to understand and integrate multimodal information.

[0096] S25: Based on low-dimensional feature vectors, the sequence mask data is reconstructed using a decoder, and the sequence mask reconstruction loss function is calculated.

[0097] Specifically, to further enhance the representation capability of information within each modality, the low-dimensional feature vectors of different modalities in the low-dimensional embedding space are input into the corresponding decoders for sequence mask reconstruction. The decoder consists of an upsampling layer, a deconvolution layer, and a fully connected or Transformer decoding module layer. Through layer-by-layer feature reconstruction, the feature vectors in the low-dimensional embedding space are restored to the original sequence dimension, thereby achieving data recovery at the sequence mask location.

[0098] The visual decoder receives low-dimensional feature vectors from the image and reconstructs the image data at the sequence mask location through cascaded processing of upsampling layers, deconvolution layers, and fully connected layers, generating reconstructed image sequence data.

[0099] The acoustic decoder receives low-dimensional acoustic feature vectors and reconstructs the acoustic data at the sequence mask positions through cascaded processing of upsampling layers, deconvolution layers, and fully connected layers, generating reconstructed acoustic sequence data.

[0100] The text decoder receives the low-dimensional feature vector of the text, processes it through the Transformer decoding module, reconstructs the text data at the sequence mask position, and generates the reconstructed text sequence data.

[0101] The formula for calculating the sequence mask reconstruction loss function is:

[0102] ;

[0103] in, Represents the original sequence data. This represents the sequence data reconstructed by the decoder based on the mask data. Reconstruction losses are calculated for the visual, acoustic, and text modalities respectively. By calculating the mean squared error between the original and reconstructed data, the reconstruction effect for each modality is ensured to be as close as possible to the original input data.

[0104] Sequence masking reconstruction, as a self-supervised learning method, forces the model to learn the intrinsic structure and patterns of the data in order to recover the masked data. While ensuring cross-modal feature sharing, it enhances the representation ability within a modality, helps the model learn more robust feature representations, and improves the model's generalization ability and robustness.

[0105] S26: Based on the contrastive learning loss function and the sequence mask reconstruction loss function, the relative contribution of the two loss functions is adjusted by hyperparameters, and optimized by gradient backpropagation to generate a pre-trained multimodal model.

[0106] Specifically, the contrastive learning loss function and the sequence mask reconstruction loss function are weighted and combined to construct the overall loss function. By optimizing the overall loss function, the parameters of the encoder, projector head, and decoder are updated using the gradient backpropagation algorithm, enabling large-scale multimodal pre-training. The pre-training process allows the model to effectively learn shared semantic information across different modalities, achieving good generalization ability and multimodal representation capabilities, thus generating a pre-trained multimodal model.

[0107] S3: Based on a pre-trained multimodal model, the encoder parameters are frozen, and cue vectors are designed and concatenated with multimodal data. The parameters are then finely tuned efficiently through self-regularization constraints to generate a classification model.

[0108] Step S3 includes the following sub-steps:

[0109] S31: Freeze the encoder parameters of the pre-trained multimodal model, remove the projector head and decoder, and connect the frozen encoder to a parameter-trainable classifier.

[0110] Specifically, such as Figure 3As shown, a cue-based learning method is used to efficiently fine-tune the parameters of a pre-trained large model. The core idea is to precisely select the model parts that need fine-tuning to avoid updating all parameters via gradients, thus achieving efficient, fast, and computationally efficient model optimization. Utilizing prior knowledge gained from pre-training, a small number of task-related parameters are fine-tuned to quickly adapt to downstream tasks, effectively addressing the few-shot learning, task gap, and domain adaptation problems in marine target detection. The encoder parameters in the pre-trained multimodal model, which possesses good generalization and multimodal representation capabilities, are frozen and not updated during downstream task fine-tuning. The projector and decoder, primarily used in the pre-training stage to enhance the encoder's feature extraction capabilities, are not retained during fine-tuning. The visual encoder, acoustic encoder, and text encoder with frozen parameters are connected to a parameter-trainable classifier to construct a deep neural network model for target detection tasks. The classifier consists of fully connected layers and a softmax output layer, used to map the feature vectors extracted by the encoder to the target class probability distribution.

[0111] S32: Design learnable soft cue vectors for the visual and acoustic modalities respectively, and design hard cue vectors based on natural language templates for the text modal.

[0112] Specifically, based on actual application scenarios, task-related prompts are designed to provide clear and explicit task guidance for the model, enabling efficient fine-tuning of parameters.

[0113] For visual modalities, learnable soft cue feature vectors are designed. These soft cue vectors represent cue information through learnable vectors, and are continuously updated during training. Visual soft cue vectors help the model focus on target regions or key features in images, gradually guiding the model to detect targets. The dimensionality of the visual soft cue vectors is consistent with the dimensionality of the image sequence, and they are concatenated with the image sequence data before being input into the visual encoder.

[0114] Learnable soft cue feature vectors are designed for acoustic modes. These soft cue vectors, through learning, help the model identify specific patterns or target signals in acoustic spectral data, improving target recognition performance. The dimensions of the acoustic soft cue vectors are consistent with the dimensions of the acoustic sequence, and they are concatenated with the acoustic sequence data before being input into the acoustic encoder.

[0115] For the text modality, a hard cue vector based on natural language templates is designed. Hard cues are typically provided through fixed, manually designed natural language text. Using the text templates in Table 1 as text hard cues, multi-sensor numerical data and class labels are embedded into the text templates to form hard cue vectors. These hard cue vectors are not updated during training; instead, they are directly concatenated with the text data as guiding signals and input into the text encoder for application in target detection tasks.

[0116] S33: Concatenate the multimodal data with the corresponding cue vector to form a new input. Only fine-tune the cue vector and the output layer parameters of the classifier, while keeping the other parameters frozen.

[0117] Specifically, image data from the synchronized multimodal data is concatenated with visual soft cue vectors to form an image input sequence containing cue information. Acoustic data is concatenated with acoustic soft cue vectors to form an acoustic input sequence containing cue information. Text data is concatenated with text hard cue vectors to form a text input sequence containing cue information.

[0118] The stitched image input sequence is fed into a frozen visual encoder for feature extraction, the stitched acoustic input sequence is fed into a frozen acoustic encoder for feature extraction, and the stitched text input sequence is fed into a frozen text encoder for feature extraction. The extracted multimodal feature vectors are then fed into a classifier for target category prediction.

[0119] During fine-tuning, only the soft cue vectors and the output layer parameters of the classifier are updated and optimized using gradients, while the frozen encoder parameters remain unchanged. Compared to methods that perform full fine-tuning, this approach, which fine-tunes only less than 10% of the parameters, achieves efficient parameter fine-tuning, reaching or exceeding the effectiveness of full fine-tuning while saving training time and computational resources.

[0120] By concatenating modal data and cue vectors, the model can focus more on task-related parts during training by utilizing cue information, avoiding over-reliance on noise and irrelevant components in the original data. By leveraging cross-modal information retrieval and the complementarity between modalities, the model's adaptability to downstream tasks is improved, effectively addressing the issues of few-shot learning, task gap, and domain adaptation in marine target detection.

[0121] S34: Based on the class probability distributions obtained with and without prompts, a classification model is generated by using Kullback-Leibler divergence for self-regularization constraints and combining it with the cross-entropy loss function for training.

[0122] Specifically, to address issues such as generalization performance degradation, knowledge forgetting, or overfitting that may occur when performing efficient parameter fine-tuning in cue learning, especially when data is limited, a self-regularization method is introduced, combined with Kullback-Leibler divergence for regularization constraints.

[0123] During parameter fine-tuning, the model output is calculated for both cases with and without cue vectors. The Kullback-Leibler divergence is used to measure the difference between the two probability distributions. The formula for calculating the self-regularization constraint is:

[0124] ;

[0125] in, The first one obtained after adding the hint vector The class probability distribution of each sample The first one obtained when no hint vector was added The probability distribution of the categories of each sample. and The difference reflects the degree to which the prompt information affects the model's predictions.

[0126] By minimizing and The Kullback-Leibler divergence between the two ensures that the model does not over-rely on cue information during fine-tuning. By adding self-regularization constraints, the model can avoid overfitting to cue information, avoid overfitting and catastrophic forgetting, retain pre-trained knowledge, adapt to new tasks, and improve the model's robustness and generalization ability in multiple scenarios and tasks.

[0127] The cross-entropy classification loss and the self-regularization constraint loss are weighted and combined to construct the overall loss function in the fine-tuning stage. The soft cue vector and classifier parameters are optimized through gradient backpropagation to complete parameter fine-tuning and generate a classification model for target detection missions of marine unmanned equipment.

[0128] Through cue-based learning and self-regularization, the model can learn how to effectively transfer knowledge between different modalities. In cases where acoustic data is lacking, information from visual and textual modalities can be complementary, thus supporting zero-shot underwater target recognition. Even without sufficient acoustic training samples, the model can still perform target recognition based on knowledge from visual and textual modalities.

[0129] Marine unmanned equipment has relatively limited computing resources. To address different marine target detection scenarios, marine unmanned equipment can deploy different modalities of deep neural network models for routine real-time detection tasks, such as deploying visual models for surface target recognition to assist obstacle avoidance and navigation, and deploying acoustic models for marine mammal surveys. Cross-modal retrieval can address the small-sample / zero-sample learning problem in marine observation and detection, and a "visual-text-acoustic" model can be used to identify target types lacking acoustic training samples by utilizing information from different modalities.

[0130] S4: The classification model is compressed based on pruning and quantization operations and deployed to the edge computing unit of marine unmanned equipment.

[0131] Step S4 includes the following sub-steps:

[0132] S41: Perform attention head pruning on the classification model using the Transformer multi-head attention mechanism to remove redundant attention heads with low contribution and generate the pruned model.

[0133] Specifically, the pre-trained classification model is designed to be lightweight to suit the edge computing unit of marine unmanned equipment. Multimodal representation learning uses a Transformer as the encoder for feature extraction. Within the Transformer, a multi-head attention mechanism allows the model to learn information in different subspaces, with each attention head representing a different attention subspace.

[0134] The importance of each attention head is quantified by calculating the L2 norm of its output feature vector. Attention heads with larger L2 norms contribute more to the model output, while those with smaller L2 norms contribute less. Attention heads are ranked according to their L2 norms, and redundant attention heads with smaller L2 norms and lower contributions are selectively removed. Redundant attention heads are then removed through pruning, reducing the number of model parameters and computational cost, resulting in a pruned model.

[0135] Quantization is a technique that reduces model size and improves inference speed by decreasing the precision of numerical representations in the model. This method converts high-precision floating-point numbers into low-precision integers, thereby compressing model parameters and increasing inference speed. After pruning and quantization, the compressed model is deployed on marine unmanned equipment. The inference computing unit should be equipped with GPUs or NPUs to achieve fast real-time inference of deep neural networks.

[0136] S42: Quantize the weight parameters in the pruned model, converting high-precision floating-point numbers into low-precision integers to generate a compressed model.

[0137] Specifically, quantization is a technique that reduces model size and improves inference speed by decreasing the precision of numerical representations in the model. Quantization is performed on the weight parameters in the pruned model, converting 32-bit high-precision floating-point numbers (FP32) into 8-bit or 16-bit low-precision integers (INT8 or INT16). The quantization process includes determining the quantization range, calculating the scaling factor and zero-point offset, and discretizing the weight parameters. By compressing the storage space of model parameters and improving inference speed through quantization, the size of model parameters is significantly reduced while sacrificing a small amount of precision, resulting in a compressed model.

[0138] S43: Deploy a compressed model on the edge computing unit of the marine unmanned equipment, and load the encoder of the corresponding mode for target detection according to the actual application requirements.

[0139] Specifically, the marine unmanned equipment adopts a hardware platform that supports multiple sensor types and is equipped with a high-efficiency data acquisition, processing, and transmission system. The hardware platform provides various interface protocols (such as USB, Ethernet, CAN bus, etc.) and has a certain degree of expandability to support data acquisition from different types of sensors. Standardized communication protocols and data formats are adopted, including unified message formats, data encoding methods, and data synchronization methods, ensuring compatibility between all sensor modules and processing units, enabling efficient and error-free data exchange, and improving compatibility and interoperability between modules.

[0140] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of this invention.

Claims

1. A target detection method for marine unmanned equipment based on multimodal representation learning, characterized in that, include: S1: Preprocess image data, acoustic data, and numerical data separately, and generate synchronized multimodal data through spatiotemporal alignment; S2: Based on synchronized multimodal data, multimodal feature vectors are extracted using a visual encoder, acoustic encoder, and text encoder. These vectors are then mapped to a low-dimensional embedding space using a projection head. Large-scale pre-training based on contrastive learning loss and sequence mask reconstruction loss is performed to generate a pre-trained multimodal model, including: S21: Convert the synchronized multimodal data into a sequence format, perform random masking on the sequence, and generate encoded sequence data through linear mapping and positional encoding. S22: Extract features from the encoded sequence data using a visual encoder, an acoustic encoder, and a text encoder based on the Transformer self-attention mechanism, respectively, to generate corresponding image feature vectors, acoustic feature vectors, and text feature vectors; S23: The image feature vector, acoustic feature vector, and text feature vector are mapped to a low-dimensional embedding space of the same dimension by the projection head to generate a low-dimensional feature vector. S24: Based on low-dimensional feature vectors, perform bidirectional cross-modal contrastive learning of image-text and text-image, as well as bidirectional cross-modal contrastive learning of acoustic signal-text and text-acoustic signal, and calculate the contrastive learning loss function; S25: Based on low-dimensional feature vectors, the sequence mask data is reconstructed using a decoder, and the sequence mask reconstruction loss function is calculated. S26: Based on the contrastive learning loss function and the sequence mask reconstruction loss function, the relative contribution of the two loss functions is adjusted by the hyperparameter, and optimized by gradient backpropagation to generate a pre-trained multimodal model; S3: Based on a pre-trained multimodal model, the encoder parameters are frozen, and cue vectors are designed and concatenated with multimodal data. The parameters are then finely tuned efficiently through self-regularization constraints to generate a classification model. S4: The classification model is compressed based on pruning and quantization operations and deployed to the edge computing unit of marine unmanned equipment.

2. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, S1 includes: S11: Perform denoising, sharpening, enhancement, cropping, scaling and rotation on the image data to generate preprocessed image data; S12: Filter and denoise the acoustic data, and perform spectral conversion to generate preprocessed acoustic data; S13: Clean, normalize, and standardize the numerical data, embed text templates to form text representations, and generate preprocessed text data; S14: Based on the preprocessed image data, preprocessed acoustic data, and preprocessed text data, spatial and temporal alignment is achieved using interpolation, resampling, and coordinate transformation to generate synchronized multimodal data.

3. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, S3 include: S31: Freeze the encoder parameters of the pre-trained multimodal model, remove the projector head and decoder, and connect the frozen encoder to a parameter-trainable classifier. S32: Design learnable soft cue vectors for visual and acoustic modalities respectively, and design hard cue vectors based on natural language templates for text modalities; S33: Concatenate the multimodal data with the corresponding cue vector to form a new input. Only fine-tune the cue vector and the output layer parameters of the classifier, while keeping the other parameters frozen. S34: Based on the class probability distribution obtained by adding prompts and the class probability distribution obtained by not adding prompts, the Kullback-Leibler divergence is used for self-regularization constraint, and the cross-entropy loss function is combined for training to generate a classification model.

4. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, S4 include: S41: Perform attention head pruning on the classification model using the Transformer multi-head attention mechanism to remove redundant attention heads with low contribution and generate the pruned model. S42: Quantize the weight parameters in the pruned model, converting high-precision floating-point numbers into low-precision integers to generate a compressed model; S43: Deploy a compressed model on the edge computing unit of the marine unmanned equipment, and load the encoder of the corresponding mode for target detection according to the actual application requirements.

5. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the contrastive learning loss function in S24 is: ； in, The total number of samples within the batch. and They are Matching pairs of the same label and matching pairs of different labels, To control the smoothness of similarity using temperature parameters, This represents the cosine similarity.

6. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, The formula for calculating the sequence mask reconstruction loss function in S25 is as follows: ； in, Represents the original sequence data. This represents the sequence data reconstructed by the decoder based on the mask data.

7. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 3, characterized in that, The formula for calculating the self-regularization constraint in S34 is: ； in, The first one obtained after adding the hint vector The class probability distribution of each sample The first one obtained when no hint vector was added The probability distribution of the categories of each sample.

8. The target detection method for marine unmanned equipment based on multimodal representation learning according to claim 1, characterized in that, The masking strategy employs local region masking and blurring masking in the visual modality; random character masking, random word masking, and random sentence masking in the text modality; and frequency masking, time masking, and noise masking in the acoustic modality. The decoder consists of an upsampling layer, a deconvolution layer, and a fully connected layer or a Transformer decoding module. It restores the feature vectors in the low-dimensional embedding space to the original sequence dimension through layer-by-layer feature reconstruction, thereby realizing data recovery at the sequence mask location.

Images