A multi-modal deep feature fusion-based underwater acoustic target recognition system and method

Abstract

The application relates to a kind of underwater acoustic target recognition system and method based on multimodal deep feature fusion, belong to underwater acoustic target recognition field.First, the original audio and associated metadata of the obtained underwater acoustic target are preprocessed, and a multimodal dataset is constructed;Subsequently, the constructed multimodal sample is input into the recognition model, the model extracts the deep representation of each mode through a multi-branch feature coding network, uses the cross-modal cross-attention mechanism guided by latent query to realize the deep alignment and complementary aggregation of different modal features, and outputs the target recognition result based on the decision network of mixed experts.The application is verified by experiments on two public underwater acoustic datasets, and the experimental results verify the effectiveness of the multimodal deep fusion and mixed expert adaptive decision strategy adopted by the application, which significantly improves the robustness and generalization ability of the underwater acoustic target recognition system in strong noise and multi-working-condition environment by exploiting the complementary advantages of acoustic features and semantic priori.

Description

Technical Field

[0001] This invention relates to an underwater acoustic target recognition system and method based on multimodal deep feature fusion, belonging to the field of underwater acoustic target recognition. Background Technology

[0002] Underwater acoustic target recognition, as a core technology of underwater situational awareness, aims to identify target categories by deeply analyzing and processing complex underwater sound waves received by sonar equipment and extracting effective discriminative features. This technology is widely used in marine ecological environment monitoring, deep-sea resource exploration, and underwater security, providing important decision-making basis for safeguarding national maritime rights and promoting marine economic development. It has become a research hotspot and challenge in the field of underwater acoustic signal processing. Although a large amount of research has been dedicated to developing ship radiated noise recognition systems, underwater acoustic target recognition still faces significant challenges due to the complexity and variability of the marine environment, the nonlinearity of target radiated noise generation mechanisms, and the scarcity of high-quality measured databases. Existing technologies still have the following key shortcomings when facing complex, non-stationary, and highly disturbed marine environments:

[0003] 1. Feature extraction relies on manual design, which has significant limitations: Traditional methods require experts to design features manually based on prior knowledge. This not only involves cumbersome feature engineering, but also only extracts shallow statistical features, making it difficult to mine the high-dimensional nonlinear information hidden in underwater acoustic signals. Furthermore, the feature expression capability is severely insufficient under complex sea conditions, resulting in limited recognition accuracy.

[0004] 2. Information gaps exist in single-mode modeling: Most existing methods only analyze acoustic signals based on a single mode, focusing only on time-series modeling and often ignoring the spatial distribution characteristics of the frequency domain, making it difficult to capture the global topological structure of the spectrogram; if only image-dimensional modeling is emphasized, the inherent temporal dynamic attributes of acoustic signals are easily severed, resulting in the loss of time-domain evolution laws.

[0005] 3. Lack of semantic guidance and error correction mechanisms: The model ignores key environmental contextual information such as sea state, water depth, and wind speed, and fails to utilize the inherent semantic attributes of the target for constraint. Relying solely on acoustic features easily leads to semantic ambiguity and category confusion under strong noise interference, resulting in a lack of model error correction capabilities.

[0006] 4. Inefficient feature fusion methods: Using only CNNs easily loses long-range temporal dependencies, and using only Transformers makes it difficult to capture fine-grained spectral structures; moreover, existing simple feature splicing strategies cannot achieve effective alignment of heterogeneous information in deep space, resulting in poor fusion effects and low information utilization.

[0007] 5. Insufficient robustness of decision networks: The time-varying multipath effect of underwater acoustic channels leads to huge differences in the acoustic feature distribution of the same type of target in different sea areas and at different distances. A single decision network is difficult to fit this complex distribution drift and has weak cross-scenario generalization ability. Summary of the Invention

[0008] To overcome the limitations of feature extraction, lack of single-modal information, lack of semantic error correction, inefficient feature fusion, and insufficient decision robustness in existing technologies, this invention provides an underwater acoustic target recognition system and method based on multimodal deep feature fusion, which achieves high-precision and high-robust recognition of underwater acoustic targets in complex marine environments.

[0009] A multimodal deep feature fusion-based underwater acoustic target recognition system includes:

[0010] Multimodal data acquisition and construction module: Acquires the raw radiative noise data and associated metadata of the target to be identified, performs preprocessing, slicing, and multimodal feature extraction to form a multimodal dataset;

[0011] Multi-branch deep feature encoding module: includes a parallel-running temporal encoding network, a cepstral texture encoding network, and a text semantic encoding network, which respectively extract acoustic long-range dependency representations, cepstral detail representations, and high-dimensional semantic prior embeddings;

[0012] A cross-modal feature fusion module is used to achieve deep alignment and aggregation of heterogeneous features. This module is equipped with a latent query fusion unit, which initializes a set of modality-independent learnable latent query vectors as query ends, and uses the acoustic temporal representation, cepstral texture representation, and semantic prior representation output by the multi-branch feature encoding module as key ends and value ends. The attention weights are calculated using a multi-head cross-attention mechanism to drive the latent query vectors to adaptively aggregate multi-source complementary information in a unified feature subspace, and output a fixed-dimensional multimodal fusion feature.

[0013] A hybrid expert decision-making module is used for adaptive reasoning and classification based on the multimodal fusion features. This module is configured with a hybrid expert reasoning unit, which includes several expert subnetworks with shared or independent parameters and a routing network. The routing network calculates the routing probability distribution based on the input fusion features, selects some expert subnetworks to participate in the calculation using a dynamic sparse activation strategy, and performs weighted aggregation on the outputs of the activated experts. Finally, the classification layer maps and outputs the category probability distribution and recognition result of the underwater acoustic target.

[0014] Model training module: Configures the loss function calculation unit, trains the encoding network and decision module using the labeled multimodal dataset, and updates the weight parameters through the backpropagation algorithm until the model converges;

[0015] Reasoning and Recognition Module: Loads the trained model, receives the data to be recognized and converts it into multimodal feature input, calculates the posterior probability distribution of the target category through forward inference, and outputs the ship target type.

[0016] A method for identifying underwater acoustic targets based on multimodal deep feature fusion includes the following steps:

[0017] Step 1: Data Acquisition: Acquire the raw audio data and associated metadata of the underwater acoustic target, and organize the audio samples in a unified manner;

[0018] Step 2: Multimodal Feature Construction: The audio samples obtained in Step 1 are processed in the order of denoising, pre-emphasis, decentering, and normalization, and then sliced ​​into fixed-length segments to construct multimodal input features; logarithmic Mel spectrogram features reflecting temporal evolution and MFCC features reflecting cepstral texture structure are extracted respectively; based on metadata information, natural language text description information containing environmental context and target attributes is generated using structured templates, which together with the acoustic features constitute an acoustic-semantic multimodal dataset;

[0019] Step 3: Multi-branch feature encoding: Construct a multi-branch feature encoding network containing an acoustic temporal branch, a cepstral texture branch, and a text semantic branch; input the multimodal dataset constructed in Step 2 into the multi-branch feature encoding network to obtain the deep representation of each modality. The acoustic temporal branch adopts a cascaded Conformer encoding module structure to extract the long-range temporal dependence and local time-frequency details of the Mel features; the cepstral texture branch adopts a convolutional neural network integrating a CBAM attention enhancement module to extract the local texture and spectral envelope structure of the MFCC features.

[0020] Step 4: Latent Query-Guided Feature Fusion and Hybrid Expert Dynamic Decision-Making: The multi-branch deep representations obtained in Step 3 undergo latent query-guided cross-modal fusion to obtain unified fused features and complete the recognition decision. Multiple learnable latent query vectors are constructed as query inputs, with multimodal representations as key and value inputs. A cross-attention mechanism is used to achieve alignment and complementary information aggregation between heterogeneous modalities, resulting in a fused feature representation. Subsequently, the fused features are input into the hybrid expert dynamic decision-making layer. A TopK strategy is used to select several experts to participate in inference, and the expert outputs are weighted and converged to obtain the target category probability distribution and output the recognition result. Learnable latent query vectors are initialized, using these latent query vectors as query terms and the three modal features extracted in Step 2 as key and value terms. A multi-head cross-attention mechanism maps the heterogeneous features to a unified latent subspace, and alignment and complementary fusion yield a multimodal joint representation.

[0021] Specifically, a learnable latent query vector is initialized, with the latent query vector as the query term and the three types of modal features extracted in step two as the key and value terms. The heterogeneous features are mapped to a unified latent subspace through a multi-head cross-attention mechanism, and a multimodal joint representation is obtained through alignment and complementary fusion. A hybrid expert classification decision layer is constructed, which includes a routing network and N expert subnetworks. The routing network generates routing weights for each expert subnetwork based on the fusion representation in step three. The top K expert subnetworks with the highest routing weights are selected for parallel inference. After the expert outputs are weighted and aggregated according to the routing weights, the final ship type identification result is output through the classification layer.

[0022] Step 5: Based on the training samples, train and update the parameters of the multi-branch feature encoding network in Step 3, the fusion features in Step 4, and the hybrid expert dynamic decision layer until the model converges; after training, process the underwater acoustic samples to be identified into a multimodal input form consistent with the training stage, and input it into the trained model for inference to obtain the recognition result of the underwater acoustic target.

[0023] The specific implementation process of processing the audio samples obtained in step one in step two according to the order of denoising, pre-emphasis, decentering, and normalization is as follows:

[0024] S1.1 Spectrum Threshold Denoising: A short-time Fourier transform is performed on the acquired discrete audio sequence of ship radiated noise to obtain the time-frequency complex spectrum and the corresponding amplitude spectrum; the background noise amplitude spectrum is estimated based on the statistical characteristics of the amplitude spectrum, the ratio of the two is calculated and weighted by threshold coefficients to generate an initial gain, and the initial gain is smoothed and filtered in the time-frequency domain to obtain a time-frequency threshold mask; the mask suppression depth is adjusted by the attenuation intensity coefficient, the adjusted mask is multiplied by the time-frequency complex spectrum, and finally the denoised time-domain signal is reconstructed by inverse short-time Fourier transform;

[0025] S1.2. Pre-emphasis processing: A first-order high-pass filter is used to calculate the difference between the signal amplitude of each sampling point in the audio sequence and the amplitude of the previous sampling point after weighting by a pre-emphasis coefficient. The value of the pre-emphasis coefficient ranges from 0.90 to 0.98.

[0026] S1.3. Normalization Processing: Using the maximum and minimum normalization strategy, the maximum and minimum amplitudes of the audio sequence are statistically analyzed. The difference between each sampling point and the minimum amplitude is calculated as the numerator, and the difference between the maximum and minimum amplitudes plus a numerical stability term is used as the denominator. The numerator is divided by the denominator to obtain the normalized output.

[0027] The extraction process of Mel spectral features and MFCC features in step two is as follows:

[0028] A1.1. Mel Spectrum Feature Extraction: The power spectrum is obtained by performing a short-time Fourier transform on the preprocessed audio signal, and then mapped to the Mel scale by frequency domain weighted filtering through the Mel filter bank. After adding a numerical stabilization term to the obtained Mel frequency band energy, a logarithmic operation is performed to obtain a two-dimensional logarithmic Mel spectrogram.

[0029] A1.2. MFCC Feature Extraction: Perform discrete cosine transform on each time frame of the log-Mel spectrogram to remove inter-band correlation, and retain the first N-dimensional transform coefficients to construct the MFCC feature matrix, where N is a positive integer;

[0030] A1.3. Constructing text semantic features: Based on the target's meta-information, design text descriptions and generate corresponding text description sequences.

[0031] The specific implementation process of the acoustic temporal branch in step three is as follows: Mel spectral features are input into the Conformer coding network according to the time frame sequence. Each Conformer coding module contains a self-attention submodule and a convolutional submodule. The self-attention submodule models the global time dependency relationship, and the convolutional submodule extracts local time-frequency details. The outputs of each layer are passed in sequence to form a deep temporal representation, and a fixed-length temporal representation vector is obtained through pooling and linear mapping.

[0032] The specific implementation process of the cepstral texture branch in step three is as follows: the MFCC feature matrix is ​​regarded as a two-dimensional image input convolutional network, and is encoded layer by layer through a multi-level convolutional feature extraction unit; each extraction unit includes two two-dimensional convolutional layers, a CBAM attention enhancement module, a max pooling layer and a random deactivation layer. After the feature map is converted into a one-dimensional vector by the flattening layer, a fixed-length representation vector is output through the fully connected layer.

[0033] Step four specifically includes:

[0034] S4.1. Based on the temporal representation, cepstral representation and semantic representation obtained in step three, a cross-modal fusion and decision module is constructed. The fusion layer initializes a set of learnable potential query vectors. Through the cross-attention mechanism, multi-source heterogeneous features are deeply aligned and complementary information is aggregated in a unified subspace to obtain unified fusion features.

[0035] S4.2. The decision layer constructs a hybrid expert module containing a gating network and multiple expert sub-networks. It uses fusion features to drive the gating network to generate routing weights, adaptively selects several experts to participate in reasoning, and weights and aggregates the expert outputs. Finally, it outputs the probability distribution of the target category through the Softmax layer.

[0036] Step five specifically includes:

[0037] The multimodal joint neural network is trained using the dataset constructed in step two. Acoustic and textual features are input into the network in parallel, and the parameters of each branch and the fusion decision layer are jointly updated through backpropagation until the model converges on the validation set, resulting in a trained ship radiated noise recognition model. In the model application phase, the ship radiated noise data to be identified undergoes the same preprocessing, feature extraction, and text construction process as in the training phase, and is then input into the trained recognition model. Through multi-branch encoding, latent query fusion, and hybrid expert inference, the ship type recognition result corresponding to the ship radiated noise is output.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] More complete and accurate feature representation: Through the three-branch architecture of "acoustic temporal sequence - cepstral texture - text semantics", the Conformer branch captures the long-range temporal dependence of Mel features, the CNN branch mines the cepstral texture and spectral envelope structure of MFCC, and the text branch introduces semantic priors, realizing the deep integration of physical features and knowledge guidance, solving the problem of missing information in single-modal and single-dimensional modeling, and making feature expression more comprehensive.

[0040] Strong adaptability to high noise environment: The preprocessing process, such as spectrum threshold denoising and pre-emphasis, effectively suppresses background noise. Combined with the semantic prior error correction mechanism, it can still maintain stable recognition performance in low signal-to-noise ratio scenarios, solving the problem of decreased recognition accuracy caused by acoustic feature blurring under strong noise.

[0041] Excellent cross-scenario generalization ability: The hybrid expert dynamic decision-making mechanism adaptively selects the optimal combination of expert sub-networks through the routing network, which can accurately fit the feature distribution differences under different sea conditions and distances, significantly alleviate the overfitting problem caused by "large intra-class differences and high inter-class similarity", and greatly improve the robustness of cross-scenario recognition.

[0042] Significantly improved recognition accuracy: The cross-modal cross-attention mechanism based on latent queries achieves fine-grained alignment and deep interaction of heterogeneous features, maximizing the multi-source information gain. Experimental results show that the recognition accuracy reaches 98.61% and 99.36% on ShipsEar and DeepShip public datasets, respectively, which is significantly improved compared with existing single-modal or simple fusion schemes.

[0043] Highly practical for engineering applications: The preprocessing process, feature extraction method, and network architecture of this invention are all adapted to engineering deployment requirements, and the model training and inference efficiency is high, making it widely applicable in practical scenarios such as marine situational awareness and underwater security. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the underwater acoustic target recognition system and method based on multimodal deep feature fusion according to the present invention.

[0046] Figure 2 This is a schematic diagram illustrating the framework for constructing a dataset for an underwater acoustic target recognition system and method based on multimodal deep feature fusion, as described in this invention.

[0047] Figure 3 This is a schematic diagram of the construction of the recognition network model for an underwater acoustic target recognition system and method based on multimodal deep feature fusion according to the present invention;

[0048] Figure 4 This is a schematic diagram of the recognition network used in the underwater acoustic target recognition system and method based on multimodal deep feature fusion according to the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1:

[0051] Reference Figure 1 A method for underwater acoustic target recognition using multimodal deep feature fusion, comprising:

[0052] Step 1: Data Acquisition: Acquire the raw audio data and associated metadata of the underwater acoustic target, and uniformly organize the audio samples; the metadata is used to characterize the contextual information of the target, providing a foundation for subsequent semantic description construction. The acquired audio samples are standardized in format and sampling rate, and outlier samples are removed to ensure consistency and stability in subsequent processing.

[0053] Step 2: Multimodal Feature Construction: The audio samples obtained in Step 1 are processed in the order of denoising, pre-emphasis, decentering, and normalization, and then sliced ​​into fixed-length segments to construct multimodal input features; logarithmic Mel spectrogram features reflecting temporal evolution and MFCC features reflecting cepstral texture structure are extracted respectively; based on metadata information, natural language text description information containing environmental context and target attributes is generated using structured templates, which together with the acoustic features constitute an acoustic-semantic multimodal dataset;

[0054] The audio samples obtained in step one are preprocessed and sliced ​​to construct multimodal input features. Preprocessing may include, but is not limited to, spectral threshold denoising, pre-emphasis, and normalization. Sampling may include fixed-length slicing, framing, and windowing to form sample segments of uniform length. Further, logarithmic Mel-frequency spectroscopic features and Mel-frequency cepstral coefficient features are extracted from each sample segment, serving as acoustic inputs reflecting the signal's time-frequency energy and cepstral texture, respectively. Simultaneously, a corresponding text description is generated based on the associated metadata, transforming discrete information into natural language descriptions, thus forming a multimodal input composed of both acoustic and semantic features.

[0055] Step 3: Multi-branch feature encoding: Construct a multi-branch feature encoding network containing an acoustic temporal branch, a cepstral texture branch, and a text semantic branch; input the multimodal dataset constructed in Step 2 into the multi-branch feature encoding network to obtain the deep representation of each modality. The acoustic temporal branch adopts a cascaded Conformer encoding module structure to extract the long-range temporal dependence and local time-frequency details of the Mel features; the cepstral texture branch adopts a convolutional neural network integrating a CBAM attention enhancement module to extract the local texture and spectral envelope structure of the MFCC features.

[0056] The multimodal samples constructed in step two are input into a multi-branch feature encoding network to obtain deep representations of each modality. Preferably, log-Mel spectrum features are input into a temporal encoding network to model the global temporal correlation of acoustic signals and extract local time-frequency details, outputting an acoustic temporal representation; Mel frequency cepstral coefficient features are input into a convolutional network to extract cepstral domain texture structure and energy envelope information, outputting a cepstral texture representation; and textual description information is input into a pre-trained text encoder to obtain semantic embedding representations. These multi-branch representations are used together for subsequent cross-modal fusion and decision-making.

[0057] Step 4: Latent Query-Guided Feature Fusion and Hybrid Expert Dynamic Decision-Making: The multi-branch deep representations obtained in Step 3 undergo latent query-guided cross-modal fusion to obtain unified fused features and complete the recognition decision. Multiple learnable latent query vectors are constructed as query inputs, with multimodal representations as key and value inputs. A cross-attention mechanism is used to achieve alignment and complementary information aggregation between heterogeneous modalities, resulting in a fused feature representation. Subsequently, the fused features are input into the hybrid expert dynamic decision-making layer. A TopK strategy is used to select several experts to participate in inference, and the expert outputs are weighted and converged to obtain the target category probability distribution and output the recognition result. Learnable latent query vectors are initialized, using these latent query vectors as query terms and the three modal features extracted in Step 2 as key and value terms. A multi-head cross-attention mechanism maps the heterogeneous features to a unified latent subspace, and alignment and complementary fusion yield a multimodal joint representation.

[0058] Specifically, a learnable latent query vector is initialized, with the latent query vector as the query term and the three types of modal features extracted in step two as the key and value terms. The heterogeneous features are mapped to a unified latent subspace through a multi-head cross-attention mechanism, and a multimodal joint representation is obtained through alignment and complementary fusion. A hybrid expert classification decision layer is constructed, which includes a routing network and N expert subnetworks. The routing network generates routing weights for each expert subnetwork based on the fusion representation in step three. The top K expert subnetworks with the highest routing weights are selected for parallel inference. After the expert outputs are weighted and aggregated according to the routing weights, the final ship type identification result is output through the classification layer.

[0059] Step 5: Based on the training samples, train and update the parameters of the multi-branch feature encoding network in Step 3, the fusion features in Step 4, and the hybrid expert dynamic decision layer until the model converges; after training, process the underwater acoustic samples to be identified into a multimodal input form consistent with the training stage, and input it into the trained model for inference to obtain the recognition result of the underwater acoustic target.

[0060] like Figure 2 The diagram shows the framework for constructing the dataset. Step two specifically includes:

[0061] S2.1. Preprocess the acquired raw audio of ship radiated noise; wherein the preprocessing steps are performed in the order of denoising, pre-emphasis, decentering and normalization, in order to suppress background noise and enhance the effective radiated noise characteristics.

[0062] The raw audio data of ship radiated noise collected is subjected to basic standardization processing. First, a resampling operation is performed to unify audio from different sources to a preset standard sampling rate. Then, decentering processing is performed by subtracting the signal mean to eliminate the DC component and calibrate the signal zero point. Further, normalization processing is performed on the decentered audio, preferably using the maximum-minimum normalization method to linearly map the audio amplitude to the [0,1] interval, so as to unify the signal amplitude range and reduce the impact of the difference in dimensions between samples on subsequent training.

[0063] Furthermore, step S2.1 specifically includes:

[0064] S2.11. To suppress broadband background noise and weak amplitude interference components in ship radiated noise data, a spectral threshold denoising method is adopted. This method obtains the time-frequency representation of the signal through short-time Fourier transform, obtains the noise spectrum based on the noise reference segment or adaptive noise estimation, constructs a time-frequency threshold mask, attenuates time-frequency components below the threshold, and finally reconstructs the denoised time-domain signal through inverse transform.

[0065] Its calculation relationship can be expressed as:

[0066]

[0067]

[0068]

[0069]

[0070] in, The input is a discrete sequence of ship radiated noise. Its short-time Fourier transform result, Amplitude spectrum; For noise amplitude spectrum estimation; For threshold mask, This indicates smoothing in both the time and frequency directions; For threshold coefficients, The attenuation intensity coefficient, It is a stable term; This is the denoised time-domain output signal. By using spectral threshold denoising and time-frequency smoothing, the interference of background noise on subsequent feature extraction and recognition can be reduced, and the artifacts introduced by threshold suppression can be reduced.

[0071] S2.12. The audio signal after denoising in step S2.1 is pre-emphasized to compensate for high-frequency attenuation and enhance the relative energy of high-frequency components, making the signal more balanced in the frequency domain and improving the stability of subsequent feature extraction. The calculation relationship can be expressed as:

[0072]

[0073] in, For the pre-emphasized input sequence, For pre-emphasized output sequence, This is the pre-emphasis coefficient, with a value range of 0.90 to 0.98.

[0074] S2.13. The pre-emphasized audio signal from step S2.12 is decentered to eliminate any potential DC component bias, calibrate the signal zeros, and prevent the DC component from affecting the accuracy of subsequent feature extraction. The calculation relationship can be expressed as:

[0075]

[0076] in, The input sequence is after pre-emphasis. The total length of the sequence. For the summation index, This is the output sequence after decentralization.

[0077] S2.14. Normalize the decentralized audio signal from step S2.13 to unify the amplitude dimensions and reduce the impact of energy differences between different samples on model training. The calculation relationship can be expressed as:

[0078]

[0079] in, For the sequence to be normalized, and These are the minimum and maximum values ​​of the sequence, respectively. As a stable term, This is the normalized output sequence. Normalization helps avoid a small number of high-amplitude samples dominating parameter updates during training, thus improving training stability.

[0080] S2.2. Perform sample slicing and frame-by-frame windowing processing on the audio signal output from step S2.1 to form uniform sample segments suitable for subsequent time-frequency feature construction. The slice duration is set to... seconds, step size seconds, of which ; Take 3 seconds; optional settings include overlapping slices to amplify sample size, for example... Framing and windowing are performed on each slice sample to reduce spectral leakage and enhance feature stability.

[0081] S2.3. Extracting Mel Spectral Features: The preprocessed signal undergoes a short-time Fourier transform and is mapped to the Mel scale using a Mel filter bank. Taking the logarithm yields a two-dimensional Mel spectrogram, which serves as the input for the time-series branch. The calculation relationship for the logarithmic Mel spectrum can be expressed as:

[0082]

[0083] in, This is the result of the short-time Fourier transform. For the first Mel filter weights, It is a stable term.

[0084] S2.4. Extracting MFCC Features: Perform a discrete cosine transform on the Mel spectrum to construct the MFCC feature matrix, which serves as the input for the cepstral texture branch. The calculation relationship of MFCC can be expressed as:

[0085]

[0086] in, For logarithmic Mel-energy, This is the cepstral coefficient index.

[0087] For each obtained audio slice, multimodal input features, including acoustic features and semantic information, are constructed in parallel. On one hand, Mel spectrogram features and Mel frequency cepstral coefficient features are extracted from the slices respectively. The Mel spectrogram is used to represent the energy distribution in the time-frequency domain, and the Mel frequency cepstral coefficients are used to represent the cepstral texture structure and spectral envelope information. Both serve as inputs to the acoustic branch. On the other hand, a structured text description is constructed based on the metadata associated with the audio samples, and text prompts are generated and encoded into a high-dimensional semantic information vector as input to the semantic branch.

[0088] S2.5. Constructing Text Semantic Features: Design structured text descriptions based on the target's meta-information and encode them into corresponding semantic features. Label the obtained multimodal samples according to categories, and randomly shuffle and mix samples from different categories. Finally, divide the dataset into training, validation, and test sets in a 7:1.5:1.5 ratio for subsequent training, optimization, and performance evaluation of the multimodal fusion model.

[0089] like Figure 3 The diagram shown illustrates the construction of the recognition network model. Step three specifically includes:

[0090] S3.1. A time series model is used to process time series features. Its self-attention mechanism is used to capture global time correlations, while its internal convolution module is used to extract local time-frequency details.

[0091] S3.2. A convolutional network is used to process the feature matrix of a two-dimensional image and extract the texture structure and energy envelope features of the cepstral domain.

[0092] S3.3. A pre-trained text encoder with frozen parameters is used to convert text description information into high-dimensional semantic embedding vectors as prior knowledge guidance.

[0093] S3.4. Construct a joint underwater acoustic target recognition network based on temporal model, visual model and text model.

[0094] S3.5. Input the ship radiated noise dataset into the ship radiated noise recognition network for training to obtain a trained underwater acoustic target recognition network.

[0095] Furthermore, step S3.1 specifically includes:

[0096] S3.11. Input the Mel spectrogram features obtained in step two into the Conformer coding network in the form of time frame sequences for feature encoding; the Conformer coding network is composed of cascaded Conformer coding modules, each Conformer coding module includes a self-attention sub-module and a convolutional sub-module: the self-attention sub-module is used to model the global time dependency of the Mel spectrogram sequence, and the convolutional sub-module is used to extract local time-frequency detail features; the outputs of each layer are passed in sequence to form a deep temporal representation, and the final output is pooled and linearly mapped to obtain a fixed-length temporal representation vector for subsequent cross-modal fusion.

[0097] Furthermore, step S3.2 specifically includes:

[0098] S3.21. Input the extracted Mel spectrum data into the visual neural network.

[0099] S3.22. The input MFCC feature matrix is ​​encoded using a convolutional network. The MFCC feature matrix is ​​treated as a two-dimensional image input network and encoded layer by layer through multiple convolutional feature extraction units. Each convolutional feature extraction unit includes two layers of two-dimensional convolution, a CBAM attention enhancement module, a max pooling layer, and a random deactivation layer. The two-dimensional convolution is used to extract local texture and spectral envelope structure features in the cepstral domain. CBAM is used to adaptively emphasize key channels and spatial regions. Max pooling is used for downsampling to reduce feature scale and computational cost. Random deactivation is used to suppress overfitting. After completing the multi-level feature extraction, the feature map is converted into a one-dimensional vector through a flattening layer and a fixed-length representation vector of the MFCC branch is output through a fully connected layer for subsequent cross-modal fusion and decision-making.

[0100] Furthermore, the feature fusion guided by the potential query in step four specifically involves:

[0101] A set of learnable latent query vectors is initialized and used as queries. The temporal features, cepstral features, and semantic features extracted in step two are concatenated as keys and values. The latent query vectors are iteratively updated through a multi-head cross-attention mechanism, thereby achieving automatic alignment and compression of multimodal information in the compressed latent space and outputting a fixed-dimensional fusion feature vector.

[0102] The basic computational relationship of multi-head cross-attention can be expressed as:

[0103]

[0104] in, This is a query matrix used to represent the information to be retrieved; The key matrix is ​​used for... Calculate the correlation; is a value matrix used for weighted summation based on attention weights; d is a scaling factor, representing the feature dimension corresponding to each attention head; The normalization function is used to ensure that the attention weights are non-negative and sum to 1 in the Key dimension; This is a correlation matrix; This represents the output feature representation after attention aggregation.

[0105] Potential query fusion can be represented as:

[0106]

[0107] in, For the potential query matrix, by Composed of learnable potential query vectors; Number of potential queries; Feature dimensions for each potential query vector; and The feature representations from multiple modal branches serve as the information source for retrieval and aggregation; For cross-attention operators, the above-mentioned method is preferred. The computational form is implemented; This is the set of potential representations after fusion, used for subsequent pooling or projection to obtain the fused vector.

[0108] Furthermore, the hybrid expert dynamic decision-making in step four specifically involves:

[0109] Construct a set of N expert networks with identical structures and a routing network. The routing network receives the fusion features output from step three, calculates the routing weight for each expert network, and then performs a weighted sum of the outputs of all expert networks based on the routing weights, passing the sum through a Softmax layer to obtain the final classification probability distribution.

[0110] The routing weights, TopK aggregation outputs, and classification probabilities of the hybrid experts can be expressed as follows:

[0111]

[0112]

[0113]

[0114] in, The feature vectors are fused and used as input to the hybrid expert decision layer; This is a routing network parameter matrix, used to... Mapped to the routing scores of each expert and then... Obtain the route weight vector ; Indicates from Select the one with the largest weight A collection of expert indexes; For the first A network of experts; The output representation is obtained by weighted aggregation of the outputs of activated experts. To output the mapping parameter matrix; This represents the final category probability distribution.

[0115] Step five specifically includes:

[0116] The multimodal joint neural network is trained using the training set constructed in step two. Acoustic features and text features are input into the network in parallel, and the parameters of each branch and the fusion decision layer are jointly updated through backpropagation until the model converges on the validation set to obtain the trained ship radiated noise recognition model. In the model application stage, the ship radiated noise data to be identified is preprocessed, feature extracted and text constructed in the same way as in the training stage, and then input into the trained recognition model. Through multi-branch encoding, latent query fusion and hybrid expert inference, the ship type recognition result corresponding to the ship radiated noise is output.

[0117] like Figure 4 The diagram shows a network model used for recognition.

[0118] The following are the recognition results of the multimodal deep feature fusion network model built in a specific embodiment:

[0119] The experimental environment used in this embodiment is as follows: For hardware, the GPU is an NVIDIA V100 32GB, and the CPU is an Intel Xeon Gold 6130 with a clock speed of 2.10GHz. For software, the operating system is Ubuntu 22.04, the deep learning framework is PyTorch 2.5.1, the programming language is Python 3.12, and the CUDA 12.4 acceleration library is used to improve training and inference efficiency.

[0120] To comprehensively evaluate the classification performance of the model, this invention selects accuracy, precision, recall, and F1 score as evaluation metrics.

[0121] An end-to-end joint optimization strategy was adopted during model training, performing overall training and collaborative updates for multi-branch encoding, cross-modal fusion, and hybrid expert decision-making modules. For hyperparameters, the batch size for the ShipsEar dataset was set to 32, and for the DeepShip dataset to 512; the total number of training epochs was set to 100; the AdamW optimizer was selected, with a base learning rate of 1E-4, and the classification cross-entropy function was used as the loss function. Each sample segment was processed for 3 seconds, with no overlap between segments, and the training, validation, and test sets were constructed in a 7:1.5:1.5 ratio.

[0122] The recognition results of the multimodal deep feature fusion model on two public datasets are shown in Tables 1 and 2 below:

[0123] Table 1 Experimental Results

[0124]

[0125] Table 2 Experimental Results

[0126]

[0127] The quantitative analysis of experimental data from the above embodiments shows that the multimodal deep feature fusion network proposed in this invention achieves recognition accuracies of 98.61% and 99.36% on two public datasets, respectively, and can perform classification tasks well.

[0128] This embodiment has the following significant and beneficial technical effects:

[0129] (1) This invention does not simply superimpose temporal and visual models, but innovatively constructs a three-branch architecture of "temporal-cepstrum-semantic". By capturing the long-range temporal dependence of the Mel spectrum through the Conformer branch and mining the cepstrum texture structure of MFCC through the CNN branch, complementary modeling of global correlation and local details of underwater acoustic signals is realized, which solves the problem of missing single acoustic feature information from the physical level.

[0130] (2) Unlike traditional methods that rely solely on data-driven approaches, this invention utilizes a pre-trained text encoder to introduce high-dimensional semantic priors. This "knowledge-guided" mechanism provides contextual constraints for the model, enabling it to correct the reasoning direction based on semantic cues when acoustic features are obscured by strong noise.

[0131] (3) This invention abandons the traditional simple weighted fusion strategy and designs a cross-modal cross-attention mechanism based on latent queries. By adaptively aggregating multimodal features in a unified subspace through learnable latent vectors, fine-grained alignment and deep interaction of heterogeneous information are achieved, maximizing the gain of multi-source information.

[0132] (4) To address the non-stationary distribution characteristics of underwater acoustic targets, this invention integrates a hybrid expert structure at the decision-making level. Through a dynamic routing strategy, the model can automatically activate the most suitable expert network combination based on the characteristics of the input signal. This "dynamic division of labor" mechanism not only improves the model's parameter capacity and fitting ability but also effectively mitigates the risk of overfitting in complex scenarios, balancing accuracy and generalization.

[0133] Example 2: System Architecture

[0134] This embodiment provides an underwater acoustic target recognition system based on multimodal basic model fusion, which is implemented based on the method described in Embodiment 1. The system mainly includes the following core functional modules:

[0135] The multimodal data acquisition and construction module is used to acquire the raw radiated noise data and associated metadata of the target to be identified, and to preprocess the acquired raw data to obtain a dataset of ship radiated noise.

[0136] The multi-branch deep feature encoding module is used for deep feature extraction from multimodal inputs. This module contains three sub-networks running in parallel: a temporal encoding network for extracting long-range acoustic dependency representations; a cepstral texture encoding network for extracting cepstral domain detail representations; and a text semantic encoding network for extracting high-dimensional semantic prior embeddings.

[0137] A cross-modal feature fusion module is used to achieve deep alignment and aggregation of heterogeneous features. This module is equipped with a latent query fusion unit, which initializes a set of modality-independent learnable latent query vectors as query ends, and uses the acoustic temporal representation, cepstral texture representation, and semantic prior representation output by the multi-branch feature encoding module as key ends and value ends. The attention weights are calculated using a multi-head cross-attention mechanism to drive the latent query vectors to adaptively aggregate multi-source complementary information in a unified feature subspace, and output a fixed-dimensional multimodal fusion feature.

[0138] A hybrid expert decision-making module is used for adaptive reasoning and classification based on the multimodal fusion features. This module is configured with a hybrid expert reasoning unit, which includes several expert subnetworks with shared or independent parameters and a routing network. The routing network calculates the routing probability distribution based on the input fusion features, selects some expert subnetworks to participate in the calculation using a dynamic sparse activation strategy, and performs weighted aggregation on the outputs of the activated experts. Finally, the classification layer maps and outputs the category probability distribution and recognition result of the underwater acoustic target.

[0139] The model training module is used to train the aforementioned network using a labeled multimodal underwater acoustic dataset. This module is equipped with a loss function calculation unit and updates the weight parameters of each encoding network and decision module through the backpropagation algorithm until the model converges.

[0140] The reasoning and recognition module is configured to load the trained multimodal fusion network model, receive the underwater acoustic target data to be identified and convert it into multimodal feature input; by performing forward inference operations of the model, it calculates the posterior probability distribution of each target category and finally outputs the type of ship target.

[0141] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A hydroacoustic target recognition system based on multimodal deep feature fusion, characterized in that: include: Multimodal data acquisition and construction module: Acquires the raw radiative noise data and associated metadata of the target to be identified, performs preprocessing, slicing, and multimodal feature extraction to form a multimodal dataset; Multi-branch deep feature encoding module: includes a parallel-running temporal encoding network, a cepstral texture encoding network, and a text semantic encoding network, which respectively extract acoustic long-range dependency representations, cepstral detail representations, and high-dimensional semantic prior embeddings; The cross-modal feature fusion module is used to achieve deep alignment and aggregation of heterogeneous features. It is equipped with a latent query fusion unit, which initializes a set of modality-independent learnable latent query vectors as the query end, and uses the acoustic temporal representation, cepstral texture representation, and semantic prior representation output by the multi-branch deep feature encoding module as the key end and value end. It uses a multi-head cross-attention mechanism to calculate attention weights, drive the latent query vectors to adaptively aggregate multi-source complementary information in a unified feature subspace, and output a fixed-dimensional multimodal fusion feature. The hybrid expert decision module is used for adaptive reasoning and classification based on multimodal fusion features. It is equipped with a hybrid expert reasoning unit, which includes several expert subnetworks with shared or independent parameters and a routing network. The routing network calculates the routing probability distribution based on the input fusion features, selects some expert subnetworks to participate in the calculation using a dynamic sparse activation strategy, weights and aggregates the outputs of the activated experts, and outputs the category probability distribution and recognition results of the underwater acoustic target through the classification layer. Model training module: Configures the loss function calculation unit, trains the encoding network and decision module using the labeled multimodal dataset, and updates the weight parameters through the backpropagation algorithm until the model converges; Reasoning and Recognition Module: Loads the trained model, receives the data to be recognized and converts it into multimodal feature input, calculates the posterior probability distribution of the target category through forward inference, and outputs the ship target type.

2. A recognition method based on the underwater acoustic target recognition system based on multimodal deep feature fusion as described in claim 1, characterized in that: Includes the following steps: Step 1: Data Acquisition: Acquire the raw audio data and associated metadata of the underwater acoustic target, and organize the audio samples in a unified manner; Step 2: Multimodal feature construction: The audio samples obtained in Step 1 are processed in the order of denoising, pre-emphasis, decentering and normalization, and then sliced ​​into fixed lengths to construct multimodal input features; Log-Melbourne spectral features reflecting temporal evolution and MFCC features reflecting cepstral texture structure were extracted respectively. Based on metadata information, natural language text description information containing environmental context and target attributes is generated using structured templates, which together with acoustic features constitute an acoustic-semantic multimodal dataset. Step 3: Multi-branch feature encoding: Construct a multi-branch feature encoding network containing an acoustic temporal branch, a cepstral texture branch, and a text semantic branch; input the multimodal dataset constructed in Step 2 into the multi-branch feature encoding network to obtain the deep representation of each modality. The acoustic temporal branch adopts a cascaded Conformer encoding module structure to extract the long-range temporal dependence and local time-frequency details of the Mel features; the cepstral texture branch adopts a convolutional neural network integrating a CBAM attention enhancement module to extract the local texture and spectral envelope structure of the MFCC features. Step 4: Latent Query-Guided Feature Fusion: The multi-branch deep representations obtained in Step 3 are subjected to latent query-guided cross-modal fusion to obtain unified fused features and complete the recognition decision. Multiple learnable latent query vectors are constructed and used as query inputs. Multimodal representations are used as key and value inputs. The alignment and complementary information aggregation between heterogeneous modalities are achieved through a cross-attention mechanism to obtain the fused feature representation. Subsequently, the fused features are input into the hybrid expert dynamic decision layer. The TopK strategy is used to select several experts to participate in reasoning, and the expert outputs are weighted and converged to obtain the target category probability distribution and output the recognition result. The learnable latent query vectors are initialized. The latent query vectors are used as query terms, and the three types of modal features extracted in Step 2 are used as key and value terms. The heterogeneous features are mapped to a unified latent subspace through a multi-head cross-attention mechanism. After alignment and complementary fusion, a multimodal joint representation is obtained. Step 5: Based on the training samples, train and update the parameters of the multi-branch feature encoding network in Step 3, the fusion features in Step 4, and the hybrid expert dynamic decision layer until the model converges; after training, process the underwater acoustic samples to be identified into a multimodal input form consistent with the training stage, and input it into the trained model for inference to obtain the recognition result of the underwater acoustic target.

3. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: The specific implementation process of processing the audio samples obtained in step one in step two according to the order of denoising, pre-emphasis, decentering, and normalization is as follows: S1.1 Spectrum Threshold Denoising: Perform a short-time Fourier transform on the original discrete audio sequence of the acquired ship radiated noise to obtain the time-frequency complex spectrum and the corresponding amplitude spectrum; estimate the background noise amplitude spectrum based on the statistical characteristics of the amplitude spectrum, calculate the ratio between the two and generate an initial gain by weighted scaling with threshold coefficients, and perform time-frequency domain smoothing filtering on the initial gain to obtain a time-frequency threshold mask. The mask suppression depth is adjusted by the attenuation intensity coefficient, the adjusted mask is multiplied by the time-frequency complex spectrum, and finally the denoised time domain signal is reconstructed by inverse short-time Fourier transform. S1.

2. Pre-emphasis processing: A first-order high-pass filter is used to calculate the difference between the signal amplitude of each sampling point in the audio sequence and the amplitude of the previous sampling point after being weighted by the pre-emphasis coefficient; S1.

3. Normalization Processing: Using the maximum and minimum normalization strategy, the maximum and minimum amplitudes of the audio sequence are statistically analyzed. The difference between each sampling point and the minimum amplitude is calculated as the numerator, and the difference between the maximum and minimum amplitudes plus a numerical stability term is used as the denominator. The numerator is divided by the denominator to obtain the normalized output.

4. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: The extraction process of Mel spectral features and MFCC features in step two is as follows: A1.

1. Mel Spectrum Feature Extraction: The power spectrum is obtained by performing a short-time Fourier transform on the preprocessed audio signal, and then mapped to the Mel scale by frequency domain weighted filtering through the Mel filter bank. After adding a numerical stabilization term to the obtained Mel frequency band energy, a logarithmic operation is performed to obtain a two-dimensional logarithmic Mel spectrogram. A1.

2. MFCC Feature Extraction: Perform discrete cosine transform on each time frame of the log-Mel spectrogram to remove inter-band correlations, and retain the first N-dimensional transform coefficients to construct the MFCC feature matrix, where N is a positive integer; A1.

3. Constructing text semantic features: Design structured text descriptions based on the target's meta-information and encode them into corresponding semantic features.

5. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: The specific implementation process of the acoustic temporal branch in step three is as follows: Mel spectral features are input into the Conformer coding network according to the time frame sequence. Each Conformer coding module contains a self-attention submodule and a convolutional submodule. The self-attention submodule models the global time dependency relationship, and the convolutional submodule extracts local time-frequency details. The outputs of each layer are passed in sequence to form a deep temporal representation, and a fixed-length temporal representation vector is obtained through pooling and linear mapping.

6. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: The specific implementation process of the cepstral texture branch in step three is as follows: the MFCC feature matrix is ​​regarded as a two-dimensional image input convolutional network, and is encoded layer by layer through a multi-level convolutional feature extraction unit; each extraction unit includes two two-dimensional convolutional layers, a CBAM attention enhancement module, a max pooling layer and a random deactivation layer. After the feature map is converted into a one-dimensional vector by the flattening layer, a fixed-length representation vector is output through the fully connected layer.

7. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: Step four specifically includes: S4.

1. Based on the temporal representation, cepstral representation and semantic representation obtained in step three, a cross-modal fusion and decision module is constructed. The fusion layer initializes a set of learnable potential query vectors. Through the cross-attention mechanism, multi-source heterogeneous features are deeply aligned and complementary information is aggregated in a unified subspace to obtain unified fusion features. S4.

2. The decision layer constructs a hybrid expert module containing a gating network and multiple expert sub-networks. It uses fusion features to drive the gating network to generate routing weights, adaptively selects several experts to participate in reasoning, and weights and aggregates the expert outputs. Finally, it outputs the probability distribution of the target category through the Softmax layer.

8. The recognition method of an underwater acoustic target recognition system based on multimodal deep feature fusion according to claim 2, characterized in that: Step five specifically includes: The multimodal joint neural network is trained using the dataset constructed in step two. Acoustic features and text features are input into the network in parallel, and the parameters of each branch and the fusion decision layer are jointly updated through backpropagation until the model converges on the validation set to obtain the trained ship radiated noise identification model. In the model application stage, the ship radiated noise data to be identified is preprocessed, feature extracted and text constructed in the same way as in the training stage and then input into the trained identification model. Through multi-branch encoding, latent query fusion and hybrid expert inference, the ship type identification result corresponding to the ship radiated noise is output.

Images