Underwater acoustic signal classification method based on comparative learning and feature fusion

Abstract

The invention discloses an underwater acoustic signal classification method based on comparative learning and feature fusion, and the method comprises the steps: collecting an original underwater acoustic signal, and carrying out the enhancement preprocessing of the original underwater acoustic signal, and obtaining a preprocessed underwater acoustic signal; outputting the preprocessed underwater acoustic signals to a classification neural network model to obtain an underwater acoustic signal classification result; wherein the classification neural network model is obtained according to the following steps: constructing a multi-modal heterogeneous feature fusion network; performing self-supervised comparison pre-training on the constructed multi-modal heterogeneous feature fusion network to obtain a pre-trained multi-modal heterogeneous feature fusion network; a classification module is built, and the classification module takes fusion features output by the multi-mode heterogeneous feature fusion network as input and takes probability distribution of each category as output; and performing progressive supervision fine tuning on the pre-trained multi-modal heterogeneous feature fusion network and the classification module to obtain a classification neural network model.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater acoustic signal intelligent processing, and in particular to an underwater acoustic signal classification method based on contrast learning and feature fusion. Background Art

[0002] With the continuous development of fields such as marine resource development, environmental monitoring, national defense security, and marine biological research, the importance of underwater acoustic signal classification technology has become increasingly prominent. In existing technologies, the processing of underwater acoustic signals mainly relies on traditional manual feature extraction methods, such as short-time Fourier transform, Mel-frequency spectrogram, Mel-frequency cepstral coefficients, and constant Q transform. These methods capture the frequency domain and time domain characteristics of the signal by converting the original one-dimensional time signal into a time-frequency representation. Based on these features, researchers further use traditional machine learning algorithms (such as support vector machines and hidden Markov models) and deep learning models (such as convolutional neural networks and recurrent neural networks) to classify acoustic signals.

[0003] However, existing technologies have many shortcomings and limitations. First, traditional manual feature extraction methods often find it difficult to fully capture the multi-level and rich time-frequency information contained in the signal when faced with complex underwater environments. They are also sensitive to noise and environmental changes, which seriously affects the classification performance. Secondly, although deep learning methods can automatically extract high-level features to a certain extent, their training process relies on a large amount of labeled data. However, data collection and labeling in underwater environments are expensive, and data sets are usually limited, resulting in insufficient model generalization capabilities. In addition, current research often uses a single modality feature representation (such as only using two-dimensional time-frequency spectra or only using one-dimensional time domain signals), which makes it difficult to take into account the complementary advantages of time domain and frequency domain information. Summary of the Invention

[0004] Purpose of the Invention: To address the technical shortcomings of existing underwater acoustic signal classification methods, such as insufficient feature representation capabilities in complex marine environments and a scarcity of annotated data, which leads to poor model generalization, this invention proposes an underwater acoustic signal classification method based on contrastive learning and dynamic feature fusion. This method significantly improves classification accuracy and noise robustness in small sample scenarios through the collaborative enhancement of multimodal features and a self-supervised-supervised dual-stage training mechanism.

[0005] Technical solution: A method for underwater acoustic signal classification based on contrastive learning and feature fusion, including the following steps:

[0006] Step 1: Collect the original underwater acoustic signal and perform enhancement preprocessing on the original underwater acoustic signal to obtain the preprocessed underwater acoustic signal;

[0007] Step 2: Output the preprocessed underwater acoustic signal to the classification neural network model to obtain the underwater acoustic signal classification result;

[0008] The classification neural network model is obtained according to the following steps:

[0009] Construct a multimodal heterogeneous feature fusion network;

[0010] Perform self-supervised comparative pre-training on the constructed multimodal heterogeneous feature fusion network to obtain a pre-trained multimodal heterogeneous feature fusion network;

[0011] Building a classification module that takes the fused features output by the multimodal heterogeneous feature fusion network as input and outputs the probability distribution of each category;

[0012] The pre-trained multimodal heterogeneous feature fusion network and classification module are progressively fine-tuned to obtain a classification neural network model.

[0013] Furthermore, the aforementioned enhancement preprocessing of the original underwater acoustic signal to obtain the preprocessed underwater acoustic signal specifically includes the following operations:

[0014] The original underwater acoustic signal is subjected to noise reduction and amplitude normalization processing in sequence to obtain a standardized underwater acoustic signal;

[0015] The standardized underwater acoustic signal is divided into a plurality of one-dimensional underwater acoustic signal segments of fixed length;

[0016] Perform time-frequency conversion on each one-dimensional underwater acoustic signal segment to obtain a two-dimensional Mel-spectrogram;

[0017] The one-dimensional underwater acoustic signal segment and the corresponding two-dimensional Mel spectrum graph together constitute the preprocessed underwater acoustic signal.

[0018] Furthermore, the construction of the multimodal heterogeneous feature fusion network specifically includes the following operations:

[0019] A one-dimensional feature channel branch module is constructed; the one-dimensional feature channel branch module takes a one-dimensional underwater acoustic signal segment as input, first uses multiple cascaded one-dimensional convolution kernels to perform multi-level downsampling on the input, then uses a one-dimensional convolution layer as the input layer, followed by two consecutive stacked residual connection module stages, and finally obtains the output feature f through a global average pooling operation. 1D ;

[0020] Construct a two-dimensional feature channel branch module; the two-dimensional feature channel branch module takes the two-dimensional Mel spectrum map as input, adopts a two-dimensional convolution layer as the input layer, and then performs a maximum pooling operation to halve the size of the input two-dimensional Mel spectrum map, and then connects multiple continuously stacked multi-branch attention convolution modules, wherein a convolution kernel is used for downsampling after each multi-branch attention convolution module, and finally the output feature f is obtained through a global average pooling operation. 2D ;

[0021] Construct a feature fusion module; the feature fusion module outputs feature f 1D And the output feature f 2D As input, the output feature f 1D And the output feature f 2D Splicing is performed to obtain the initial splicing feature vector, and the output feature f is converted into 1D And the output feature f 2D Perform adaptive dynamic fusion to obtain the weighted fusion feature vector; add the initial splicing feature vector and the weighted fusion feature vector to perform residual connection to obtain the fusion feature f fusion .

[0022] Furthermore, each of the residual connection modules is composed of a plurality of convolutional layers stacked in series, each convolutional layer includes a one-dimensional convolution kernel, a batch normalization layer, and a ReLU activation function. A one-dimensional convolution kernel is added after each convolutional layer, and the input features of the residual connection module are added to the output features of the residual connection module to form a residual connection.

[0023] The two continuously stacked residual connection module stages include a first residual connection module stage and a second residual connection module stage; wherein the first residual connection module stage is composed of three continuously stacked residual connection modules; and the second residual connection module stage is composed of three continuously stacked residual connection modules.

[0024] Furthermore, the multi-branch attention convolution module is constructed according to the following steps:

[0025] Construct multiple two-dimensional convolutional layers of different sizes and the same channels, and synchronously input the input features of the multi-branch attention convolution module to multiple two-dimensional convolutional layers to obtain multiple output features;

[0026] The following operations are performed on each output feature: a channel description vector is extracted from the output features of the two-dimensional convolution layer using a global average pooling operation, the channel is compressed and expanded using two two-dimensional convolution kernels of size 1x1, and finally a weight vector is generated using the Soigmoid function. This weight vector is multiplied channel by channel with the output features of the two-dimensional convolution layer to obtain the channel attention weighted feature.

[0027] The obtained multiple channel attention weighted features are spliced ​​along the channel direction to obtain spliced ​​features;

[0028] A two-dimensional convolution layer is used to compress the splicing features, which are then added to the input features of the multi-branch attention convolution module for residual connection to obtain the output features of the multi-branch attention convolution module.

[0029] Furthermore, the fusion weight network is used to output the feature f 1D And the output feature f 2D Perform adaptive dynamic fusion to obtain the fusion feature f fusion Specific operations include:

[0030] Construct a fully connected layer, followed by a batch normalization layer and a ReLU activation function, and output the feature f 1D And the output feature f 2D Get intermediate representation;

[0031] Construct a fully connected layer as the output layer, connect the Sigmoid activation function to transform the intermediate representation, and output the fusion weight vector a = [a1, a2], where a1 corresponds to the weight of the one-dimensional feature channel branch module, and a2 corresponds to the weight of the two-dimensional feature channel branch module;

[0032] Use the fusion weight vector to output the feature f 1D And the output feature f 2D Perform dynamic weighting to obtain a weighted fusion feature vector.

[0033] Furthermore, the self-supervised comparative pre-training of the constructed multimodal heterogeneous feature fusion network is performed to obtain the pre-trained multimodal heterogeneous feature fusion network. The specific operations include:

[0034] Design corresponding data enhancement methods for one-dimensional underwater acoustic signal segments and their corresponding two-dimensional Mel-spectrograms;

[0035] For each sample x in the dataset i , using data enhancement method to generate two different enhanced versions to form a positive sample pair (x i ,x j ), and then randomly extract several enhanced samples from other samples to form negative sample pairs, where each sample includes a one-dimensional underwater acoustic signal segment and its corresponding two-dimensional Mel spectrum map;

[0036] Using the multimodal heterogeneous feature fusion network, forward propagation is performed on each enhanced version of the positive and negative sample pairs to obtain the corresponding fused feature vector, which is expressed as:

[0037]

[0038] in, Represents a multimodal heterogeneous feature fusion network;

[0039] A two-layer fully connected network is introduced as the projection head to map the fused feature vector to the contrastive learning space, which is expressed as:

[0040] z i =MLP(fi ),z j =MLP(f j )

[0041] Among them, MLP is a fully connected network;

[0042] The following normalized temperature-scale cross entropy loss function is used to calculate the loss function values ​​of all positive and negative sample pairs, and the average is taken as the overall loss function value. The network parameters of the multimodal heterogeneous feature fusion network are optimized using the Adam optimizer;

[0043]

[0044] Where sim(·) is the cosine similarity, τ is the temperature parameter, N is the number of samples in the batch, and sign is the indicator function;

[0045] Repeat the above steps to obtain the pre-trained multimodal heterogeneous feature fusion network.

[0046] Furthermore, the classification module takes the fusion features output by the multimodal heterogeneous feature fusion network as input and the probability distribution of each category as output, specifically including:

[0047] The fused features output by the multimodal heterogeneous feature fusion network are input into a fully connected layer, followed by a batch normalization layer and a ReLU activation function, and further connected to a Dropout layer. Finally, a fully connected layer is used to map the features to the number of categories, and the Softmax function is used to generate the probability distribution of each category to build a classification module.

[0048] Furthermore, the pre-trained multimodal heterogeneous feature fusion network and classification module are subjected to progressive supervised fine-tuning to obtain a classification neural network model. The specific operations include:

[0049] Construct a labeled dataset and divide it into training set, validation set and test set;

[0050] The cross entropy loss function and Adam optimizer are used to perform progressive supervised fine-tuning on the pre-trained multimodal heterogeneous feature fusion network and classification module on the training set.

[0051] Furthermore, the progressive supervised fine-tuning includes the following specific operations:

[0052] In the initial stage, the one-dimensional feature channel branch module and the two-dimensional feature channel branch module are frozen, and only the feature fusion module and the classification module are optimized and updated;

[0053] When the feature fusion module and classification module are stable, a gradual unfreezing strategy is adopted to gradually unfreeze the frozen parameters in the one-dimensional feature channel branch module and the two-dimensional feature channel branch module according to a fixed round, and the pre-trained multimodal heterogeneous feature fusion network and classification module are jointly fine-tuned to finally obtain a classification neural network model.

[0054] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0055] (1) The method of the present invention solves the problem of limited expressive ability of traditional single-modal features by adaptively fusing time-domain waveform features and frequency-domain spectrum features through a learnable dynamic weight allocation strategy;

[0056] (2) The method of the present invention designs an enhancement method for the time-frequency characteristics of underwater acoustic signals, constructs a cross-modal contrast loss function, and effectively utilizes unlabeled data to improve the generalization ability of the model;

[0057] (3) The method of the present invention adopts a freeze-thaw alternating training method to achieve progressive learning of classification task-specific features while retaining the general feature extraction capability;

[0058] (4) The method of the present invention develops a model compression pipeline for embedded devices, realizing the real-time deployment of high-precision classification models in a low computing power environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 It is a one-dimensional sound time signal diagram collected;

[0061] Figure 3 It is the one-dimensional sound time signal diagram after slicing;

[0062] Figure 4 is the two-dimensional Mel spectrum corresponding to the slice signal;

[0063] Figure 5 Schematic diagram of one-dimensional feature channel branch module;

[0064] Figure 6 Schematic diagram of the two-dimensional feature channel branch module;

[0065] Figure 7 Schematic diagram of the multi-branch attention convolution module;

[0066] Figure 8 Schematic diagram of the feature fusion module. DETAILED DESCRIPTION

[0067] The technical solution of this embodiment will now be further described with reference to the accompanying drawings and embodiments.

[0068] This embodiment proposes an underwater acoustic signal classification method based on contrastive learning and feature fusion, which mainly includes the following steps:

[0069] Step 1: Collect multi-source underwater acoustic signals and perform enhancement preprocessing on the collected multi-source underwater acoustic signals. The specific operations include:

[0070] Step 1-1: Use acoustic sensors and acquisition equipment to collect underwater acoustic signals from different sea areas, different water depths, and different environmental conditions (such as sea conditions, temperature, salinity, etc.). During the data acquisition process, environmental parameters, acquisition time, sensor model, and installation location are recorded. In some embodiments,

[0071] Step 1-2: The collected raw underwater acoustic signals are stored in a unified format, and the sampling rate of each sound file remains consistent. Based on the characteristics of the underwater acoustic signals, some of the collected data are manually labeled, including key information such as sound source category, signal-to-noise ratio, and environmental description.

[0072] Step 1-3: Perform amplitude normalization on the noise-reduced signal to achieve signal standardization and eliminate amplitude fluctuations caused by differences in acquisition equipment or environmental conditions.

[0073] Step 1-4: Determine the appropriate sampling period and serial port overlap ratio based on the signal sampling rate, and use a sliding window to divide the continuous underwater acoustic signal into segments of fixed length, that is, perform segmented processing on the signal.

[0074] Step 1-5: Perform time-frequency conversion on the underwater acoustic signal segments to obtain a two-dimensional Mel-spectrogram, which complements the original one-dimensional signal to form the model input data set.

[0075] The time-frequency conversion in Step 1-5 is now further explained.

[0076] Use window function W(t) and fixed window moving interval T ω The underwater acoustic signal segment x(t) is segmented, and the Fourier transform of each segment is calculated to obtain the amplitude spectrum, which is expressed as:

[0077]

[0078] Where m is the window index and ω is the frequency variable.

[0079] The power spectrum is obtained by squaring the obtained amplitude spectrum, which is expressed as:

[0080] S(m,ω)=|X(m,ω)| 2

[0081] The power spectrum is weighted using a Mel filter bank, and the weighted energy of each Mel filter is calculated:

[0082]

[0083] Among them, H k (ω) represents the response function of the kth Mel filter at frequency ω.

[0084] Taking the logarithm of the weighted energy of each Mel filter, we get a two-dimensional Mel spectrum, which is expressed as:

[0085] LMS(m,k)=log(S m (m,k)+∈)

[0086] Among them, ∈ is a small value to ensure the stability of the logarithmic operation.

[0087] Step 2: Construct a multimodal heterogeneous feature fusion network, which includes: a one-dimensional feature channel branch module, a two-dimensional feature channel branch module and a feature fusion module, which is used to convert the input one-dimensional time segment and two-dimensional spectrum map into fusion features.

[0088] Among them, the one-dimensional feature channel branch module takes the one-dimensional underwater acoustic signal fragment obtained in Step 1-4 as input, first uses multiple cascaded one-dimensional convolution kernels to perform multi-level downsampling to reduce the parameters of the subsequent network, and then constructs a one-dimensional convolution layer as the input layer, which includes a one-dimensional convolution kernel, a batch normalization layer and a ReLU activation function, followed by two consecutive stacked residual connection modules, whose output is subjected to a global average pooling operation to obtain the output feature f 1D .

[0089] Among them, the two-dimensional feature channel branch module takes the two-dimensional Mel spectrum obtained in Step 1-5 as input, constructs a two-dimensional convolution layer as the input layer, including a two-dimensional convolution kernel, a batch normalization layer and a ReLU activation function, and then performs a maximum pooling operation to reduce the size of the input two-dimensional spectrum by half; followed by multiple continuously stacked multi-branch attention convolution modules, each of which is down-sampled using a convolution kernel, and its output is subjected to a global average pooling operation to obtain the output feature f 2D .

[0090] Among them, the feature fusion module first outputs the feature f 1D And the output feature f 2D Splicing is performed to obtain preliminary fusion features, which are input into the fusion weight network for adaptive dynamic fusion to obtain the fusion feature f fusion .

[0091] The residual connection module mentioned in the one-dimensional feature channel branch module is constructed by stacking multiple convolutional layers in succession, where each convolutional layer contains a one-dimensional convolution kernel, a batch normalization layer, and a ReLU activation function. The input features are added to the output features of the residual connection module to form a residual connection. A one-dimensional convolution kernel is added after the continuously stacked residual connection modules, and its parameters are set to double the channels of the input features and halve their size.

[0092] The specific operations of the multi-branch attention convolution module mentioned in the two-dimensional feature channel branch module are:

[0093] Step 2-1: Construct multiple two-dimensional convolutional layers of different sizes and the same channels, including two-dimensional convolution kernels, batch normalization layers, and ReLU activation functions, and synchronously input the input features to multiple two-dimensional convolutional layers to obtain multiple input features;

[0094] Step 2-2: Use the global average pooling operation to extract the channel description vector from the input features obtained in the previous step, and then compress and expand the channel through two 1x1 two-dimensional convolution kernels. Finally, generate a weight vector through the Soigmoid function and multiply it channel by channel with the output features obtained in the previous step to obtain the channel attention weighted feature.

[0095] Step 2-3: Repeat step 2-2 to process the multiple input features obtained in step 2-1 to obtain multiple channel attention weighted features, and then splice these features along the channel direction. Use a two-dimensional convolutional layer to perform channel compression on the spliced ​​features, and then add them to the input features in step 2-1 for residual connection to obtain the output features.

[0096] Among them, the specific operation of the fusion weight network in the feature fusion module to perform adaptive dynamic fusion is:

[0097] Build a fully connected layer, followed by a batch normalization layer and a ReLU activation function to obtain an intermediate representation from the input features;

[0098] Construct a fully connected layer as the output layer, connect the Sigmoid activation function to transform the intermediate representation, and output the fusion weight vector a=[a1,a2], where a1 corresponds to the weight of the one-dimensional feature channel branch and a2 corresponds to the weight of the two-dimensional feature channel branch.

[0099] Use fusion weights to output feature f 1D And the output feature f 2D Perform dynamic weighting and calculate the weighted fusion feature vector, which is expressed as:

[0100] f fusion =a1·f 1D +a2·f 2D

[0101] The weighted fusion feature vector is added to the initial concatenated feature vector for residual connection to further alleviate the information loss and gradient disappearance problems.

[0102] Step 3: Perform self-supervised contrastive pre-training on the multimodal heterogeneous feature fusion network according to the following sub-steps:

[0103] Step 3-1: Design corresponding data enhancement methods for the one-dimensional underwater acoustic signal fragment and the two-dimensional Mel-spectrogram respectively;

[0104] Step 3-2: For each sample x in the dataset i , using data enhancement method to generate two different enhanced versions to form a positive sample pair (x i ,x j ), and then randomly extract several enhanced samples from other samples to form negative sample pairs, where each sample should contain a one-dimensional underwater acoustic signal segment and a corresponding two-dimensional Mel spectrum map.

[0105] Step 3-3: Use the multimodal heterogeneous feature fusion network to perform forward propagation on each enhanced version of the positive and negative sample pairs to obtain the corresponding fused feature vector:

[0106]

[0107] in, Represents a multimodal heterogeneous feature fusion network, and the obtained f i 、f j is the fused feature vector.

[0108] Step 3-4: Introduce a two-layer fully connected network as the projection head to map the fused feature vector to the contrastive learning space:

[0109] z i =MLP(f i ),z j =MLP(f j )

[0110] Among them, MLP is a fully connected network, in which each layer includes a batch normalization layer and a ReLU activation function;

[0111] Step 3-5: Design the normalized temperature scale cross entropy loss function, expressed as:

[0112]

[0113] in,

[0114] Among them, sim(z i,z j ) is the cosine similarity, τ is the temperature parameter, N is the number of samples in the batch, and sign is the indicator function.

[0115] Step 3-6: Use the normalized temperature-scaled cross entropy loss function to calculate the loss function values ​​of all positive and negative sample pairs in the entire batch, take the average as the overall loss function value, and optimize the network parameters using the Adam optimizer.

[0116] Step 3-7: Repeat Step 3-1 to Step 3-6, use Step 1-4 and Step 1-5 to process the unlabeled dataset obtained in Step 1-3, and iteratively train the multimodal heterogeneous feature fusion network on it.

[0117] Step 4: Perform progressive supervised fine-tuning on the trained multimodal heterogeneous feature fusion network. The specific operations are as follows:

[0118] Step 4-1: Assign accurate category labels to the one-dimensional underwater acoustic signal segments and corresponding two-dimensional Mel-spectrograms preprocessed in Steps 1-4 and 1-5 based on the partial annotation results obtained in Step 1-3, and divide the annotated dataset into a training set, a validation set, and a test set.

[0119] Step 4-2: The fusion features output by the multimodal heterogeneous feature fusion network constructed in step 2 are input to a fully connected layer, followed by a batch normalization layer and a ReLU activation function. A Dropout layer is further used to prevent overfitting. Finally, a fully connected layer is used to map the features to the number of categories, and the Softmax function is used to generate the probability distribution of each category to build a classification module.

[0120] Step 4-3: Using the cross entropy loss function and Adam optimizer, fine-tune the multimodal heterogeneous feature fusion network trained in Step 3-7 and the classification module built in Step 4-2 on the training set. During the process, a learning rate decay strategy and early stopping mechanism are used to avoid model overfitting. Specifically:

[0121] In the early stage of supervised fine-tuning, the one-dimensional feature channel branch and the two-dimensional feature channel branch modules are frozen, and only the feature fusion module and the classification module are optimized and updated;

[0122] When the training process stabilizes, a gradual unfreezing strategy is adopted to gradually unfreeze the frozen parameters in the one-dimensional feature channel branch and the two-dimensional feature channel branch modules according to a fixed round, and a lower learning rate is used to jointly fine-tune the entire network to finally obtain a classification neural network model.

[0123] Step 5: Integrate the classification neural network model with the system to complete lightweight deployment. Specific operations include:

[0124] Step 5-1: Model compression and acceleration: Calculate the L1 norm of the convolutional layer channels and prune redundant channels with scores below the set threshold; remove neuron connections in the feature fusion module whose absolute weights are less than the preset value; use the original model as the teacher network and compensate for accuracy through KL divergence loss; implement 8-bit fixed-point quantization on the feature extraction network, and retain 16-bit floating-point precision on the classification head; use the TensorRT engine for layer fusion, combining the convolution kernel, batch normalization layer, and ReLU activation function into a single computing unit;

[0125] Step 5-2: Integrate the edge computing system: Deploy a high-performance embedded GPU module as the main processor of the central server, equipped with a parallel computing core; assign signal preprocessing tasks to a dedicated DSP coprocessor; establish a high-speed data channel between processors to achieve task-level parallel processing; monitor device parameters and dynamically adjust the model inference mode based on the current device remaining power and CPU load; calculate the signal feature change rate and reuse the historical classification results when it is below the threshold;

[0126] Step 5-3: Establish a continuous learning and remote update mechanism: deploy a micro-training engine at the edge, adopt an elastic weight solidification algorithm, retain the weights of important parameters, and fine-tune the parameters of the last layer of classifiers online; the edge device uses residual merging to update the local model parameters; calculate the prediction confidence, and automatically trigger a manual review request when the prediction confidence is lower than the threshold; package the low-confidence samples and expert annotation results and send them back for iterative model training on the central server.

[0127] Example 2:

[0128] Based on Example 1, this example proposes specific cases for applying this method, including:

[0129] Step 1: Using the icListen intelligent hydrophone, underwater acoustic signals were collected in the Atlantic Ocean at a depth of 200-2000m over a two-year period. Multiple sets of one-dimensional single-channel sound time signals were obtained, and information such as environmental parameters, acquisition time, sensor model, and installation location were recorded. The collected original underwater acoustic signals were uniformly stored in .wav format, and the sampling rate of each sound file was maintained at 51200Hz. Based on the characteristics of the underwater acoustic signals and the types of ships passing at the time, the collected data were partially manually labeled and classified into four types of ships. The stored underwater acoustic signals were amplitude normalized to eliminate amplitude fluctuations caused by differences in acquisition equipment or environmental conditions. Figure 2Given an 8 minute 28 second recorded sound signal, the horizontal axis is time and the vertical axis is amplitude. Based on the sampling rate of the underwater sound signal, the window length is determined to be 0.5s, the window overlap length is 0.1s, and the continuous underwater sound signal is divided into segments of fixed length by sliding the window, as shown in the figure. Figure 3 The one-dimensional underwater acoustic signal fragment is shown in the figure, where the horizontal axis is time and the vertical axis is amplitude. The one-dimensional underwater acoustic signal fragment is converted into a two-dimensional Mel spectrum, which complements the original one-dimensional underwater acoustic signal fragment and together constitutes the data set for model input. The window function W(t) is a Gaussian window function with a fixed window movement interval T. ω = 0.001s, and the Fourier transform of each segment is calculated to obtain the amplitude spectrum. The amplitude spectrum is squared to obtain the power spectrum. The power spectrum is weighted using a Mel filter bank, and the weighted energy of each Mel filter is calculated. In this embodiment, the Mel filter bank has a total of 128 Mel filters. The logarithm of the weighted energy of each Mel filter is taken to obtain a two-dimensional Mel spectrum diagram, as shown in Figure 4 As shown, the horizontal axis is the window index m, and the vertical axis is the corresponding Mel filter identifier k. In this embodiment, ∈=10 -6 A smaller value is used to ensure the stability of the logarithmic operation.

[0130] Step 2: Construct a multimodal heterogeneous feature fusion network.

[0131] Among them, the one-dimensional feature channel branch module, such as Figure 5 As shown in the figure, a one-dimensional underwater acoustic signal fragment is used as input, and four cascaded small-scale convolutions are used for multi-level downsampling. The size of each convolution kernel is 3X1, the step size is 2, and the channel is 16. Then a one-dimensional convolution layer is constructed, which includes a one-dimensional convolution kernel, a batch normalization layer and a ReLU activation function. The convolution kernel size is 1X1, the channel is 64, and the step size is 1. It is followed by two consecutive stacked residual connection module stages, and its output is subjected to a global average pooling operation to obtain the output feature.

[0132] Among them, the residual connection module is composed of two consecutively stacked convolutional layers, where each convolutional layer contains a one-dimensional convolution kernel, a batch normalization layer and a ReLU activation function. The input features are added to the output features of the residual connection module to form a residual connection.

[0133] Specifically, in the first residual connection module stage, three residual connection modules are stacked continuously, where the convolution kernel size in each convolution layer is 3X1, the channel is 64, and the stride is 1. A one-dimensional convolution kernel is added after it, with a convolution kernel size of 1X1, the channel is 128, and the stride is 2. The input feature channel is adjusted to 128, and the size is halved.

[0134] Specifically, in the second residual connection module stage, three residual connection modules are stacked continuously, where the convolution kernel size in each convolution layer is 3X1, the channel is 128, and the stride is 1. A one-dimensional convolution kernel is added after it, with a convolution kernel size of 1X1, the channel is 256, and the stride is 2. The input feature channel is adjusted to 256, and the size is halved.

[0135] Among them, the two-dimensional feature channel branch module: Figure 6 As shown in , a two-dimensional spectrogram is used as input and a two-dimensional convolution layer is constructed as the input layer, including a two-dimensional convolution kernel, a batch normalization layer and a ReLU activation function. The convolution kernel size is 3X3, the channel is 32, and the stride is 1. The maximum pooling operation is then performed to halve the size of the input two-dimensional spectrogram; followed by three consecutively stacked multi-branch attention convolution modules, as shown in Figure 7 As shown in the figure, each module is downsampled by a convolution kernel with a step size of 2, and the channels are all 64. Then, a convolution kernel with a size of 1X1, a channel of 256, and a step size of 1 is used to adjust the feature channel to 256. Finally, its output is subjected to a global average pooling operation to obtain the output feature

[0136] Among them, the multi-branch attention convolution module includes:

[0137] Construct three 2D convolutional layers, including 2D convolution kernels, batch normalization layers, and ReLU activation functions. The convolution kernel sizes are 3x3, 5x5, and 7x7, respectively. The channels are all 64, and the stride is 1. Select an appropriate padding coefficient to ensure that the input and output feature sizes are consistent. The input features are synchronously input to the three convolutional layers to obtain three output features.

[0138] The global average pooling operation is used to extract the channel description vector from the output features. The channels are then compressed and expanded using two 1x1 two-dimensional convolution kernels with 16 and 64 channels respectively. Finally, a weight vector is generated using the Soigmoid function and multiplied channel by channel with the output features obtained in the previous step to obtain the channel attention weighted features.

[0139] The three output features obtained are processed to obtain multiple channel attention weighted features, which are then spliced ​​along the channel direction. The spliced ​​features are channel compressed using a two-dimensional convolutional layer with a convolution kernel size of 1X1 and 64 channels. They are then added to the input features of the three two-dimensional convolutional layers for residual connection to obtain the output features.

[0140] Among them, the feature fusion module, such as Figure 8 As shown, the output features are and output features Splicing to obtain preliminary fusion features Input it into the fusion weight network for adaptive dynamic fusion to obtain the fusion feature

[0141] The feature fusion module mentioned in this embodiment is constructed according to the following steps:

[0142] Construct a fully connected layer with 512 neurons, followed by a batch normalization layer and a ReLU activation function to obtain an intermediate representation from the initial fused features;

[0143] A fully connected layer is constructed as the output layer, with 512 neurons. The Sigmoid activation function is used to transform the intermediate representation and output the fusion weight vector a = [a1, a2], where a1 corresponds to the weight of the one-dimensional feature channel branch and a2 corresponds to the weight of the two-dimensional feature channel branch.

[0144] Use fusion weights to output features and output features Perform dynamic weighting and calculate the weighted fusion feature vector.

[0145] Using a fully connected network with 256 neurons, the initial fusion features Adjust to Then fused with the weighted feature vector f′ weight Add the residual connection to get the fusion feature f′ fusion , further alleviating the information loss and gradient vanishing problems.

[0146] Step 3: Self-supervised contrastive pre-training; including:

[0147] For one-dimensional underwater acoustic signal fragments, time shift, amplitude scaling, and background noise addition are used for data enhancement; for two-dimensional Mel-spectrograms, random cropping, rotation, and color jittering are used for data enhancement.

[0148] For each sample x in the dataset i , using data enhancement method to generate two different enhanced versions to form a positive sample pair (x i ,x j ), and then randomly extract several enhanced samples from other samples to form negative sample pairs, where each sample should contain a one-dimensional underwater acoustic signal segment and a corresponding two-dimensional Mel spectrum map.

[0149] Using the constructed multimodal heterogeneous feature fusion network, forward propagation is performed on each enhanced version of the positive and negative sample pairs to obtain the corresponding fused feature vector;

[0150] A two-layer fully connected network is introduced as the projection head, with 256 and 512 neurons respectively, to map the fusion features into the contrastive learning space.

[0151] Design a normalized temperature-scale cross entropy loss function, where τ = 0.1 and N = 64.

[0152] The normalized temperature-scaled cross entropy loss function is used to calculate the loss function values ​​of all positive and negative sample pairs in the entire batch, and the average is taken as the overall loss function value. The network parameters are optimized using the Adam optimizer.

[0153] Iteratively train multimodal heterogeneous feature fusion network.

[0154] Based on the partial annotation results obtained in Step 1-3, accurate category labels are assigned to the one-dimensional underwater acoustic signal segments and the corresponding two-dimensional Mel-spectrograms preprocessed in Step 1-4 and Step 1-5, and the annotated dataset is divided into a training set, a validation set, and a test set, with the proportions of 68%, 12%, and 20%, respectively.

[0155] The fusion features output by the constructed multimodal heterogeneous feature fusion network are input to a fully connected layer with 256 neurons, followed by a batch normalization layer and a ReLU activation function. A Dropout layer is further used to prevent overfitting. Finally, the features are mapped to the number of categories through a fully connected layer with 4 neurons. The Softmax function is used to generate the probability distribution of each category to build a classification module.

[0156] The cross-entropy loss function and Adam optimizer are used to fine-tune the trained multimodal heterogeneous feature fusion network and the constructed classification module on the training set. A three-stage learning rate decay strategy is adopted in the process. Linear warm-up is implemented in the first five rounds, and the learning rate is gradually increased from 10% of the initial value to the full amount of 0.001. The cosine annealing algorithm is used in the main training stage, and the learning rate is smoothly decayed from 0.001 to 0.000001 within 50 rounds. When the accuracy of the validation set does not improve for three consecutive epochs, the dynamic decay mechanism is triggered, and the learning rate is decreased by 50%. The early stopping mechanism is adopted. When the improvement of the validation loss value relative to the historical optimal value is less than 0.001, the model training is stopped to avoid model overfitting.

[0157] In the early stage of supervised fine-tuning, the one-dimensional feature channel branch and the two-dimensional feature channel branch modules are frozen, and only the feature fusion module and the classification module are optimized and updated;

[0158] When the training process stabilizes, a gradual unfreezing strategy is adopted to gradually unfreeze 10% of the frozen parameters in the one-dimensional feature channel branch and the two-dimensional feature channel branch modules every 5 rounds, and the entire network is jointly fine-tuned with a learning rate of 0.00001 to finally obtain a classification neural network model.

[0159] Step 5: Integrate the classification neural network model with the system to complete lightweight deployment. Specific operations include:

[0160] Model compression and acceleration: Calculate the L1 norm of the convolutional layer channels and prune redundant channels with a score below 0.2; remove neuron connections with absolute weights less than 0.001 in the feature fusion module; use the original model as the teacher network and compensate for accuracy through KL divergence loss; implement 8-bit fixed-point quantization on the feature extraction network, while retaining 16-bit floating-point precision on the classification head; use the TensorRT engine for layer fusion, combining the convolution kernel, batch normalization layer, and ReLU activation function into a single computational unit;

[0161] Integrated edge computing system: A high-performance embedded GPU module, equipped with a parallel computing core, is deployed as the main processor of the central server. Signal preprocessing tasks are assigned to a dedicated DSP coprocessor. A high-speed data channel is established between processors to achieve task-level parallel processing. Device parameters are monitored, and the model inference mode is dynamically adjusted based on the current device remaining power and CPU load. The signal feature change rate is calculated, and when it is less than 0.001, the historical classification results are reused.

[0162] Establish a continuous learning and remote update mechanism: deploy a micro-training engine at the edge, adopt an elastic weight solidification algorithm to retain the weights of important parameters, and fine-tune the parameters of the last layer of classifiers online; edge devices use residual merging to update local model parameters; calculate the prediction confidence, and automatically trigger a manual review request when the prediction confidence is lower than 0.1; package low-confidence samples and expert annotation results and send them back for iterative model training on the central server.

Claims

1. A method for underwater acoustic signal classification based on contrastive learning and feature fusion, characterized by: The following steps are involved: Step 1: Collect the original underwater acoustic signal and perform enhancement preprocessing on the original underwater acoustic signal to obtain the preprocessed underwater acoustic signal; Step 2: Output the preprocessed underwater acoustic signal to the classification neural network model to obtain the underwater acoustic signal classification result; The classification neural network model is obtained according to the following steps: Construct a multimodal heterogeneous feature fusion network; Perform self-supervised comparative pre-training on the constructed multimodal heterogeneous feature fusion network to obtain a pre-trained multimodal heterogeneous feature fusion network; Building a classification module that takes the fused features output by the multimodal heterogeneous feature fusion network as input and outputs the probability distribution of each category; The pre-trained multimodal heterogeneous feature fusion network and classification module are progressively fine-tuned to obtain a classification neural network model.

2. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 1, characterized in that: The aforementioned enhancement preprocessing of the original underwater acoustic signal to obtain the preprocessed underwater acoustic signal specifically includes the following operations: The original underwater acoustic signal is subjected to noise reduction and amplitude normalization processing in sequence to obtain a standardized underwater acoustic signal; The standardized underwater acoustic signal is divided into a plurality of one-dimensional underwater acoustic signal segments of fixed length; Perform time-frequency conversion on each one-dimensional underwater acoustic signal segment to obtain a two-dimensional Mel-spectrogram; The one-dimensional underwater acoustic signal segment and the corresponding two-dimensional Mel spectrum graph together constitute the preprocessed underwater acoustic signal.

3. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 2, characterized in that: The specific operations of constructing a multimodal heterogeneous feature fusion network include: A one-dimensional feature channel branch module is constructed; the one-dimensional feature channel branch module takes a one-dimensional underwater acoustic signal segment as input, first uses multiple cascaded one-dimensional convolution kernels to perform multi-level downsampling on the input, then uses a one-dimensional convolution layer as the input layer, followed by two consecutive stacked residual connection module stages, and finally obtains the output feature f through a global average pooling operation. 1D ; Construct a two-dimensional feature channel branch module; the two-dimensional feature channel branch module takes the two-dimensional Mel spectrum map as input, adopts a two-dimensional convolution layer as the input layer, and then performs a maximum pooling operation to halve the size of the input two-dimensional Mel spectrum map, and then connects multiple continuously stacked multi-branch attention convolution modules, wherein a convolution kernel is used for downsampling after each multi-branch attention convolution module, and finally the output feature f is obtained through a global average pooling operation. 2D ; Construct a feature fusion module; the feature fusion module outputs feature f 1D And the output feature f 2D As input, the output feature f 1D And the output feature f 2D Splicing is performed to obtain the initial splicing feature vector, and the output feature f is converted into 1D And the output feature f 2D Perform adaptive dynamic fusion to obtain the weighted fusion feature vector; add the initial splicing feature vector and the weighted fusion feature vector to perform residual connection to obtain the fusion feature f fusion .

4. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 3 is characterized by: Each of the residual connection modules is composed of a plurality of convolutional layers stacked in succession, each convolutional layer includes a one-dimensional convolution kernel, a batch normalization layer and a ReLU activation function, a one-dimensional convolution kernel is added after each convolutional layer, and the input features of the residual connection module are added to the output features of the residual connection module to form a residual connection; The two continuously stacked residual connection module stages include a first residual connection module stage and a second residual connection module stage; wherein the first residual connection module stage is composed of three continuously stacked residual connection modules; and the second residual connection module stage is composed of three continuously stacked residual connection modules.

5. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 4 is characterized in that: The multi-branch attention convolution module is constructed according to the following steps: Construct multiple two-dimensional convolutional layers of different sizes and the same channels, and synchronously input the input features of the multi-branch attention convolution module to multiple two-dimensional convolutional layers to obtain multiple output features; The following operations are performed on each output feature: a channel description vector is extracted from the output features of the two-dimensional convolution layer using a global average pooling operation, the channel is compressed and expanded using two two-dimensional convolution kernels of size 1x1, and finally a weight vector is generated using the Soigmoid function. This weight vector is multiplied channel by channel with the output features of the two-dimensional convolution layer to obtain the channel attention weighted feature. The obtained multiple channel attention weighted features are spliced ​​along the channel direction to obtain spliced ​​features; A two-dimensional convolution layer is used to compress the splicing features, which are then added to the input features of the multi-branch attention convolution module for residual connection to obtain the output features of the multi-branch attention convolution module.

6. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 5, characterized in that: The fusion weight network described above will output the feature f 1D And the output feature f 2D Perform adaptive dynamic fusion to obtain the fusion feature f fusion Specific operations include: Construct a fully connected layer, followed by a batch normalization layer and a ReLU activation function, and output the feature f 1D And the output feature f 2D Get intermediate representation; Construct a fully connected layer as the output layer, connect the Sigmoid activation function to transform the intermediate representation, and output the fusion weight vector a = [a1, a2], where a1 corresponds to the weight of the one-dimensional feature channel branch module, and a2 corresponds to the weight of the two-dimensional feature channel branch module; Use the fusion weight vector to output the feature f 1D And the output feature f 2D Perform dynamic weighting to obtain a weighted fusion feature vector.

7. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 6, characterized in that: The self-supervised comparative pre-training of the constructed multimodal heterogeneous feature fusion network is performed to obtain the pre-trained multimodal heterogeneous feature fusion network. The specific operations include: Design corresponding data enhancement methods for one-dimensional underwater acoustic signal segments and their corresponding two-dimensional Mel-spectrograms; For each sample x in the dataset i , using data enhancement method to generate two different enhanced versions to form a positive sample pair (x i ,x j ), and then randomly extract several enhanced samples from other samples to form negative sample pairs, where each sample includes a one-dimensional underwater acoustic signal segment and its corresponding two-dimensional Mel spectrum map; Using the multimodal heterogeneous feature fusion network, forward propagation is performed on each enhanced version of the positive and negative sample pairs to obtain the corresponding fused feature vector, which is expressed as: in, Represents a multimodal heterogeneous feature fusion network; A two-layer fully connected network is introduced as the projection head to map the fused feature vector to the contrastive learning space, which is expressed as: With i =MLP(f i ),With j =MLP(f j ) Among them, MLP is a fully connected network; The following normalized temperature-scale cross entropy loss function is used to calculate the loss function values ​​of all positive and negative sample pairs, and the average is taken as the overall loss function value. The network parameters of the multimodal heterogeneous feature fusion network are optimized using the Adam optimizer; Where sim(·) is the cosine similarity, τ is the temperature parameter, N is the number of samples in the batch, and sign is the indicator function; Repeat the above steps to obtain the pre-trained multimodal heterogeneous feature fusion network.

8. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 7, characterized in that: The classification module takes the fusion features output by the multimodal heterogeneous feature fusion network as input and the probability distribution of each category as output, specifically including: The fused features output by the multimodal heterogeneous feature fusion network are input into a fully connected layer, followed by a batch normalization layer and a ReLU activation function, and further connected to a Dropout layer. Finally, a fully connected layer is used to map the features to the number of categories, and the Softmax function is used to generate the probability distribution of each category to build a classification module.

9. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 8, characterized in that: The aforementioned progressive supervised fine-tuning of the pre-trained multimodal heterogeneous feature fusion network and classification module to obtain a classification neural network model includes the following specific operations: Construct a labeled dataset and divide it into training set, validation set and test set; The cross entropy loss function and Adam optimizer are used to perform progressive supervised fine-tuning on the pre-trained multimodal heterogeneous feature fusion network and classification module on the training set.

10. The underwater acoustic signal classification method based on contrastive learning and feature fusion according to claim 9, characterized in that: The progressive supervised fine-tuning described above includes the following specific operations: In the initial stage, the one-dimensional feature channel branch module and the two-dimensional feature channel branch module are frozen, and only the feature fusion module and the classification module are optimized and updated; When the feature fusion module and classification module are stable, a gradual unfreezing strategy is adopted to gradually unfreeze the frozen parameters in the one-dimensional feature channel branch module and the two-dimensional feature channel branch module according to a fixed round, and the pre-trained multimodal heterogeneous feature fusion network and classification module are jointly fine-tuned to finally obtain a classification neural network model.

Images