A semi-supervised underwater acoustic event detection method based on joint perturbation consistency constraint

Abstract

The present application relates to a kind of semi-supervised underwater acoustic event detection method based on joint disturbance consistency constraint, comprising: obtaining the logarithmic mel spectrum of strong label audio data, weak label audio data and unlabeled audio data;Weak enhancement and strong enhancement processing are carried out on the logarithmic mel spectrum of unlabeled audio data, to obtain the corresponding enhanced spectrum;Deep learning network model is constructed, and the spectrum of strong label audio data and weak label audio data is used for supervised training;Unlabeled audio data is used in the semi-supervised model of multiple branch feature disturbance and strong disturbance joint;Finally, the training process of deep learning network model is optimized by calculating joint loss function;The underwater acoustic event is detected using the trained deep learning network model, and the event category of audio data is output.The present application is used to realize the effective detection of sound event using a small amount of strong note data, and improve model performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater acoustic monitoring, and in particular relates to a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints. Background Art

[0002] Due to their minimal loss and maximum propagation distance in underwater environments, underwater acoustic signals have become a key means of acquiring marine information. Underwater acoustic event detection (SED) technology, which analyzes the characteristics of underwater acoustic signals to identify the type of acoustic event, has important applications in multiple fields. This technology can accurately, promptly, and covertly identify underwater targets. In non-military applications, it is not only used to monitor and manage maritime traffic and ensure shipping safety, but also helps collect data on the species and habits of marine life, which is of immeasurable value to marine ecological protection.

[0003] Early sound event detection employed methods such as hidden Markov models, random forest regression, and support vector machines. With the continuous development of deep learning, an increasing number of deep learning-based methods are outperforming traditional methods. In the task of sound event detection, data types are categorized into three types: strongly labeled data containing both the event category and the start and end times of the event; weakly labeled data containing only the event category but not the start and end times; and unlabeled data lacking any category labels or timestamp information. Traditional deep learning methods generally require large amounts of labeled data to train the model through supervised learning to achieve high performance. There are numerous types of underwater sound events, including various types of ship radiation noise, marine life noise, and sounding signal noise. For underwater sound event detection, this requires a large amount of strongly labeled training data, which increases the cost of data annotation.

[0004] In recent years, modeling methods based on semi-supervised learning (SSL) have achieved good performance by combining limited labeled data with large amounts of unlabeled data to model SED systems. This provides a sound theoretical and practical foundation for addressing the high cost of underwater sound event annotation and the scarcity of strongly labeled data. In semi-supervised learning, FixMatch is a very effective framework for utilizing unlabeled data. Its core concept is to constrain the "strong and weak consistency" of unlabeled data. Perturbations are particularly important in the FixMatch semi-supervised method. Existing methods introduce perturbations through simple data augmentation. While these methods can bring improvements, the effects are limited. Summary of the Invention

[0005] The technical problems to be solved by the present invention are:

[0006] To address the problems of insufficient strongly labeled audio data and insufficient utilization of unlabeled audio data in underwater sound event detection, the present invention provides a semi-supervised underwater sound event detection method based on joint perturbation consistency constraints, which is used to effectively detect sound events using a small amount of strongly injected data and improve model performance.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A semi-supervised underwater sound event detection method based on joint disturbance consistency constraint is characterized by comprising:

[0009] S1. Obtain the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data, and unlabeled audio data;

[0010] S2. Perform weak enhancement and strong enhancement on the logarithmic Mel spectrum of the unlabeled audio data to obtain the corresponding enhanced spectrum;

[0011] S3. Build a deep learning network model and perform supervised training using the spectra of strongly labeled audio data and weakly labeled audio data. For unlabeled audio data, adopt a semi-supervised model that combines multiple feature perturbations with strong perturbations. Finally, optimize the training process of the deep learning network model by calculating a joint loss function.

[0012] S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

[0013] A further technical solution of the present invention: Step S1 is specifically as follows:

[0014] Fast Fourier transform is performed on the strongly labeled audio data, weakly labeled audio data, and unsigned audio data to convert them into their respective original audio data. The energy spectrum of each is obtained based on the original audio data. 64 Mel filters are used to extract the Mel spectrum features of each energy spectrum. The logarithm of each Mel spectrum feature is taken to obtain the logarithmic Mel spectrum.

[0015] A further technical solution of the present invention: the weak enhancement and strong enhancement processing in step S2 both include time domain masking and frequency domain masking;

[0016] The time domain masking is performed on the continuous time channels within the range of [t0, t0+t) along the time domain axis, where 0 < t ≤ T, 0 ≤ t0 ≤ τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range;

[0017] The frequency domain masking is as follows: continuous frequency channels within the range of [f0, f0+f) along the frequency domain axis are masked, where 0<f≤F, 0≤f0≤vf, v is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range.

[0018] A further technical solution of the present invention: The deep learning network model described in step S3 is a depth-separable residual convolutional network structure AR-CNN based on asymmetric convolution, including a multi-scale convolution module, an asymmetric depth-separable convolution module, a residual module, a global average pooling layer and an output layer; the workflow of the deep learning network model is as follows:

[0019] Input data: Logarithmic Mel-spectrogram as input, containing rich frequency and time information;

[0020] Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features at different scales;

[0021] Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module, performing vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters.

[0022] Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems;

[0023] Feature splicing: Features from different channels are spliced ​​together to enhance the network’s expressive power;

[0024] Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value;

[0025] Classification prediction: The final features are passed through the output layer to reduce the dimension to the number of categories and complete the final prediction of the classification task;

[0026] Output result: Output the classification result, that is, the predicted category label.

[0027] A further technical solution of the present invention: the supervised training process includes:

[0028] Get the labeled audio data from the dataset and convert it into a log-mel spectrogram; get the corresponding labels, which represent the actual categories of the audio data;

[0029] Weakly enhance the input log-mel spectrogram to generate enhanced input data to increase the robustness and generalization ability of the model;

[0030] Feed the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward and backpropagation.

[0031] The input data is passed through the deep learning network model to calculate the predicted category probability; the output of this step is the model's prediction result for each input sample; the cross entropy loss function is used to calculate the loss between the model's predicted value and the actual label;

[0032] Backpropagate the calculated loss to update the model parameters; calculate the gradient of the loss with respect to the model parameters through backpropagation; use the Adam optimizer to update the model parameters according to the calculated gradient to minimize the loss function;

[0033] Repeat the above steps and continuously optimize the model through multiple iterations; in each iteration, train with different batches to improve training efficiency and model generalization ability.

[0034] A further technical solution of the present invention is as follows: the semi-supervised model using a combination of multiple feature perturbations and strong perturbations for unlabeled audio data includes:

[0035] For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the probability vector p of the predicted class distribution w If the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation;

[0036] On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 ; At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 supervision signal.

[0037] A computer system, characterized in that it includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0038] A computer-readable storage medium is characterized by storing computer-executable instructions, which are used to implement the above method when executed.

[0039] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0040] The beneficial effects of the present invention are:

[0041] The semi-supervised underwater sound event detection method based on joint disturbance consistency constraints provided by the present invention has the following advantages over the existing technology:

[0042] 1. Semi-supervised learning is performed using a small amount of labeled data and a large amount of unlabeled data. This method significantly reduces the dependence on a large amount of labeled data and solves the challenge of scarce ocean acoustic data. This is of great significance for improving the feasibility and practicality of underwater sound event detection.

[0043] 2. The present invention adopts a joint perturbation consistency constraint strategy, which combines multiple feature perturbations with strong perturbations, effectively expanding the perturbation space of the model and improving the adaptability and robustness of the model in unknown or rare events.

[0044] 3. The asymmetric residual convolutional neural network proposed in the invention, by combining asymmetric convolution and depthwise separable convolution technology, not only improves the processing speed and accuracy, but also realizes the lightweight network architecture, making the system more suitable for deployment on underwater equipment with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0046] Figure 1 This is a diagram illustrating the structure of a semi-supervised model for detecting sound events in an embodiment of the present invention.

[0047] Figure 2 Schematic diagram of the logarithmic Mel spectrum of ship radiation noise according to the present invention.

[0048] Figure 3 This is a detailed network model structure diagram of the present invention.

[0049] Figure 4 This is a structural diagram of the asymmetric depth-separable convolution module and residual module of the present invention.

[0050] Figure 5 Schematic diagram of the strong and weak spectrum enhancement of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0052] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] The present invention proposes a method for underwater sound event detection based on joint perturbation consistency constraints, which combines multiple strong perturbations and feature perturbations in a consistency constraint framework to effectively expand the perturbation space. This design not only improves the robustness of the model in dealing with diverse input changes, but also optimizes the utilization efficiency of unlabeled data. Specifically, by adding perturbation channels, the model's adaptability to different perturbations is improved, and the feature perturbation further broadens the model's perturbation space and enhances the model's ability to handle unknown changes. In addition, the present invention uses an asymmetric residual convolutional network to accurately identify underwater sound events, thereby improving the model's detection accuracy and efficiency in complex underwater environments. These technical improvements not only ensure efficient model training under conditions of limited labeled data, but also enable the model to adapt to a wider range of data variations, thereby improving the overall performance and reliability of underwater sound event detection.

[0054] like Figure 1 As shown, the following steps are included:

[0055] S1. Obtain the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data, and unlabeled audio data;

[0056] Specifically, strongly labeled data refers to audio data with complete annotated information, typically including the event category and event start and end times. The event category refers to the type of sound event occurring in the audio (e.g., different types of underwater sound sources, such as ships and marine life). The event start and end times refer to the specific time period during which the event occurred, typically including the event's start and end times. This provides a more precise label.

[0057] Specifically, weakly labeled data only provides the event category but does not mark the specific time period. Therefore, weakly labeled data cannot accurately indicate the start and end time of the event, but can only indicate which category the sound event in the audio belongs to (for example, a certain type of ship noise or the sound of marine life).

[0058] Specifically, unlabeled audio data: Unlabeled data lacks any annotated information, including event categories or event times. This type of data is raw, unlabeled data and is typically used in the unsupervised training process of semi-supervised learning, where the model is trained by inferring or generating pseudo labels.

[0059] S2. Perform weak enhancement and strong enhancement on the logarithmic Mel spectrum of the unlabeled audio data to obtain the corresponding enhanced spectrum;

[0060] S3. Build a deep learning network model and perform supervised training using the spectra of strongly labeled audio data and weakly labeled audio data. For unlabeled audio data, adopt a semi-supervised model that combines multiple feature perturbations with strong perturbations. Finally, optimize the training process of the deep learning network model by calculating a joint loss function.

[0061] S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

[0062] Furthermore, step S1 is specifically as follows: obtaining the logarithmic Mel spectrum includes applying fast Fourier transform to the original audio data, calculating the energy spectrum, using 64 Mel filters to extract Mel spectrum features, and taking the logarithm to obtain the logarithmic Mel spectrum.

[0063] Strongly labeled data: After extracting the log-Mel spectrum, it is aligned with the event category and time label to form training samples. Each sample includes features (log-Mel spectrum) and precise labels (category and time period).

[0064] Weakly labeled data: Extracts the log-Mel spectrum, which is aligned with the class label but does not provide temporal information. Each sample includes features and class labels.

[0065] Unlabeled data: Only the log-Mel spectrum is extracted without any label alignment. This is used for pseudo-label generation or feature perturbation training in semi-supervised learning.

[0066] Furthermore, the weak-strong enhancement strategy in step S2 is time-frequency dimension masking. The time domain dimension masking frequency dimension is: masking the continuous time channels in the range of [t0, t0+t) along the time domain axis, where 0<t≤T, 0≤t0≤τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range value. The frequency domain dimension masking frequency is: masking the continuous frequency channels in the range of [f0, f0+f) along the frequency domain axis, where 0<f≤F, 0≤f0≤vf, v is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range value.

[0067] Temporal masking:

[0068] Weak enhancement: The duration of the temporal masking is T = 20, meaning each masking period is relatively short. By masking short time periods, we can perform local perturbations without disrupting the overall framework of the event, allowing the model to learn how to identify important features within these incomplete time periods.

[0069] Strong augmentation: The duration of temporal masking is set to T = 30, meaning each masking period is longer. This means that strong augmentation further blurs the temporal information of the audio, resulting in less accurate input data for the model during training. This operation simulates more interference, enabling the model to identify underwater sound events despite inaccurate input data.

[0070] Frequency masking:

[0071] Weak enhancement: The frequency masking range is F = 10, which means the masked frequency bandwidth is narrow. By perturbing the frequency within a small range, the model can adapt to more subtle frequency changes. Especially when labels are incomplete, a certain amount of frequency information can be retained during training, thus preventing the complete loss of useful spectral features.

[0072] Strong Enhancement: The frequency masking range is F = 20, indicating a wider frequency bandwidth. This results in a greater loss of frequency information, resulting in less accurate spectra processed by the model. This is done to simulate strong interference, allowing the model to train on less ambiguous spectral information, thereby better coping with noise and other uncertainties in complex underwater environments.

[0073] Weak augmentation aims to increase local perturbations while maintaining the data's relative accuracy, allowing the model to be trained in the presence of noise or uncertain labels and still identify underwater sound events. Strong augmentation intentionally makes the data more imprecise through greater perturbations (longer duration and wider frequency masking). This allows the model to be trained on incomplete or distorted data, improving its robustness to noise and imperfect data in real applications.

[0074] Furthermore, the deep learning network model is a depthwise separable residual convolutional network (AR-CNN) based on asymmetric convolution, designed to efficiently extract and classify features from underwater acoustic data. The core components of the network include a multi-scale convolution module, an asymmetric depthwise separable convolution module, and a residual module. The following is a detailed description of each module:

[0075] Multi-scale convolution module: It uses convolution operations of three different scales (5, 9, and 17) to extract features of different granularities from the log-Mel spectrum. Each convolution kernel extracts features independently, enhancing the network's sensitivity to different features.

[0076] Asymmetric Depthwise Separable Convolution Module: Vertical convolution kernels move along the time axis, processing features within the same time frame and extracting frequency distribution features that are translation-invariant. Horizontal convolution kernels process features within consecutive time frames in the frequency domain, extracting distinct temporal features. Point-by-point convolution operations are performed on each channel to achieve parameter sharing, increasing model depth while reducing parameter count and making the model lightweight.

[0077] Residual module: Utilize the gradient signal in the residual module to pass the output of one layer as input to another layer of the architecture through a “skip connection”, reducing the possibility of model overfitting and gradient explosion.

[0078] Global average pooling layer: compresses the features of each channel at all spatial positions into a single feature value, reducing the number of parameters that the model needs to learn.

[0079] Output layer: An output layer is used to reduce the dimension of the features to the number of categories to complete the final prediction of the classification task.

[0080] Workflow Description:

[0081] ① Input data: Logarithmic Mel spectrum is used as input, which contains rich frequency and time information.

[0082] ② Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features of different scales.

[0083] ③ Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module, which performs vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters.

[0084] ④ Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems.

[0085] ⑤ Feature splicing: Features from different channels are spliced ​​to enhance the network’s expressive power.

[0086] ⑥ Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value.

[0087] ⑦Classification prediction: The final features are reduced to the number of categories through the output layer, and the final prediction of the classification task is completed.

[0088] ⑧Output results: Output the classification results, that is, the predicted category labels.

[0089] Furthermore, the loss function calculation includes the following sub-steps:

[0090] S3.1. For labeled data, supervised training is performed, and the cross entropy loss function for the weakly expanded labeled examples is: Where, let χ={(x b ,p b ):b∈(1,...,B)} is a labeled sample, where x b is the training sample, p b is the label, U={u b :b∈(1,...,μ B )} is unlabeled data, p m (y|x) is the predicted category distribution generated by the model for the input x. The loss function H(p,q) is the cross entropy loss function, and strong and weak augmentation are represented by A(·) and α(·), respectively.

[0091] The above supervised training process is as follows:

[0092] ① First, obtain the labeled audio data from the dataset and convert it into a logarithmic Mel-spectrogram. Obtain the corresponding labels, which represent the actual category of the audio data.

[0093] ② Perform weak enhancement on the input log-Mel spectrum graph to generate enhanced input data in this way to increase the robustness and generalization ability of the model.

[0094] ③ Input the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward propagation and backpropagation.

[0095] ④ Pass the input data through the deep learning network model and calculate the predicted class probabilities. The output of this step is the model's prediction for each input sample. The cross-entropy loss function is used to calculate the loss between the model's prediction and the actual label.

[0096] ⑤ Backpropagate the calculated loss and update the model parameters. Through backpropagation, the gradient of the loss with respect to the model parameters is calculated. Use the Adam optimizer to update the model parameters based on the calculated gradient to minimize the loss function.

[0097] ⑥ Repeat the above steps and continuously optimize the model through multiple iterations. In each iteration, train with different batches to improve training efficiency and model generalization ability.

[0098] Through this supervised training process, we can effectively optimize model parameters using labeled data and improve the model's classification performance. During training, the model continuously adjusts its parameters to minimize the loss function, thereby achieving higher classification accuracy and robustness in practical applications.

[0099] S3.2. For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the probability vector p of the predicted class distribution w If the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation;

[0100] S3.3. On one branch of weak enhancement, apply random inactivation of the channel dimension to the weak enhancement spectrum to perform feature perturbation, and obtain p through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 supervision signal;

[0101] S3.4. Calculate the consistency loss function for unlabeled data:

[0102] S3.5. Combine the standard cross entropy loss for labeled data and the consistency loss for unlabeled data to calculate the joint loss function l = l s +θl u .

[0103] Furthermore, in step S3.2, the probability vector p of the class distribution of the weakly enhanced input w =p m (y|α(u b )), the prediction threshold is δ=0.95.

[0104] Furthermore, in step S3.3, the feature perturbation strategy is as follows:

[0105] ew =g(α(u b )),p fp1 =d(P1(e w )),p fp2 =d(P2(e w )), where e w is to extract x w features, P1 represents feature perturbation Dropout, P2 represents adding noise, g, h are the convolution layer and output layer of the model respectively; in the double-branch strong perturbation, p s1 =p m (y|A1(u b )),p s2 =p m (y|A2(u b )).

[0106] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below with reference to specific embodiments.

[0107] Example 1

[0108] S1. Collect sound signals through the sonar array and mark some audio signals.

[0109] S2. Obtain the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data, and unlabeled audio data. Specifically, first, resample the received audio to a sampling rate of 16kHz, frame the audio and perform windowing processing. Use a window length of 1024 and an overlap of 512 to frame the audio. To address spectrum leakage, use a Hamming window as the sliding window. Then, perform a fast Fourier transform on the framed audio, calculate the energy spectrum, use 64 filters to extract Mel spectrum features, and take the logarithm to obtain the logarithmic Mel spectrum, as shown in the following example: Figure 2 shown.

[0110] S3, detailed network model such as Figure 3 As shown in the figure, DW represents depth-wise separable convolution. Each convolution module also contains a BN layer and a nonlinear activation ReLU. The fully connected layer uses the SoftMax activation function to classify and identify sound events.

[0111] S3.1. Three convolution operations of different scales (5, 9, and 17) are used to extract the features of the log-Mel spectrum obtained in S2, and features of different granularities are independently extracted for classification and recognition.

[0112] S3.2. Each channel is primarily composed of an asymmetric depthwise separable convolution module and a residual module. The shallow feature data obtained is subjected to asymmetric depthwise convolution and pointwise convolution operations to extract deep features from the underwater acoustic data. The depthwise separable convolution module performs channel-by-channel convolution operations through depthwise separable convolution and applies the results to all channels, increasing the model depth and enabling parameter sharing, reducing the model's parameters and making it lightweight. Asymmetric convolution further extracts spectral features. The vertical convolution kernel moves horizontally to process features within the same time frame, extracting frequency distribution features with translation invariance along the time axis and obtaining more stable line spectrum features. The horizontal movement of the convolution kernel processes features from consecutive time frames in the frequency domain, emphasizing the extraction of different features in the frequency domain and obtaining more stable temporal features. The residual module utilizes the gradient signal in the residual module to pass the output of one layer as input to another layer of the architecture through "skip connections," reducing the possibility of model overfitting and gradient explosion. Figure 4 Schematic diagram of the asymmetric depth-wise separable convolution module and residual module for each channel.

[0113] S3.3. This method enhances the network's expressive power by concatenating features from different channels. A global average pooling layer compresses the features of each channel at all spatial locations into a single feature value, effectively reducing the number of parameters the model must learn. An output layer further reduces the feature dimensionality to the number of categories, enabling the final prediction of the classification task.

[0114] S3.4. After feature extraction in the above network layer, the output features will be further input into the classifier.

[0115] S4. For labeled data, perform supervised training, and the standard cross entropy loss of weakly expanded labeled examples is

[0116] S5. Perform a weak enhancement strategy and a multi-branch strong enhancement strategy on the log-Mel spectrum of the unlabeled audio data. The multi-branch strong enhancement strategy requires performing two independent strong enhancements on the log-Mel spectrum of the same unlabeled audio data, respectively obtaining a weakly enhanced log-Mel spectrum and two strongly enhanced log-Mel spectrums. The weak enhancement uses a time-domain masking parameter T = 20, while the strong enhancement uses a time-domain masking parameter T = 30. The weak enhancement uses a frequency masking parameter F = 10, while the strong enhancement uses a frequency masking parameter F = 20. Figure 3 Schematic diagram of the strong and weak spectrum enhancement of the present invention.

[0117] S6. Reference Figure 1The logarithmic Mel spectrum of strongly labeled audio data and weakly labeled audio data is used as the input of the deep learning model, the category of the audio data is used as the model output, and the loss function l is calculated. s , perform supervised training; for unlabeled data, take the weakly enhanced log-Mel spectrum and the strongly enhanced log-Mel spectrum as input, expand the perturbation space based on the FixMatch framework, and build a semi-supervised model that combines multiple feature perturbations with strong perturbations. The weakly enhanced features are processed in three branches: one branch is directly input into the model to obtain pseudo labels, and the other two branches are perturbed to obtain prediction results. The pseudo labels generated by weak enhancement are used to supervise the output of the feature perturbation and the output of the two strong perturbations, and the consistency loss is calculated to obtain the loss function l u ; Specifically includes the following sub-steps:

[0118] S6.1. Input the weakly enhanced logarithmic Mel spectrum into the network model to obtain the predicted value, that is, the probability vector p of the predicted class distribution w ;

[0119] S6.2. Determine p w Is it greater than the threshold of 0.95? If so, the weakly enhanced spectrum is determined to be a pseudo label. Then execute step S6.3, otherwise return to execute S6.1;

[0120] S6.3. On one branch of weak enhancement, apply random inactivation of the channel dimension to the weak enhancement spectrum to perform feature perturbation, and obtain p through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 At the same time, the unlabeled data spectrum is subjected to two independent strong perturbations and then input into the model to obtain the prediction result p s1 and p s2 ,final As p fp1 、p fp2 、p s1 and p s2 supervision signal;

[0121] S6.4. Calculate the consistency loss function: Where λ and σ are both set to 0.5.

[0122] S6.5. Combine the standard cross entropy loss for labeled data and the consistency loss for unlabeled data to calculate the joint loss function l = l s +θl u , where θ=1.

[0123] The effects of this application are described in detail below in conjunction with simulation.

[0124] To verify the effectiveness of our method, we used the Shipsear dataset, a ship-radiated noise dataset, to test it. We segmented the audio data into 3-second samples, with a training set, validation set, and test set split ratio of 7:2:1. The distribution of sample signal categories and the specific data split used in the experiment are shown in Table 1.

[0125] Table 1

[0126]

[0127] This paper compares the performance of ShuffleNetV2, Resnet18, AR-CNN, Fixmatch+AR-CNN, and the semi-supervised framework + AR-CNN proposed in this paper under the conditions of 50, 100, and 200 labels. The experimental results are shown in Table 2, which demonstrates the classification accuracy and parameter size of several different deep learning models under different amounts of labeled data.

[0128] Table 2

[0129]

[0130] As can be seen from Table 2, ShuffleNetV2 performs relatively poorly under all conditions of the number of labels. Although its parameter size is small (1.26M), its classification accuracy is significantly lower than other models. Resnet18 performs better than ShuffleNetV2, and its accuracy improves significantly with the increase in the number of labels. However, Resnet18 has a model parameter size of up to 11.17M, which requires high resources. In contrast, AR-CNN performs well under all conditions of the number of labels, with higher accuracy than Resnet18, while its parameter size is only 0.74M. Furthermore, by introducing the Fixmatch semi-supervised learning method, the performance of Fixmatch+AR-CNN is improved with small amounts of labeled data, reaching 60.38% and 69.81% accuracy with 50 and 100 labels, respectively.

[0131] The proposed semi-supervised framework combined with AR-CNN performs best across all label counts. In particular, with 200 labels, its accuracy reaches 81.91%, demonstrating superior classification performance. Meanwhile, the model parameters remain at 0.74M, demonstrating that the proposed method improves performance while maintaining a small parameter size. In summary, the effectiveness of the proposed semi-supervised framework combined with AR-CNN in few-shot underwater sound event detection has been fully demonstrated, outperforming existing methods in both accuracy and parameter efficiency.

[0132] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. A semi-supervised underwater sound event detection method based on joint disturbance consistency constraints, characterized in that: include: S1. Obtain the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data, and unlabeled audio data; S2. Perform weak enhancement and strong enhancement on the logarithmic Mel spectrum of the unlabeled audio data to obtain the corresponding enhanced spectrum; S3. Build a deep learning network model and perform supervised training using the spectra of strongly labeled audio data and weakly labeled audio data. A semi-supervised model combining multiple feature perturbations and strong perturbations is used for unlabeled audio data. Finally, the training process of the deep learning network model is optimized by calculating the joint loss function. The semi-supervised model for unlabeled audio data using a combination of multiple feature perturbations and strong perturbations includes: For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the probability vector p of the predicted class distribution w If the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation; On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 ; At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 supervision signal; S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

2. The semi-supervised underwater sound event detection method based on joint disturbance consistency constraint according to claim 1 is characterized in that: The step S1 is specifically as follows: Fast Fourier transform is performed on the strongly labeled audio data, weakly labeled audio data, and unsigned audio data to convert them into their respective original audio data. The energy spectrum of each is obtained based on the original audio data. 64 Mel filters are used to extract the Mel spectrum features of each energy spectrum. The logarithm of each Mel spectrum feature is taken to obtain the logarithmic Mel spectrum.

3. The semi-supervised underwater sound event detection method based on joint disturbance consistency constraint according to claim 1 is characterized in that: The weak enhancement and strong enhancement processing in step S2 both include time domain masking and frequency domain masking; The time domain masking is performed on the continuous time channels within the range of [t0, t0+t) along the time domain axis, where 0 < t ≤ T, 0 ≤ t0 ≤ τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range; The frequency domain masking is as follows: continuous frequency channels within the range of [f0, f0+f) along the frequency domain axis are masked, where 0<f≤F, 0≤f0≤vf, v is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range.

4. The semi-supervised underwater sound event detection method based on joint disturbance consistency constraint according to claim 1 is characterized in that: The deep learning network model described in step S3 is a depth-separable residual convolutional network structure AR-CNN based on asymmetric convolution, including a multi-scale convolution module, an asymmetric depth-separable convolution module, a residual module, a global average pooling layer and an output layer; the workflow of the deep learning network model is as follows: Input data: Logarithmic Mel-spectrogram as input, containing rich frequency and time information; Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features at different scales; Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module, performing vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters. Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems; feature Splicing: Splicing features from different channels to enhance the network’s expressive power; Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value; Classification prediction: The final features are passed through the output layer to reduce the dimension to the number of categories and complete the final prediction of the classification task; Output result: Output the classification result, that is, the predicted category label.

5. The semi-supervised underwater sound event detection method based on joint disturbance consistency constraint according to claim 4 is characterized in that: The supervised training process includes: Get the labeled audio data from the dataset and convert it into a log-mel spectrogram; get the corresponding labels, which represent the actual categories of the audio data; Weakly enhance the input log-mel spectrogram to generate enhanced input data to increase the robustness and generalization ability of the model; Feed the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward and backpropagation. The input data is passed through the deep learning network model to calculate the predicted category probability; the output of this step is the model's prediction result for each input sample; the cross entropy loss function is used to calculate the loss between the model's predicted value and the actual label; Backpropagate the calculated loss to update the model parameters; calculate the gradient of the loss with respect to the model parameters through backpropagation; use the Adam optimizer to update the model parameters according to the calculated gradient to minimize the loss function; Repeat the above steps and continuously optimize the model through multiple iterations; in each iteration, train with different batches to improve training efficiency and model generalization ability.

6. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

7. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

8. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when the instructions are executed.

Images