Semi-supervised underwater sound event detection method based on joint disturbance consistency constraint

Abstract

The invention relates to a semi-supervised underwater sound event detection method based on joint disturbance consistency constraint. The method comprises the following steps: acquiring logarithmic Mel spectrums of strong label audio data, weak label audio data and unlabeled audio data; performing weak enhancement and strong enhancement processing on the logarithmic Mel spectrum of the unlabeled audio data to obtain a corresponding enhanced spectrum; constructing a deep learning network model, and performing supervised training by using the frequency spectrums of the strong label audio data and the weak label audio data; adopting a multi-feature disturbance and strong disturbance combined semi-supervised model for the unlabeled audio data; and finally, optimizing the training process of the deep learning network model by calculating a joint loss function. And detecting an underwater sound event by using the trained deep learning network model, and outputting an event category of the audio data. The method is used for realizing effective detection of the sound event by using a small amount of strong injection data and improving the model performance.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater acoustic monitoring, and in particular relates to a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints. Background Art

[0002] Marine affairs have become a core part of national planning, and the world is paying close attention to and actively competing for this. In this context, underwater acoustic signals have become a key means of obtaining marine information due to their minimal loss and longest propagation distance in underwater environments. Underwater sound event detection technology (SED) identifies the types of sound events by analyzing the characteristics of underwater acoustic signals, and has important applications in many fields. This technology can accurately, timely and covertly identify underwater targets. In the non-military field, it is not only used to monitor and manage maritime traffic and ensure shipping safety, but also helps collect data on the species and habits of marine organisms, which is of immeasurable value to the protection of marine ecology.

[0003] Early sound event detection used methods such as hidden Markov models, random forest regression, and support vector machines. With the continuous development of deep learning, more and more deep learning-based methods are outperforming traditional methods. In the task of sound event detection, data types are divided into three categories. Data that contains both event categories and the start and end times of the event is called strongly labeled data; data that only contains event categories but not the start and end times of the event is called weakly labeled data; data without any category labels or timestamp information is called unlabeled data. Traditional deep learning methods generally require the use of a large amount of labeled data to train the model through supervised learning to achieve high performance. There are many types of underwater sound events, including various types of ship radiation noise, marine biological noise, sounding signal noise, etc. For underwater sound event detection tasks, this requires a large amount of training data with strong labels, which increases the cost of data annotation.

[0004] In recent years, modeling methods based on semi-supervised learning (SSL) have achieved good performance in recent years by combining limited labeled data with a large amount of unlabeled data to model the SED system, providing a good theoretical and practical basis for the high cost of underwater sound event annotation and the lack of strong labeled data. In semi-supervised learning, FixMatch is a very effective framework for utilizing unlabeled data. Its core idea is to constrain the "strong and weak consistency" of unlabeled data. Perturbations are particularly important in the FixMatch semi-supervised method. Existing methods introduce perturbations through simple data augmentation. Although they can bring improvements, the effects are limited. Summary of the invention

[0005] The technical problems to be solved by the present invention are:

[0006] In view of the problems of insufficient strongly labeled audio data and insufficient utilization of unlabeled audio data in underwater sound event detection, the present invention provides a semi-supervised underwater sound event detection method based on joint perturbation consistency constraints, which is used to effectively detect sound events using a small amount of strongly injected data and improve model performance.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A semi-supervised underwater sound event detection method based on joint disturbance consistency constraint, characterized by comprising:

[0009] S1, obtaining the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data and unlabeled audio data;

[0010] S2, performing weak enhancement and strong enhancement processing on the logarithmic Mel spectrum of the unlabeled audio data to obtain a corresponding enhanced spectrum;

[0011] S3. Build a deep learning network model and use the spectrum of strongly labeled audio data and weakly labeled audio data for supervised training; use a semi-supervised model that combines multiple feature perturbations with strong perturbations for unlabeled audio data; finally, optimize the training process of the deep learning network model by calculating the joint loss function;

[0012] S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

[0013] A further technical solution of the present invention: Step S1 is specifically as follows:

[0014] The strong label audio data, weak label audio data and unsigned audio data are respectively fast Fourier transformed into their respective original audio data, and their respective energy spectra are obtained based on their respective original audio data. 64 Mel filters are used to extract their respective Mel spectrum features from their respective energy spectra, and the logarithm of their respective Mel spectrum features is taken to obtain their respective logarithmic Mel spectrum.

[0015] A further technical solution of the present invention: the weak enhancement and strong enhancement processing in step S2 both include time domain masking and frequency domain masking;

[0016] The time domain masking: masking the continuous time channels in the range of [t0, t0+t) along the time domain axis direction, where 0<t≤T, 0≤t0≤τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range value;

[0017] The frequency domain masking: masking continuous frequency channels within the range of [f0, f0+f) along the frequency domain axis, where 0<f≤F, 0≤f0≤ν-f, ν is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range value.

[0018] A further technical solution of the present invention: The deep learning network model described in step S3 is a deep separable residual convolutional network structure AR-CNN based on asymmetric convolution, including a multi-scale convolution module, an asymmetric deep separable convolution module, a residual module, a global average pooling layer and an output layer; the workflow of the deep learning network model is as follows:

[0019] Input data: Logarithmic Mel-spectrogram as input, containing rich frequency and time information;

[0020] Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features of different scales;

[0021] Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module, which performs vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters.

[0022] Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems;

[0023] Feature splicing: Features from different channels are spliced ​​to enhance the network’s expressiveness;

[0024] Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value;

[0025] Classification prediction: The final features are passed through the output layer to reduce the dimension to the number of categories and complete the final prediction of the classification task;

[0026] Output results: Output the classification results, that is, the predicted category labels.

[0027] A further technical solution of the present invention: the supervised training process includes:

[0028] Get the labeled audio data from the dataset and convert it into a logarithmic Mel-spectrogram; get the corresponding labels, which represent the actual categories of the audio data;

[0029] Weakly enhance the input log-mel spectrogram to generate enhanced input data to increase the robustness and generalization ability of the model;

[0030] Feed the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward and back-propagation.

[0031] The input data is passed through the deep learning network model to calculate the predicted category probability; the output of this step is the model's prediction result for each input sample; the cross entropy loss function is used to calculate the loss between the model's predicted value and the actual label;

[0032] Back-propagate the calculated loss to update the model parameters; calculate the gradient of the loss relative to the model parameters through back-propagation; use the Adam optimizer to update the model parameters according to the calculated gradient to minimize the loss function;

[0033] Repeat the above steps and continuously optimize the model through multiple iterations; in each iteration, train with different batches to improve training efficiency and the generalization ability of the model.

[0034] A further technical solution of the present invention is as follows: the semi-supervised model combining multiple feature perturbations and strong perturbations is used for unlabeled audio data, including:

[0035] For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the probability vector p of the predicted class distribution w , if the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation;

[0036] On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 ; At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 supervision signal.

[0037] A computer system, characterized in that it comprises: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0038] A computer-readable storage medium, characterized in that it stores computer-executable instructions, which are used to implement the above method when executed.

[0039] A computer program product, characterized by comprising computer executable instructions, wherein the instructions are used to implement the above method when executed.

[0040] The beneficial effects of the present invention are:

[0041] The semi-supervised underwater sound event detection method based on joint disturbance consistency constraint provided by the present invention has the following advantages compared with the prior art:

[0042] 1. Using a small amount of labeled data and a large amount of unlabeled data for semi-supervised learning, this method significantly reduces the dependence on a large amount of labeled data and solves the challenge of scarce ocean acoustic data. This is of great significance for improving the feasibility and practicality of underwater sound event detection.

[0043] 2. The present invention adopts a joint disturbance consistency constraint strategy, which combines multiple characteristic disturbances with strong disturbances, effectively expands the disturbance space of the model, and improves the adaptability and robustness of the model in unknown or rare events.

[0044] 3. The asymmetric residual convolutional neural network proposed in the invention, by combining asymmetric convolution and depthwise separable convolution technology, not only improves the processing speed and accuracy, but also realizes the lightweight network architecture, making the system more suitable for deployment on underwater equipment with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0046] Figure 1 4 is a diagram showing the structure of a semi-supervised model for detecting sound events in an embodiment of the present invention.

[0047] Figure 2 Schematic diagram of the logarithmic Mel spectrum of ship radiation noise according to the present invention.

[0048] Figure 3 This is a detailed network model structure diagram of the present invention.

[0049] Figure 4 This is a structural diagram of the asymmetric depth-separable convolution module and residual module of the present invention.

[0050] Figure 5 It is a schematic diagram of the strong and weak spectrum enhancement of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] The present invention proposes a method for underwater sound event detection based on joint disturbance consistency constraints, which combines multiple strong disturbances and feature disturbances in a consistency constraint framework to effectively expand the disturbance space. This design not only improves the robustness of the model in dealing with various input changes, but also optimizes the utilization efficiency of unlabeled data. Specifically, by adding disturbance channels, the adaptability of the model to different disturbances is improved, and the feature disturbance further broadens the disturbance space of the model and enhances the model's ability to handle unknown changes. In addition, the present invention uses an asymmetric residual convolutional network to accurately identify underwater sound events, thereby improving the detection accuracy and efficiency of the model in complex underwater environments. These technical improvements not only ensure efficient model training under the condition of limited labeled data, but also enable the model to adapt to a wider range of data variations, thereby improving the overall performance and reliability of underwater sound event detection.

[0054] like Figure 1 As shown, the following steps are included:

[0055] S1, obtaining the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data and unlabeled audio data;

[0056] Specifically, strongly labeled data refers to audio data with complete annotation information, which usually contains information about event categories and event start and end times; among them, event categories: the categories of sound events that occur in the audio (such as different types of underwater sound sources, such as ships, marine life, etc.). Event start and end times: the specific time period when the event occurs, usually including the start time and end time of the event. This is a more accurate label.

[0057] Specifically, weakly labeled data only provides the event category, but does not mark the specific time period. Therefore, weakly labeled data cannot accurately indicate the start and end time of the event, but can only indicate which category the sound event in the audio belongs to (for example, a certain type of ship noise or the sound of marine life).

[0058] Specifically, unlabeled audio data: Unlabeled data does not have any label information, including no event category or event time. This type of data is unlabeled raw data and is usually used in the unsupervised training process in semi-supervised learning. The model is trained by inferring or generating pseudo labels.

[0059] S2, performing weak enhancement and strong enhancement processing on the logarithmic Mel spectrum of the unlabeled audio data to obtain a corresponding enhanced spectrum;

[0060] S3. Build a deep learning network model and use the spectrum of strongly labeled audio data and weakly labeled audio data for supervised training; use a semi-supervised model that combines multiple feature perturbations with strong perturbations for unlabeled audio data; finally, optimize the training process of the deep learning network model by calculating the joint loss function;

[0061] S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

[0062] Further, step S1 is specifically as follows: obtaining the logarithmic Mel spectrum includes applying fast Fourier transform to the original audio data, obtaining the energy spectrum, extracting Mel spectrum features using 64 Mel filters, and taking the logarithm to obtain the logarithmic Mel spectrum.

[0063] Strongly labeled data: After extracting the logarithmic Mel spectrum, it is aligned with the category and time label of the event to form a training sample. Each sample includes features (logarithmic Mel spectrum) and precise labels (category and time period).

[0064] Weakly labeled data: extract the log-Mel spectrum, align with the class label, but do not provide temporal information. Each sample includes features and class labels.

[0065] Unlabeled data: Only extract the log-Mel spectrum without aligning it with any label. Used for pseudo-label generation or feature perturbation training in semi-supervised learning.

[0066] Furthermore, the weak-strong enhancement strategy in step S2 is time-frequency dimension masking. The time domain dimension masks the frequency dimension as follows: masking the continuous time channels in the range of [t0, t0+t) along the time domain axis, where 0<t≤T, 0≤t0≤τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range value. The frequency domain dimension masks the frequency as follows: masking the continuous frequency channels in the range of [f0, f0+f) along the frequency domain axis, where 0<f≤F, 0≤f0≤ν-f, ν is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range value.

[0067] Temporal masking:

[0068] Weak enhancement: The duration of the temporal masking is T = 20, which means that each masking period is relatively short. By masking a short period of time, local perturbations can be performed without disturbing the overall framework of the event, allowing the model to learn how to identify important features in these incomplete periods.

[0069] Strong enhancement: The duration of the temporal masking is T=30, which means that each masking period is longer. This means that strong enhancement will further blur the temporal information of the audio, causing the model to face more inaccurate input data during training. This operation simulates more interference, allowing the model to identify underwater sound events in an inaccurate manner.

[0070] Frequency masking:

[0071] Weak enhancement: The frequency masking range is F=10, which means that the masked frequency bandwidth is narrow. Through small-range frequency perturbations, the model can adapt to more subtle frequency changes, especially when the labels are incomplete, and a certain frequency information can be maintained during the training process, so as not to completely lose useful spectral features.

[0072] Strong enhancement: The frequency masking range is F=20, which means that the masked frequency bandwidth is wider. This will cause a larger range of frequency information to be lost, resulting in a more inaccurate spectrum processed by the model. This is to simulate strong interference and allow the model to be trained in more ambiguous spectrum information, so as to better cope with noise and other uncertainties in complex underwater environments.

[0073] The purpose of weak enhancement is to increase local disturbances, but relatively maintain the "precision" of the data, so that the model can be trained in the presence of noise or uncertain labels and still be able to identify underwater sound events. Strong enhancement deliberately makes the data more imprecise through greater disturbances (longer time and wider frequency masking), with the aim of allowing the model to be trained on incomplete or distorted data, improving its robustness in the face of noise and imperfect data in practical applications.

[0074] Furthermore, the deep learning network model is a deep separable residual convolutional network structure (AR-CNN) based on asymmetric convolution, which aims to efficiently extract and classify the features of underwater acoustic data. The core components of the network include multi-scale convolution module, asymmetric deep separable convolution module and residual module. The following is a detailed description of each module:

[0075] Multi-scale convolution module: Three convolution operations of different scales (5, 9, 17) are used to extract features of different granularities from the log-Mel spectrum. Each convolution kernel extracts features independently, which enhances the network's sensitivity to different features.

[0076] Asymmetric depthwise separable convolution module: The vertical convolution kernel moves on the time axis, processes the features of the same time frame, and extracts frequency distribution features with translation invariance. The horizontal convolution kernel processes the features of continuous time frames in the frequency domain and extracts different time features. Point-by-point convolution operations are performed on each channel to achieve parameter sharing, increase the model depth while reducing the number of parameters, and make the model lightweight.

[0077] Residual module: Using the gradient signal in the residual module, the output of one layer is passed as input to another layer of the architecture through a "skip connection", reducing the possibility of model overfitting and gradient explosion.

[0078] Global average pooling layer: compresses the features of each channel at all spatial positions into a single feature value, reducing the number of parameters that the model needs to learn.

[0079] Output layer: An output layer is used to reduce the dimension of the features to the number of categories to complete the final prediction of the classification task.

[0080] Workflow Description:

[0081] ① Input data: Logarithmic Mel spectrum graph is used as input, which contains rich frequency and time information.

[0082] ② Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features of different scales.

[0083] ③ Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module to perform vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters.

[0084] ④ Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems.

[0085] ⑤ Feature splicing: Features from different channels are spliced ​​to enhance the network’s expressive power.

[0086] ⑥ Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value.

[0087] ⑦Classification prediction: The final features are passed through the output layer to reduce the dimension to the number of categories and complete the final prediction of the classification task.

[0088] ⑧Output results: Output the classification results, that is, the predicted category labels.

[0089] Furthermore, the loss function calculation includes the following sub-steps:

[0090] S3.1. For labeled data, supervised training is performed, and the cross entropy loss function of the weakly expanded labeled examples is: In which, let χ={(x b ,p b ):b∈(1,...,B)} is a labeled sample, where x b is the training sample, p b is the label, U={u b :b∈(1,...,μ B )} is the unlabeled data, p m (y|x) is the predicted category distribution generated by the model for the input x. The loss function H(p,q) is the cross entropy loss function, and strong enhancement and weak enhancement are represented by A(·) and α(·), respectively.

[0091] The above supervised training process is as follows:

[0092] ①First, obtain the labeled audio data from the dataset and convert it into a logarithmic Mel-spectrogram. Obtain the corresponding labels, which represent the actual categories of the audio data.

[0093] ② Weakly enhance the input logarithmic Mel-spectrogram to generate enhanced input data in this way to increase the robustness and generalization ability of the model.

[0094] ③ Input the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward propagation and backpropagation.

[0095] ④ Pass the input data through the deep learning network model to calculate the predicted category probability. The output of this step is the model's prediction result for each input sample. The cross entropy loss function is used to calculate the loss between the model's predicted value and the actual label.

[0096] ⑤ Back-propagate the calculated loss and update the model parameters. Through back-propagation, the gradient of the loss relative to the model parameters is calculated. Use the Adam optimizer to update the model parameters according to the calculated gradient to minimize the loss function.

[0097] ⑥ Repeat the above steps and continuously optimize the model through multiple iterations. In each iteration, train with different batches to improve training efficiency and the generalization ability of the model.

[0098] Through the above supervised training process, the model parameters can be effectively optimized using labeled data to improve the classification performance of the model. During the training process, the model continuously adjusts the parameters to minimize the loss function, thereby achieving higher classification accuracy and robustness in practical applications.

[0099] S3.2. For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the predicted class distribution probability vector p w , if the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation;

[0100] S3.3. On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 The supervisory signal;

[0101] S3.4. Calculate the consistency loss function of unlabeled data:

[0102] S3.5. Combine the standard cross entropy loss of labeled data and the consistency loss of unlabeled data to calculate the joint loss function l = l s +θl u .

[0103] Furthermore, in step S3.2, the probability vector p of the class distribution of the weakly enhanced input w =p m (y|α(u b )), the prediction threshold is δ=0.95.

[0104] Furthermore, in step S3.3, the feature perturbation strategy is as follows:

[0105] e w=g(α(u b )),p fp1 =d(Ρ1(e w )),p fp2 =d(Ρ2(e w )), where e w is to extract x w , Ρ1 represents feature perturbation Dropout, Ρ2 represents adding noise, g and h are the convolutional layer and output layer of the model respectively; in the two-branch strong perturbation, p s1 =p m (y|A1(u b )),p s2 =p m (y|A2(u b )).

[0106] In order to enable those skilled in the art to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0107] Example 1

[0108] S1. Collect sound signals through the sonar array and mark some audio signals.

[0109] S2. Obtain the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data, and unlabeled audio data. Specifically, first, resample the received audio to a sampling rate of 16kHz, perform frame and window processing on the audio, use a window length of 1024 and an overlap of 512 to frame the audio, and use a Hamming window to solve the spectrum leakage problem. Then, perform fast Fourier transform on the framed audio, calculate the energy spectrum, use 64 filters to extract Mel spectrum features, and take the logarithm to obtain the logarithmic Mel spectrum, such as Figure 2 shown.

[0110] S3, detailed network model as follows Figure 3 As shown in the figure, DW represents depth-wise separable convolution. Each convolution module also contains a BN layer and a nonlinear activation ReLU. The fully connected layer uses the SoftMax activation function to classify and identify sound events.

[0111] S3.1, three different scales (5, 9, 17) of convolution operations are used to extract the features of the logarithmic Mel spectrum obtained in S2, and features of different granularities are independently extracted for classification and recognition.

[0112] S3.2, each channel is mainly composed of an asymmetric depth-separable convolution module and a residual module. The shallow feature data obtained is subjected to asymmetric depth-separable convolution and point-by-point convolution operations to extract the deep features of the underwater acoustic data. The depth-separable convolution module performs channel-by-channel convolution operations through depth-separable convolution, and applies the results to all channels to enhance the depth of the model and realize parameter sharing, reduce the parameters of the model, and make the model lightweight. Asymmetric convolution can further extract spectrogram features, and the vertical convolution kernel moves horizontally to process the features of the same time frame, so that it extracts frequency distribution features with translation invariance on the time axis, and obtains more stable line spectrum features. The movement of the horizontal convolution kernel processes the features of continuous time frames in the frequency domain, emphasizing the extraction of different features in the frequency domain, and obtaining more stable time features. The residual module network uses the gradient signal in the residual module to realize the output of one layer as input to another layer of the architecture through "jump connection", reducing the possibility of model overfitting and gradient explosion. Figure 4 Schematic diagram of the asymmetric depth-wise separable convolutional module and residual module for each channel.

[0113] S3.3. This method enhances the network's expressiveness by concatenating features from different channels. A global average pooling layer is used to compress the features of each channel at all spatial positions into a single feature value. This step effectively reduces the number of parameters that the model needs to learn. Further, an output layer is used to reduce the dimension of the features to the number of categories to complete the final prediction of the classification task.

[0114] S3.4. After feature extraction in the above network layer, the output features will be further input into the classifier.

[0115] S4. For labeled data, supervised training is performed, and the standard cross entropy loss of weakly expanded labeled examples is

[0116] S5. Perform a weak enhancement strategy and multiple strong enhancement strategies on the logarithmic Mel spectrum of the unlabeled audio data, wherein the multiple strong enhancement strategy requires two independent strong enhancements on the logarithmic Mel spectrum of the same unlabeled data, respectively obtaining a weakly enhanced logarithmic Mel spectrum and two strongly enhanced logarithmic Mel spectrums; the parameter T in the time domain masking means used in the weak enhancement is 20, and the parameter T in the time domain masking means used in the strong enhancement is 30. The parameter F in the frequency masking means used in the weak enhancement is 10, and the parameter F in the frequency masking means used in the strong enhancement is 20. Figure 3 It is a schematic diagram of the strong and weak spectrum enhancement of the present invention.

[0117] S6. Reference Figure 1The logarithmic Mel spectrum of strongly labeled audio data and weakly labeled audio data is used as the input of the deep learning model, and the category of the audio data is used as the model output to calculate the loss function l s , perform supervised training; for unlabeled data, use weakly enhanced log-Mel spectrum and strongly enhanced log-Mel spectrum as input, expand the perturbation space based on the FixMatch framework, and build a semi-supervised model that combines multiple feature perturbations with strong perturbations. The weakly enhanced features are processed in three branches: one branch is directly input into the model to obtain pseudo labels, and the other two branches are perturbed to obtain prediction results. Use the pseudo labels generated by weak enhancement to supervise the output of feature perturbations and the outputs of the two strong perturbations, and calculate the consistency loss to obtain the loss function l u ; Specifically includes the following sub-steps:

[0118] S6.1. Input the weakly enhanced logarithmic Mel spectrum into the network model to obtain the predicted value, that is, the probability vector p of the predicted class distribution w ;

[0119] S6.2. Determine p w Is it greater than the threshold of 0.95? If so, the weakly enhanced spectrum is determined to be a pseudo label. Then execute step S6.3, otherwise return to execute S6.1;

[0120] S6.3. On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 At the same time, the unlabeled data spectrum is subjected to two independent strong perturbations and then input into the model to obtain the prediction result p s1 and p s2 ,final As p fp1 、p fp2 、p s1 and p s2 The supervisory signal;

[0121] S6.4. Calculate the consistency loss function: Where λ and σ are both set to 0.5.

[0122] S6.5. Combine the standard cross entropy loss of labeled data and the consistency loss of unlabeled data to calculate the joint loss function l = l s +θl u , where θ=1.

[0123] The effects of this application are described in detail below in conjunction with simulation.

[0124] In order to verify the effectiveness of the method of the present invention, the following verification is carried out on the ship radiation noise dataset Shipsear, the audio data is divided into samples of 3 seconds each, and the ratio of the training set, validation set and test set is 7:2:1. The distribution of the sample signal categories used in the experiment and the specific division of the data are shown in Table 1.

[0125] Table 1

[0126]

[0127] The present invention compares the performance of ShuffleNetV2, Resnet18, AR-CNN, Fixmatch+AR-CNN and the semi-supervised framework + AR-CNN proposed in the present invention under the conditions of 50, 100 and 200 labels. The experimental results are shown in Table 2, which shows the classification accuracy and parameter scale of several different deep learning models under different amounts of labeled data.

[0128] Table 2

[0129]

[0130] As can be seen from Table 2, ShuffleNetV2 performs relatively poorly under all conditions of the number of labels. Although its parameter scale is small (1.26M), its classification accuracy is significantly lower than other models. Resnet18 performs better than ShuffleNetV2, and its accuracy increases significantly with the increase in the number of labels. However, the model parameters of Resnet18 are as high as 11.17M, and its resource requirements are high. In contrast, AR-CNN performs well under all conditions of the number of labels, with higher accuracy than Resnet18, and its parameter scale is only 0.74M. Furthermore, by introducing the Fixmatch semi-supervised learning method, the performance of Fixmatch+AR-CNN with a small amount of labeled data has been improved, with an accuracy of 60.38% and 69.81% in the case of 50 labels and 100 labels, respectively.

[0131] The semi-supervised framework + AR-CNN proposed in the present invention performs best under all conditions of the number of labels. In particular, in the case of 200 labels, its accuracy reaches 81.91%, showing superior classification performance. At the same time, the model parameters are maintained at 0.74M, which verifies that the method proposed in the present invention can maintain a small parameter scale while improving performance. In summary, the effectiveness of the semi-supervised framework + AR-CNN proposed in the present invention in the detection of underwater sound events with few samples has been fully verified, and it is superior to existing methods in terms of accuracy and parameter efficiency.

[0132] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should be included in the protection scope of the present invention.

Claims

1. A semi-supervised underwater sound event detection method based on joint disturbance consistency constraints, characterized in that: include: S1, obtaining the logarithmic Mel spectrum of strongly labeled audio data, weakly labeled audio data and unlabeled audio data; S2, performing weak enhancement and strong enhancement processing on the logarithmic Mel spectrum of the unlabeled audio data to obtain a corresponding enhanced spectrum; S3, build a deep learning network model and perform supervised training using the spectrum of strongly labeled audio data and weakly labeled audio data; A semi-supervised model combining multiple feature perturbations and strong perturbations is used for unlabeled audio data. Finally, the training process of the deep learning network model is optimized by calculating the joint loss function. S4. Use the trained deep learning network model to detect underwater sound events and output the event category of the audio data.

2. According to claim 1, a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints is characterized in that: The step S1 is specifically as follows: The strong label audio data, weak label audio data and unsigned audio data are respectively fast Fourier transformed into their respective original audio data, and their respective energy spectra are obtained based on their respective original audio data. 64 Mel filters are used to extract their respective Mel spectrum features from their respective energy spectra, and the logarithm of their respective Mel spectrum features is taken to obtain their respective logarithmic Mel spectrum.

3. According to claim 1, a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints is characterized in that: The weak enhancement and strong enhancement processing in step S2 both include time domain masking and frequency domain masking; The time domain masking: masking the continuous time channels in the range of [t0, t0+t) along the time domain axis direction, where 0<t≤T, 0≤t0≤τ-t, T is the time domain masking parameter, τ is the duration of the spectrogram, and t and t0 are randomly selected within the range value; The frequency domain masking: masking continuous frequency channels within the range of [f0, f0+f) along the frequency domain axis, where 0<f≤F, 0≤f0≤ν-f, ν is the number of channels, F is the frequency masking parameter, and f and f0 are randomly selected within the range value.

4. According to claim 1, a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints is characterized in that: The deep learning network model described in step S3 is a deep separable residual convolutional network structure AR-CNN based on asymmetric convolution, including a multi-scale convolution module, an asymmetric deep separable convolution module, a residual module, a global average pooling layer and an output layer; the workflow of the deep learning network model is as follows: Input data: Logarithmic Mel-spectrogram as input, containing rich frequency and time information; Feature extraction: The data passes through a three-channel multi-scale convolution module to extract features of different scales; Asymmetric depth-wise separable convolution processing: Shallow features are processed through an asymmetric depth-wise separable convolution module, which performs vertical and horizontal convolution as well as point-by-point convolution to extract deep features and share parameters. Residual module processing: The extracted deep features are further processed through the residual module using skip connections to avoid gradient vanishing and explosion problems; feature Splicing: Splicing the features of different channels to enhance the network's expressiveness; Global average pooling: After the global average pooling layer, the features at all spatial positions are compressed into a single feature value; Classification prediction: The final features are passed through the output layer to reduce the dimension to the number of categories and complete the final prediction of the classification task; Output results: Output the classification results, that is, the predicted category labels.

5. According to claim 4, a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints is characterized in that: The supervised training process includes: Get the labeled audio data from the dataset and convert it into a logarithmic Mel-spectrogram; get the corresponding labels, which represent the actual categories of the audio data; Weakly enhance the input log-mel spectrogram to generate enhanced input data to increase the robustness and generalization ability of the model; Feed the data into the AR-CNN model and set the model to training mode to enable gradient calculation during forward and back-propagation. The input data is passed through the deep learning network model to calculate the predicted category probability; the output of this step is the model's prediction result for each input sample; the cross entropy loss function is used to calculate the loss between the model's predicted value and the actual label; Back-propagate the calculated loss to update the model parameters; calculate the gradient of the loss relative to the model parameters through back-propagation; use the Adam optimizer to update the model parameters according to the calculated gradient to minimize the loss function; Repeat the above steps and continuously optimize the model through multiple iterations; in each iteration, train with different batches to improve training efficiency and the generalization ability of the model.

6. According to claim 1, a semi-supervised underwater sound event detection method based on joint disturbance consistency constraints is characterized in that: The semi-supervised model using multiple feature perturbations and strong perturbations for unlabeled audio data includes: For unlabeled data, the weakly enhanced spectrum is input into the deep learning network model AR-CNN to obtain the predicted value corresponding to the weakly enhanced spectrum, that is, the probability vector p of the predicted class distribution w , if the predicted value is greater than the set decision threshold δ, the predicted value is used as a pseudo label And further perform feature perturbation; On one branch of weak enhancement, random inactivation of the channel dimension is applied to the weak enhancement spectrum for feature perturbation, and p is obtained through model prediction. fp1 ; In the other branch of weak enhancement, Gaussian white noise is added to the feature representation of the selected layer, and p is obtained through model prediction. fp2 ; At the same time, the unlabeled data spectrum is subjected to two independent strong enhancement processes, and then input into the model to obtain the prediction result p s1 and p s2 , and finally p w As p fp1 、p fp2 、p s1 and p s2 supervision signal.

7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images