A noise event detection method and device

Abstract

The present invention discloses a noise event detection method and device, which comprises: obtaining environmental audio data, if the sound pressure of the environmental audio data is lower than a threshold, extracting acoustic feature data of the environmental audio data and inputting the data into a noise event detection network, generating frame-level time prediction results and event category prediction results; the noise event detection network is trained by an initial noise event detection network through training data; the training data is constructed from labeled noise data; the encoder network of the initial noise event detection network is trained by the initial encoder network with unlabeled noise data; after filtering the acoustic feature data and correcting the abnormal points, the start and end time points are detected, and the start and end time annotations are generated, and the noise event detection results are generated, so as to realize the training of the encoder network of the initial noise event detection network through the unlabeled noise data, and jointly optimize the noise event detection network through the labeled data, so as to improve the performance and effect of noise event detection.

Description

Technical Field

[0001] The present invention relates to the technical field of sound event detection, and in particular to a noise event detection method, device and storage medium. Background Art

[0002] As urban noise issues become increasingly prominent, the demand for noise pollution prevention and control is gradually increasing, and the noise monitoring system needs to be continuously improved. Noise event detection belongs to the field of sound event detection in audio pattern recognition. Its purpose is to detect whether a target noise audio event appears in an audio segment and mark its start and end time. By analyzing the noise event detection data, it can provide regional noise events and statistical analysis for noise monitoring management departments.

[0003] At present, deep learning methods have achieved a lot of results in the field of audio pattern recognition, providing a solid foundation for noise event detection. However, in traditional supervised learning tasks, the lack of noise annotated data and insufficient real-world scene data sets lead to poor performance of traditional supervised training methods and poor training accuracy for audio data. In reality, there is a large amount of unlabeled data, and in self-supervised learning tasks, the audio data scene is usually relatively simple, and there is no calibration training using field data, which leads to low optimization efficiency for noise detection models and poor audio detection performance. Summary of the invention

[0004] The present invention provides a noise event detection method and device, so as to realize the first encoding network of the initial noise event detection network training through unlabeled noise data, and jointly optimize the noise event detection network through labeled data, so as to improve the performance and effect of noise event detection.

[0005] The present invention provides a noise event detection method and device, comprising: obtaining environmental audio data, and detecting whether the sound pressure of the environmental audio data is lower than a first threshold; if so, extracting first acoustic feature data of the environmental audio data; inputting the first acoustic feature data into a noise event detection network, and generating a frame-level time prediction result and an event category prediction result;

[0006] The noise event detection network is trained by training the initial noise event detection network with training data; the training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by training the first initial encoder network with unlabeled noise data;

[0007] After the first acoustic feature data is screened and outliers are corrected, the start and end time points are detected and start and end time labels are generated; the start and end time labels and the event category prediction results are used as noise event detection results.

[0008] Furthermore, the first encoder network is trained by the first initial encoder network using unlabeled noise data, specifically:

[0009] Acquiring unlabeled noise data, and performing acoustic feature extraction on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data;

[0010] The first encoder network is iteratively trained according to the second acoustic feature data; when the number of iterations reaches a preset value, the iterative training is completed; and the first encoder network after the iterative training is used as the encoder network of the initial noise event detection network.

[0011] Further, the first encoder network is iteratively trained according to the second acoustic feature data; when the number of iterations reaches a preset value, the iterative training is completed, specifically:

[0012] Performing a first data enhancement and a second data enhancement on the second acoustic feature data respectively, to obtain first audio transformation data and second audio transformation data respectively; wherein the enhancement amplitudes of the first data enhancement and the second data enhancement are different;

[0013] Inputting the first audio transformation data into a first initial encoder network and a first multilayer perceptron network connected in sequence to generate first feature representation data; inputting the second audio transformation data into a second initial encoder network to generate second feature representation data; the first initial encoder network and the second initial encoder network have the same structure;

[0014] Calculating the cosine similarity loss of the first feature representation data and the second feature representation data; updating the parameters of the first initial encoder network and the second initial encoder network respectively according to the back propagation of the cosine similarity loss; wherein the second initial encoder network is updated using the momentum method;

[0015] The prediction data and the feature data are recalculated according to the first audio transformation data, the second audio transformation data and the updated first initial encoder network and the second initial encoder network until the number of updates reaches a preset value, thereby completing the iterative training.

[0016] As a preferred solution, the present invention trains an encoder network through unlabeled noise data as an encoder module of an initial noise event detection network, which is used to pre-train the input data of the initial noise event detection network, and can obtain an audio representation for noise scenes while retaining time domain information.

[0017] Furthermore, the noise event detection network is trained by using training data to train the initial noise event detection network, wherein the training data is constructed from the labeled noise data, specifically:

[0018] The obtained manually annotated strong label data set and the single-category weak label data set are randomly mixed to generate a training data set; the acoustic features of the training data in the training data set are extracted to generate third acoustic feature data; the enhanced third acoustic feature data is input into the initial noise event detection network to generate an initial prediction result;

[0019] According to the initial prediction result, the global loss of the initial noise event detection network is calculated; the global loss is back-propagated, and the parameters of the initial noise event detection network are iteratively updated until the global loss calculated using the updated initial noise event detection network converges, and the updated initial noise event detection network is used as the noise event detection network.

[0020] Furthermore, the acquired manually annotated strong label dataset and the single-category weak label dataset are randomly mixed to generate a training dataset, specifically:

[0021] Obtain manually annotated strong label data and several single-category first noise audios, and perform silence detection on the first noise audios; according to the detection result of the silence detection, remove the silence audios of the first noise audios to obtain several single-category second noise audios; randomly mix several second noise audios with manually annotated strong label data to generate a training data set with strong labels of event start and end times and weak labels of corresponding event names.

[0022] Furthermore, the enhanced third acoustic feature data is input into the initial noise event detection network to generate an initial prediction result, specifically:

[0023] The third acoustic feature data is input into the initial noise event detection network, and the third acoustic feature data is controlled to output the first feature after passing through the first encoder network; the first feature is controlled to output the second feature after passing through the bidirectional gated recurrent neural network; the second feature is controlled to output the first matrix after the dimension reduction is performed on the second feature through the second multi-layer perceptron network; the first matrix is ​​controlled to output the initial frame-level time prediction result through the activation function; and the first matrix is ​​input into the aggregation layer to output the initial event category prediction result.

[0024] Furthermore, after the first matrix is ​​input into the aggregation layer, the initial event category prediction result is output, specifically:

[0025] Get the first matrix output by the multilayer perceptron module: [C, L]; where C is the number of noise categories and L is the time domain clue;

[0026] The first matrix is ​​input into the aggregation layer for aggregation, and the prediction result of the category is output as the initial event category prediction result; wherein the aggregation layer adopts mean aggregation and maximum aggregation:

[0027] Y class =(Y frame_max +Y frame_mean ) / 2;

[0028] Y class is the predicted result of the category, Y frame is the initial frame-level temporal prediction result; Y frame_max is the maximum value of the initial frame-level temporal prediction result, Y frame_mean is the mean of the initial frame-level temporal prediction results.

[0029] Further, according to the initial prediction result, the global loss of the initial noise event detection network is calculated, specifically:

[0030] Calculate the binary cross entropy loss of the initial event category prediction results and the smoothed absolute error loss of the initial frame-level time prediction results; calculate the global loss function based on the binary cross entropy loss and the smoothed absolute error loss:

[0031] L(X)=BCE(Y class ,Z class )+α*SmoothL1loss(Y frame ,Z frame );

[0032] Among them, L(X) is the global loss function; BCE(Y class ,Z class ) is the binary cross entropy loss, SmoothL1loss(Y frame ,Z frame ) is the smoothed absolute error loss, Y class is the initial event category prediction result, Z class is the event label, Y frame is the initial frame-level temporal prediction result, Z frame is the frame-level label and α is a hyperparameter.

[0033] As a preferred solution, the initial noise event detection network of the present invention adjusts the network structure, pre-trains the input data using a trained encoder module, and uses a gated recurrent neural network for time series modeling to obtain a noise event category with time information; and uses the prediction of strong and weak labels to jointly optimize the initial noise event detection network to obtain the final noise event detection network, thereby improving the performance of the noise event detection network in noise event detection.

[0034] Furthermore, the first acoustic feature data is screened and outliers are corrected, specifically:

[0035] According to the confidence of the event category prediction result, the first acoustic feature data with the confidence higher than the second threshold is screened out; the frame-level time prediction result of the screened third acoustic feature data is detected for outliers; if an outlier exists, the outlier is corrected.

[0036] As a preferred solution, the noise event detection method of the present invention adopts a self-supervised training method of contrastive learning, effectively uses the collected unlabeled noise data to pre-train the encoder module and the multi-layer perceptron module of the noise event detection network, obtains an audio representation for the noise scene, retains the time domain information, and captures the dependencies with large intervals in the time series data through the gated recurrent neural network module, performs time series modeling, and improves the accuracy and efficiency of processing unlabeled environmental audio data; in addition, the training loss function of the noise detection model is adjusted using labeled noise data, the noise detection network is optimized, the performance of the noise detection network is improved, and the accuracy of noise event detection is improved.

[0037] Accordingly, the present invention also provides a noise event detection device, comprising: a detection module and a result generation module;

[0038] The detection module is used to obtain environmental audio data and detect whether the sound pressure of the environmental audio data is lower than a first threshold; if so, extract first acoustic feature data of the environmental audio data; input the first acoustic feature data into a noise event detection network to generate a frame-level time prediction result and an event category prediction result;

[0039] The noise event detection network is trained by training the initial noise event detection network with training data; the training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by training the first initial encoder network with unlabeled noise data;

[0040] The result generation module is used to screen the first acoustic feature data and correct abnormal points, then detect the start and end time points and generate start and end time labels; and use the start and end time labels and the event category prediction results as noise event detection results.

[0041] As a preferred solution, the detection module of the noise event detection device of the present invention adopts a self-supervised training method of contrastive learning, and effectively uses the collected unlabeled environmental audio data to pre-train through the encoder module and the multi-layer perceptron module of the noise event detection network, so as to obtain an audio representation for the noise scene, retain the time domain information, and capture the dependencies with large intervals in the time series data through the gated recurrent neural network module, and perform time series modeling, thereby improving the accuracy and efficiency of processing the unlabeled environmental audio data; in addition, the training module uses the labeled noise data to adjust the training loss function of the noise detection model, optimizes the noise detection network, improves the performance of the noise detection network, and improves the accuracy of noise event detection.

[0042] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute a noise event detection method as described in the content of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of an embodiment of a noise event detection method provided by the present invention;

[0044] Figure 2 It is a structural schematic diagram of an embodiment of a noise event detection device provided by the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Embodiment 1

[0047] Please refer to Figure 1 , a noise event detection method provided by an embodiment of the present invention, comprising steps S101-S102:

[0048] Step S101: Acquire environmental audio data, and detect whether the sound pressure of the environmental audio data is lower than a first threshold; if so, extract first acoustic feature data of the environmental audio data; input the first acoustic feature data into a noise event detection network to generate a frame-level time prediction result and an event category prediction result;

[0049] The noise event detection network is trained by training the initial noise event detection network with training data; the training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by training the first initial encoder network with unlabeled noise data;

[0050] As a preferred solution, detecting whether the sound pressure of the ambient audio data is lower than a first threshold value is specifically as follows:

[0051] Calculate the intermediate variable sound pressure p e , the unit is Pa, sound pressure p e The calculation formula is as follows:

[0052]

[0053] The unit is dB, and the calculation formula for sound pressure level is as follows:

[0054]

[0055] Where Pref is the reference sound pressure, which is generally 2×10 -5 (Pa). Due to the different sensitivities of digital microphones, the time domain sampling point values ​​collected by the microphone need to be converted before the effective sound pressure is calculated. The specific method is as follows:

[0056]

[0057] Among them, p AOP is the maximum sound pressure level of the microphone (in dB), N is the number of time domain sampling points, and x(n) is the value of the time domain sampling point (the standardized value, ranging from -1 to 1).

[0058] The final calculated sound pressure level is judged by the threshold method. If there is a sound signal exceeding the threshold, noise event detection is performed.

[0059] The first encoder network is trained by using unlabeled noise data to train the first initial encoder network, specifically:

[0060] Acquiring unlabeled noise data, and performing acoustic feature extraction on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data;

[0061] The first encoder network is iteratively trained according to the second acoustic feature data; when the number of iterations reaches a preset value, the iterative training is completed; and the first encoder network after the iterative training is used as the encoder network of the initial noise event detection network.

[0062] As a preferred solution, acoustic feature extraction is performed on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data, specifically:

[0063] To ensure the consistency of the input audio length in subsequent models, the collected unlabeled noise audio should be divided into several t-second audio segments of equal length as input audio;

[0064] Perform a short-time Fourier transform on the input audio to obtain the amplitude spectrum of the short-time Fourier transform:

[0065]

[0066] S(n,f)=|X(n,f)| 2 (5)

[0067] Where m is the sample of the input audio; x(m) is the time domain signal of the input audio, w(nm) is the window function with an offset of n sample points, f is the frequency, and S(n,f) is the amplitude spectrum of the short-time Fourier transform.

[0068] The short-time Fourier transform amplitude spectrum is passed through the Mel filter bank to obtain the Mel short-time Fourier transform amplitude spectrum:

[0069] MelS(n,f)=mel(S(n,f)) (6)

[0070] Where mel(·) is the Mel filter bank function and MelS(n,f) is the Mel short-time Fourier transform amplitude spectrum.

[0071] Take the logarithm to obtain the logarithmic Mel short-time Fourier transform amplitude spectrum, that is, the logarithmic Mel time spectrum:

[0072] LogMelS(n,f)=20log 10 MelS(n,f) (7)

[0073] Among them, LogMelS(n,f) is the logarithmic Mel-time spectrum.

[0074] The logarithmic Mel-time spectrum is then normalized according to the following formula:

[0075] X norm =(XX mean ) / X std (8)

[0076] Among them, X norm is the signal after amplitude normalization, and X mean and X std are the mean and standard deviation of the input signal respectively.

[0077] The signal after amplitude normalization is used as the second acoustic feature data of the unlabeled noise data.

[0078] Similarly, acoustic feature extraction is performed on the ambient audio data to obtain the first acoustic feature data; and the acoustic feature extraction is performed on the training data in the training data set to generate the third acoustic feature data. The acoustic feature extraction method is the same as the above-mentioned acoustic feature extraction method for unlabeled noise data, and will not be repeated later.

[0079] Further, the first encoder network is iteratively trained according to the second acoustic feature data; when the number of iterations reaches a preset value, the iterative training is completed, specifically:

[0080] Performing a first data enhancement and a second data enhancement on the second acoustic feature data respectively, to obtain first audio transformation data and second audio transformation data respectively; wherein the enhancement amplitudes of the first data enhancement and the second data enhancement are different;

[0081] Inputting the first audio transformation data into a first initial encoder network and a first multilayer perceptron network connected in sequence to generate first feature representation data; inputting the second audio transformation data into a second initial encoder network to generate second feature representation data; the first initial encoder network and the second initial encoder network have the same structure;

[0082] Calculating the cosine similarity loss of the first feature representation data and the second feature representation data; updating the parameters of the first initial encoder network and the second initial encoder network respectively according to the back propagation of the cosine similarity loss; wherein the second initial encoder network is updated using the momentum method;

[0083] The prediction data and the feature data are recalculated according to the first audio transformation data, the second audio transformation data and the updated first initial encoder network and the second initial encoder network until the number of updates reaches a preset value, thereby completing the iterative training.

[0084] As a preferred solution, the first initial encoder network and the second initial encoder network have the same structure, and the initial encoder network includes three two-dimensional convolution modules and one multi-layer perceptron module;

[0085] The convolution kernel size of each layer of the two-dimensional convolution module is 3 and the stride is 1. The stride of the two-dimensional maximum pooling layer is 2. The multi-layer perceptron module includes a first linear layer, a first non-linear layer, a Dropout layer, a second linear layer, and a second non-linear layer stacked in sequence. For example, a 512-dimensional feature is mapped to a 2048-dimensional feature through the first linear layer, and a 2048-dimensional feature is output.

[0086] As a preferred solution, the first multi-layer perceptron network is a temporary network added in the pre-training stage for prediction. The features output by the first initial encoder network and the second initial encoder network are distinguished by the multi-layer perceptron network. The features output by the first initial encoder network and the second initial encoder network are essentially the same thing. Different feature representations are constructed by the multi-layer perceptron network to perform comparative learning. The input data is extracted into feature vectors by the first initial encoder network and the second initial encoder network. The feature vector output by the first initial encoder network passes through the multi-layer perceptron network, and an additional layer of transformation is performed on the feature vector. The two are distinguished at the data level, but they are still the same in terms of semantics.

[0087] As a preferred solution, the second initial encoder network uses the momentum method for updating, while the first initial encoder network does not. The second initial encoder network has an additional momentum parameter for gradient updating, and the gradient update strategy uses the Adam optimizer.

[0088] Furthermore, the noise event detection network is trained by using training data to train the initial noise event detection network, wherein the training data is constructed from the labeled noise data, specifically:

[0089] The obtained manually annotated strong label data set and the single-category weak label data set are randomly mixed to generate a training data set; the acoustic features of the training data in the training data set are extracted to generate third acoustic feature data; the enhanced third acoustic feature data is input into the initial noise event detection network to generate an initial prediction result;

[0090] According to the initial prediction result, the global loss of the initial noise event detection network is calculated; the global loss is back-propagated, and the parameters of the initial noise event detection network are iteratively updated until the global loss calculated using the updated initial noise event detection network converges, and the updated initial noise event detection network is used as the noise event detection network.

[0091] As a preferred solution, the global loss is back-propagated, and the parameters of the initial noise event detection network are iteratively updated until the global loss calculated using the updated initial noise event detection network converges. The training method uses the Adam optimizer, sets 100 epochs (cycles), batch_size (the number of training samples invested each time) to 64, the learning rate to 0.001, and the learning rate update strategy is to automatically reduce the learning rate every 20 epochs.

[0092] Furthermore, the acquired manually annotated strong label dataset and the single-category weak label dataset are randomly mixed to generate a training dataset, specifically:

[0093] Obtain manually annotated strong label data and several single-category first noise audios, and perform silence detection on the first noise audios; according to the detection result of the silence detection, remove the silence audios of the first noise audios to obtain several single-category second noise audios; randomly mix several second noise audios with manually annotated strong label data to generate a training data set with strong labels of event start and end times and weak labels of corresponding event names.

[0094] As a preferred solution, in order to improve the detection ability of the model, it is necessary to expand the training data set. New strongly labeled data is synthesized based on the existing small amount of strongly labeled data and some single-category noise audio, so as to achieve data expansion. Specifically, energy-based silence detection is performed on the single-category noise audio, and the silence part in the audio is removed. Then, multiple categories of noise audio are randomly mixed to generate a training data set with strong labels of event start and end times and weak labels of corresponding event names.

[0095] Furthermore, the enhanced third acoustic feature data is input into the initial noise event detection network to generate an initial prediction result, specifically:

[0096] The third acoustic feature data is input into the initial noise event detection network, and the third acoustic feature data is controlled to output the first feature after passing through the first encoder network; the first feature is controlled to output the second feature after passing through the bidirectional gated recurrent neural network; the second feature is controlled to output the first matrix after the dimension reduction is performed on the second feature through the second multi-layer perceptron network; the first matrix is ​​controlled to output the initial frame-level time prediction result through the activation function; and the first matrix is ​​input into the aggregation layer to output the initial event category prediction result.

[0097] As a preferred solution, the initial noise event detection network includes: a first encoder network, a gated recurrent neural network, a second multi-layer perceptron network and an aggregation layer connected in series in sequence; wherein the hidden layer size of the gated recurrent neural network is 2048, adopts a bidirectional structure, and outputs 4096-dimensional features.

[0098] Exemplarily, after the third acoustic feature data is input into the initial noise event detection network, it passes through the encoder network to obtain 2048-dimensional features, and then passes through the bidirectional gated recurrent neural network to output 4096-dimensional features. The multilayer perceptron module 2 reduces the dimensionality of the 4096-dimensional features and outputs a [C, L] matrix, and obtains the initial frame-level time prediction result through the Sigmoid activation function.

[0099] Furthermore, the first matrix is ​​input into the aggregation layer and then the initial event category prediction result is output, specifically:

[0100] Get the first matrix output by the multilayer perceptron module: [C, L]; where C is the number of noise categories and L is the time domain clue;

[0101] The first matrix is ​​input into the aggregation layer for aggregation, and the prediction result of the category is output as the initial event category prediction result; wherein the aggregation layer adopts mean aggregation and maximum aggregation:

[0102] Y class =(Y frame_max +Y frame_mean ) / 2;

[0103] Y class is the predicted result of the category, Y frame is the initial frame-level temporal prediction result; Y frame_max is the maximum value of the initial frame-level temporal prediction result, Y frame_mean is the mean of the initial frame-level temporal prediction results.

[0104] Further, according to the initial prediction result, the global loss of the initial noise event detection network is calculated, specifically:

[0105] Calculate the binary cross entropy loss of the initial event category prediction results and the smoothed absolute error loss of the initial frame-level time prediction results; calculate the global loss function based on the binary cross entropy loss and the smoothed absolute error loss:

[0106] L(X)=BCE(Y class ,Z class )+α*SmoothL1loss(Y frame ,Z frame );

[0107] Among them, L(X) is the global loss function; BCE(Y class ,Z class ) is the binary cross entropy loss, SmoothL1loss(Y frame ,Z frame ) is the smoothed absolute error loss, Y class is the initial event category prediction result, Z class is the event label, Y frame is the initial frame-level temporal prediction result, Z frame is the frame-level label and α is a hyperparameter.

[0108] In this embodiment, it is not necessary to use a large-scale strongly labeled noise event data set for training, and the unlabeled noise data in the real environment can be fully utilized through self-supervised training. Only a small amount of strongly labeled data sets are needed in the training of the initial noise event detection network. The first encoding network of the initial noise event detection network is trained by contrastive learning using large-scale unlabeled noise data. This training method is a pre-training method of self-supervised learning. An audio representation is obtained by contrastive learning pre-training in self-supervised learning. Based on the audio representation as a feature in noise event detection, the noise event detection network is further trained and optimized, avoiding the problem that the model training requires a lot of manpower to strongly label the collected data set.

[0109] In this embodiment, training data is constructed by expanding the strongly labeled data set. After feature extraction from the training data, a gated recurrent neural network of the initial noise event detection network is used to model the context of the time domain information. A small amount of labeled data is used to train the entire noise detection network. Different loss functions are used during training to optimize the model in terms of event category and frame-level information, making the sound event detection method more accurate.

[0110] Step S102: After screening the first acoustic feature data and correcting the abnormal points, the start and end time points are detected and the start and end time labels are generated; the start and end time labels and the event category prediction results are used as noise event detection results.

[0111] Furthermore, the first acoustic feature data is screened and outliers are corrected, specifically:

[0112] According to the confidence of the event category prediction result, the first acoustic feature data with the confidence higher than the second threshold is screened out; the frame-level time prediction result of the screened third acoustic feature data is detected for outliers; if an outlier exists, the outlier is corrected.

[0113] In this embodiment, the performance of the noise event detection network is verified and measured in two dimensions, namely, audio tag-based measurement and sound event detection-based measurement. The audio tag-based measurement is evaluated using the multi-label classification task indicator mAP (Mean Average Precision). In this embodiment, the mAP value is 79.13%; while the sound event detection-based measurement uses the F1-score indicator and PSDS (Polyphonic Sound Detection Score). In this embodiment, the F1-score indicator value is 55.46% and the PSDS value is 34.41%.

[0114] Implementing the embodiments of the present invention has the following effects:

[0115] The noise event detection method of the present invention adopts a self-supervised training method of contrastive learning, effectively utilizes the collected unlabeled noise data to pre-train the encoder module and the multi-layer perceptron module of the noise event detection network, obtains an audio representation for the noise scene, retains the time domain information, and captures the dependency with large intervals in the time series data through the gated recurrent neural network module, performs time series modeling, and improves the accuracy and efficiency of processing the unlabeled environmental audio data; in addition, the training loss function of the noise detection model is adjusted by utilizing the labeled noise data, the noise detection network is optimized, the performance of the noise detection network is improved, and the noise event detection accuracy is improved.

[0116] Embodiment 2

[0117] Please refer to Figure 2 , a noise event detection device provided by an embodiment of the present invention includes: a detection module 201 and a result generation module 202;

[0118] The detection module 201 is used to obtain environmental audio data and detect whether the sound pressure of the environmental audio data is lower than a first threshold; if so, extract the first acoustic feature data of the environmental audio data; input the first acoustic feature data into the noise event detection network to generate a frame-level time prediction result and an event category prediction result;

[0119] The noise event detection network is trained by training the initial noise event detection network with training data; the training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by training the first initial encoder network with unlabeled noise data;

[0120] The result generation module 202 is used to detect the start and end time points after screening and correcting the abnormal points of the first acoustic feature data, and generate start and end time labels; and use the start and end time labels and the event category prediction results as noise event detection results.

[0121] The detection module 201 includes: a first training submodule and a second training submodule;

[0122] The first training submodule is used to obtain unlabeled noise data, and perform acoustic feature extraction on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data;

[0123] The first encoder network is iteratively trained according to the second acoustic feature data; when the number of iterations reaches a preset value, the iterative training is completed; and the first encoder network after the iterative training is used as the encoder network of the initial noise event detection network.

[0124] The first training submodule includes a first training unit;

[0125] The first training unit is used to perform first data enhancement and second data enhancement on the second acoustic feature data, respectively, to obtain first audio transformation data and second audio transformation data, respectively; wherein the enhancement amplitudes of the first data enhancement and the second data enhancement are different;

[0126] Inputting the first audio transformation data into a first initial encoder network and a first multilayer perceptron network connected in sequence to generate first feature representation data; inputting the second audio transformation data into a second initial encoder network to generate second feature representation data; the first initial encoder network and the second initial encoder network have the same structure;

[0127] Calculating the cosine similarity loss of the first feature representation data and the second feature representation data; updating the parameters of the first initial encoder network and the second initial encoder network respectively according to the back propagation of the cosine similarity loss; wherein the second initial encoder network is updated using the momentum method;

[0128] The prediction data and the feature data are recalculated according to the first audio transformation data, the second audio transformation data and the updated first initial encoder network and the second initial encoder network until the number of updates reaches a preset value, thereby completing the iterative training.

[0129] The second training submodule is used to randomly mix the acquired manually annotated strong label data set and the single-category weak label data set to generate a training data set; extract acoustic features from the training data in the training data set to generate third acoustic feature data; input the enhanced third acoustic feature data into the initial noise event detection network to generate an initial prediction result;

[0130] According to the initial prediction result, the global loss of the initial noise event detection network is calculated; the global loss is back-propagated, and the parameters of the initial noise event detection network are iteratively updated until the global loss calculated using the updated initial noise event detection network converges, and the updated initial noise event detection network is used as the noise event detection network.

[0131] The second training submodule includes a second training unit; the second training unit includes a training data generation subunit, a prediction subunit and an update subunit;

[0132] The training data generation subunit is used to obtain manually annotated strong label data and several single-category first noise audios, and perform silence detection on the first noise audios; according to the detection result of the silence detection, the silence audio of the first noise audio is eliminated to obtain several single-category second noise audios; several second noise audios and manually annotated strong label data are randomly mixed to generate a training data set with strong labels of event start and end times and weak labels of corresponding event names.

[0133] The prediction subunit is used to input the third acoustic feature data into the initial noise event detection network, control the third acoustic feature data to output the first feature after passing through the first encoder network; control the first feature to output the second feature after passing through the bidirectional gated recurrent neural network; control the second feature to output the first matrix after reducing the dimension of the second feature through the second multi-layer perceptron network; control the first matrix to output the initial frame-level time prediction result through the activation function; and output the initial event category prediction result after inputting the first matrix into the aggregation layer.

[0134] After the first matrix is ​​input into the aggregation layer, the initial event category prediction result is output, which is specifically:

[0135] Get the first matrix output by the multilayer perceptron module: [C, L]; where C is the number of noise categories and L is the time domain clue;

[0136] The first matrix is ​​input into the aggregation layer for aggregation, and the prediction result of the category is output as the initial event category prediction result; wherein the aggregation layer adopts mean aggregation and maximum aggregation:

[0137] Y class =(Y frame_max +Y frame_mean ) / 2;

[0138] Y class is the predicted result of the category, Y frame is the initial frame-level temporal prediction result; Y frame_max is the maximum value of the initial frame-level temporal prediction result, Y frame_mean is the mean of the initial frame-level temporal prediction results.

[0139] The update subunit is used to calculate the binary cross entropy loss of the initial event category prediction result and the smoothed absolute error loss of the initial frame-level time prediction result; based on the binary cross entropy loss and the smoothed absolute error loss, the global loss function is calculated:

[0140] L(X)=BCE(Y class ,Z class )+α*SmoothL1loss(Y frame ,Zframe );

[0141] Among them, L(X) is the global loss function; BCE(Y class ,Z class ) is the binary cross entropy loss, SmoothL1loss(Y frame ,Z frame ) is the smoothed absolute error loss, Y class is the initial event category prediction result, Z class is the event label, Y frame is the initial frame-level temporal prediction result, Z frame is the frame-level label and α is a hyperparameter.

[0142] The result generation module includes: a result preprocessing module, which is used to filter out the first acoustic feature data with a confidence level higher than a second threshold value according to the confidence level of the event category prediction result; perform outlier detection on the frame-level time prediction result of the filtered third acoustic feature data; and correct the outlier if any.

[0143] The above-mentioned noise event detection device can implement the noise event detection method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of the present application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0144] Implementing the embodiments of the present invention has the following effects:

[0145] The detection module of the noise event detection device of the present invention adopts a self-supervised training method of contrastive learning, and effectively uses the collected unlabeled environmental audio data to pre-train through the encoder module and the multi-layer perceptron module of the noise event detection network, so as to obtain an audio representation for the noise scene, retain the time domain information, and capture the dependency with large intervals in the time series data through the gated recurrent neural network module, and perform time series modeling, thereby improving the accuracy and efficiency of processing the unlabeled environmental audio data; in addition, the training module uses the labeled noise data to adjust the training loss function of the noise detection model, optimizes the noise detection network, improves the performance of the noise detection network, and improves the accuracy of noise event detection.

[0146] Embodiment 3

[0147] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the noise event detection method described in any one of the above embodiments.

[0148] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0149] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0150] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0151] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0152] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0153] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A noise event detection method, characterized in that: include: Acquire environmental audio data, and detect whether the sound pressure of the environmental audio data is lower than a first threshold; If yes, extracting first acoustic feature data of the ambient audio data; Inputting the first acoustic feature data into a noise event detection network to generate a frame-level time prediction result and an event category prediction result; Wherein, the noise event detection network is trained by using training data to train an initial noise event detection network; The training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by the first initial encoder network using unlabeled noise data, specifically: Acquiring unlabeled noise data, and performing acoustic feature extraction on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data; Iteratively training the first encoder network according to the second acoustic feature data; completing the iterative training when the number of iterations reaches a preset value; using the iteratively trained first encoder network as the encoder network of the initial noise event detection network; The iterative training of the first encoder network according to the second acoustic feature data is performed; when the number of iterations reaches a preset value, the iterative training is completed, specifically: Performing a first data enhancement and a second data enhancement on the second acoustic feature data respectively, to obtain first audio transformation data and second audio transformation data respectively; wherein the enhancement amplitudes of the first data enhancement and the second data enhancement are different; Inputting the first audio transformation data into a first initial encoder network and a first multi-layer perceptron network connected in sequence to generate first feature representation data; inputting the second audio transformation data into a second initial encoder network to generate second feature representation data; the first initial encoder network and the second initial encoder network have the same structure; Calculating the cosine similarity loss of the first feature representation data and the second feature representation data; back-propagating the cosine similarity loss, respectively updating the parameters of the first initial encoder network and the second initial encoder network; wherein the second initial encoder network is updated using a momentum method; Recalculating the prediction data and the feature data according to the first audio transformation data, the second audio transformation data, and the updated first initial encoder network and the second initial encoder network until the number of updates reaches a preset value, thereby completing iterative training; After the first acoustic feature data is screened and outliers are corrected, the start and end time points are detected and start and end time labels are generated; the start and end time labels and the event category prediction results are used as noise event detection results.

2. A noise event detection method as claimed in claim 1, characterized in that: The noise event detection network is trained by using training data to train the initial noise event detection network. The training data is constructed from label noise data, specifically: The acquired manually annotated strong label data set and the single-category weak label data set are randomly mixed to generate a training data set; acoustic feature extraction is performed on the training data in the training data set to generate third acoustic feature data; Inputting the enhanced third acoustic feature data into the initial noise event detection network to generate an initial prediction result; According to the initial prediction result, the global loss of the initial noise event detection network is calculated; the global loss is back-propagated, and the parameters of the initial noise event detection network are iteratively updated until the global loss calculated using the updated initial noise event detection network converges, and the updated initial noise event detection network is used as the noise event detection network.

3. A noise event detection method as claimed in claim 2, characterized in that: The obtained manually annotated strong label data set and the single-category weak label data set are randomly mixed to generate a training data set, specifically: Obtain manually annotated strong label data and several single-category first noise audios, and perform silence detection on the first noise audios; according to the detection result of the silence detection, remove the silence audios of the first noise audios to obtain several single-category second noise audios; randomly mix several second noise audios with manually annotated strong label data to generate a training data set with strong labels of event start and end times and weak labels of corresponding event names.

4. A noise event detection method as claimed in claim 2, characterized in that: The enhanced third acoustic feature data is input into the initial noise event detection network to generate an initial prediction result, specifically: The third acoustic feature data is input into the initial noise event detection network, and the third acoustic feature data is controlled to output the first feature after passing through the first encoder network; the first feature is controlled to output the second feature after passing through the bidirectional gated recurrent neural network; the second feature is controlled to output the first matrix after the dimension reduction is performed on the second feature through the second multi-layer perceptron network; the first matrix is ​​controlled to output the initial frame-level time prediction result through the activation function; and the first matrix is ​​input into the aggregation layer to output the initial event category prediction result.

5. A noise event detection method as claimed in claim 4, characterized in that: After inputting the first matrix into the aggregation layer, outputting the initial event category prediction result is specifically: Get the first matrix output by the multilayer perceptron module: [C, L]; where C is the number of noise categories and L is the time domain clue; The first matrix is ​​input into the aggregation layer for aggregation, and the prediction result of the category is output as the initial event category prediction result; wherein the aggregation layer adopts mean aggregation and maximum aggregation: ； Y class is the predicted result of the category, Y frame_max is the maximum value of the initial frame-level temporal prediction result, Y frame_mean is the mean of the initial frame-level temporal prediction results.

6. A noise event detection method as claimed in claim 4, characterized in that: The global loss of the initial noise event detection network is calculated according to the initial prediction result, specifically: Calculate the binary cross entropy loss of the initial event category prediction results and the smoothed absolute error loss of the initial frame-level time prediction results; calculate the global loss function based on the binary cross entropy loss and the smoothed absolute error loss: ； Among them, L(X) is the global loss function; is the binary cross entropy loss, is the smoothed absolute error loss, is the initial event category prediction result, is the event label, is the initial frame-level temporal prediction result, is the frame-level label, is a hyperparameter.

7. A noise event detection method as claimed in claim 2, characterized in that: The screening and abnormal point correction of the first acoustic feature data is specifically as follows: According to the confidence of the event category prediction result, the first acoustic feature data with the confidence higher than the second threshold is screened out; and anomaly point detection is performed on the frame-level time prediction result of the screened third acoustic feature data; If there are abnormal points, the abnormal points are corrected.

8. A noise event detection device, characterized in that: include: Detection module and result generation module; The detection module is used to obtain environmental audio data and detect whether the sound pressure of the environmental audio data is lower than a first threshold; if so, extract first acoustic feature data of the environmental audio data; input the first acoustic feature data into a noise event detection network to generate a frame-level time prediction result and an event category prediction result; The noise event detection network is trained by training the initial noise event detection network with training data; the training data is constructed from labeled noise data; the initial noise event detection network includes a first encoder network; the first encoder network is trained by training the first initial encoder network with unlabeled noise data, specifically: Acquiring unlabeled noise data, and performing acoustic feature extraction on the unlabeled noise data to generate second acoustic feature data of the unlabeled noise data; Iteratively training the first encoder network according to the second acoustic feature data; completing the iterative training when the number of iterations reaches a preset value; using the iteratively trained first encoder network as the encoder network of the initial noise event detection network; The iterative training of the first encoder network according to the second acoustic feature data is performed; when the number of iterations reaches a preset value, the iterative training is completed, specifically: Performing a first data enhancement and a second data enhancement on the second acoustic feature data respectively, to obtain first audio transformation data and second audio transformation data respectively; wherein the enhancement amplitudes of the first data enhancement and the second data enhancement are different; Inputting the first audio transformation data into a first initial encoder network and a first multi-layer perceptron network connected in sequence to generate first feature representation data; inputting the second audio transformation data into a second initial encoder network to generate second feature representation data; the first initial encoder network and the second initial encoder network have the same structure; Calculating the cosine similarity loss of the first feature representation data and the second feature representation data; back-propagating the cosine similarity loss, respectively updating the parameters of the first initial encoder network and the second initial encoder network; wherein the second initial encoder network is updated using a momentum method; Recalculating the prediction data and the feature data according to the first audio transformation data, the second audio transformation data, and the updated first initial encoder network and the second initial encoder network until the number of updates reaches a preset value, thereby completing iterative training; The result generation module is used to screen the first acoustic feature data and correct abnormal points, then detect the start and end time points and generate start and end time labels; and use the start and end time labels and the event category prediction results as noise event detection results.

Images