A detection method, device and storage medium based on noise event detection model

Abstract

The present invention discloses a detection method, device and storage medium based on a noise event detection model. The method comprises: inputting noise monitoring data into the noise event detection model to generate a prediction result; wherein the noise event detection model is obtained by performing joint strong and weak supervision training on an initial model; after the initial model is pre-trained on a pre-trained neural network by training data, preset parameters in the pre-trained neural network are migrated; the prediction result comprises: a first label probability and a second label probability; if the second label probability of the prediction result is higher than a threshold, the noise monitoring data is saved in a noise database, and the noise event detection model is re-trained; if not, the prediction result is output, and the noise event detection model is obtained by transfer learning using deep learning pre-trained neural network parameters, and the noise event detection model is continuously trained and parameter updates are performed on the noise event detection model through field data, so as to improve the accuracy and efficiency of noise event detection.

Description

Technical Field

[0001] The present invention relates to the field of noise monitoring technology, and in particular to a detection method, device and storage medium based on a noise event detection model. Background Art

[0002] In real life, the amount of data obtained by monitoring various noises is huge. However, due to the lack of effective noise intelligent identification technology, it is impossible to effectively process big data. In the data preprocessing and model training process of noise event detection in the existing technology, the data processing is too simple, the audio data scene is single, and no calibration training is performed using field data, so that the noise event detection model cannot adapt to various complex situations, resulting in poor performance and low efficiency of the noise event detection model in actual scenarios. Summary of the invention

[0003] The present invention provides a detection method, device and storage medium based on a noise event detection model, which uses deep learning to pre-train neural network parameters for transfer learning to obtain a noise event detection model, and continuously trains the noise event detection model and updates its parameters through field data to improve the accuracy and efficiency of noise event detection.

[0004] The present invention provides a detection method based on a noise event detection model, comprising: inputting acquired noise monitoring data into the noise event detection model to generate a model prediction result; wherein the noise event detection model is obtained by performing joint strong and weak supervision training on a first initial noise event detection model; the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data, and then migrating preset parameters in the pre-trained neural network to a second initial noise event detection model; the model prediction result comprises: a first label probability and a second label probability of the noise monitoring data;

[0005] According to the model prediction result, determine whether the second label probability is higher than the first threshold; if so, save the noise monitoring data to the noise database for data update, and perform joint strong and weak supervision training on the noise event detection model according to the noise database after data update; if not, output the model prediction result.

[0006] As a preferred solution, the present invention uses deep learning to transfer learning of pre-trained neural network parameters to obtain a noise event detection model, and continuously conducts joint strong and weak supervision training and updates to the noise event detection model through field data to improve the accuracy and efficiency of noise event detection. In addition, the joint strong and weak supervision training is to train the model according to the first label and the second label of the input data; wherein, the first initial noise event detection model is subjected to joint strong and weak supervision training, which is to conduct joint strong and weak supervision training on the first initial noise event detection model according to the first label and the second label of the input training data; the noise event detection model is subjected to joint strong and weak supervision training according to the noise database after the data is updated, which is to conduct joint strong and weak supervision training on the noise event detection model according to the first label and the second label of the training data in the noise database after the data is updated; the model has better robustness and detection performance through joint strong and weak supervision training, and the performance of data preprocessing and model training of noise event detection is improved.

[0007] Furthermore, the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data, and then migrating preset parameters in the pre-trained neural network to the second initial noise event detection model, specifically:

[0008] Acquire training data to pre-train a pre-trained neural network, and update the parameters of the pre-trained neural network; acquire feature coding layer parameters of the updated pre-trained neural network, migrate the feature coding layer parameters to a second initial noise event detection model, assign values ​​to the feature coding layer parameters of the second initial noise event detection model, and obtain a first initial noise event detection model.

[0009] Furthermore, after assigning values ​​to the feature encoding layer parameters of the second initial noise event detection model, it also includes: adding an LSTM layer to the downstream task module of the second initial noise event detection model.

[0010] As a preferred solution, the present invention uses large-scale deep learning pre-trained neural network parameters for transfer learning, obtains feature encoding layer parameters to accelerate the training of the transfer learning model, and uses a long short-term memory (LSTM) network to transform downstream tasks so that the model obtains better performance, thereby achieving the most effective noise event detection.

[0011] Furthermore, the noise event detection model is obtained by training the first initial noise event detection model with joint strong and weak supervision, specifically:

[0012] Extracting features from the training data to obtain input feature data; inputting the input feature data into a first initial noise event detection model to output an initial recognition result; the initial recognition result includes: an initial first label probability and an initial second label probability;

[0013] According to the true probability of the first label and the true probability of the second label, the overall loss of the initial recognition result is calculated; and according to the overall loss, the parameters of the first initial noise event detection model are gradient updated to obtain the noise event detection model.

[0014] Furthermore, the noise event detection model is obtained by training the first initial noise event detection model with joint strong and weak supervision, and further includes:

[0015] The first initial noise event detection model is trained with joint strong and weak supervision using different training data to obtain several noise event detection models; the noise event detection model with the highest average precision of all class labels is selected as the final noise event detection model.

[0016] Furthermore, feature extraction is performed on the training data to obtain input feature data, specifically:

[0017] Acquire a time domain signal of training data; perform short-time Fourier transform on the time domain signal to obtain an amplitude spectrum of the short-time Fourier transform; compress the amplitude spectrum of the short-time Fourier transform using a Mel filter group to obtain a Mel short-time Fourier transform amplitude spectrum; take the logarithm of the obtained Mel short-time Fourier transform amplitude spectrum to obtain a logarithmic Mel time-frequency spectrum feature as input feature data.

[0018] Furthermore, according to the true probability of the first label and the true probability of the second label, the overall loss of the initial recognition result is calculated, specifically:

[0019] The binary cross entropy loss is calculated for the true probability of the first label and the initial first label probability to obtain the first label loss; the binary cross entropy loss is calculated for the true probability of the second label and the initial second label probability to obtain the second label loss; the first label loss is added to the second label loss to obtain the overall loss of the initial recognition result.

[0020] As a preferred solution, the present invention performs feature extraction on the training data, and obtains the probabilities of the first label and the second label of the training data according to the first initial noise event detection model; and calculates the overall loss of the recognition result according to the probabilities of the true first label and the second label, and performs gradient update on the parameters of the first initial noise event detection model according to the overall loss, so as to complete the joint strong and weak supervision training of the first initial noise event detection model, so that the model has better robustness and detection performance, and improves the performance of data preprocessing and model training of noise event detection.

[0021] Furthermore, the noise event detection model is trained with joint strong and weak supervision according to the noise database after data update, specifically:

[0022] Acquire noise data of the noise database after data update, and normalize the noise data; perform data enhancement on the normalized noise data to obtain enhanced audio data;

[0023] Acquire the audio event of the enhanced audio data, construct a first tag for the enhanced audio data corresponding to the audio event with the start time and the end time; construct a second tag for the enhanced audio data corresponding to the audio event without the start time and the end time and with the segment-level tag;

[0024] The enhanced audio data with the first label or the second label is input into the noise event detection model, the recognition result is output, and the overall loss of the recognition result is calculated; the parameters of the noise event detection model are gradient updated according to the overall loss, and the noise event detection model is updated.

[0025] As a preferred solution, the present invention stores the screened field noise data into a database, mixes the noise data in the database and constructs them into first label and second label data sets, and improves the robustness and detection performance of the model through joint strong and weak supervision training.

[0026] Accordingly, the present invention also provides a detection device based on a noise event detection model, comprising: a detection module and an update module;

[0027] The detection module is used to input the acquired noise monitoring data into the noise event detection model to generate a model prediction result; wherein, the noise event detection model is obtained by performing joint strong and weak supervision training on a first initial noise event detection model; the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and then migrating the preset parameters in the pre-trained neural network to a second initial noise event detection model; the joint strong and weak supervision training is to train the initial noise event detection model according to the first label and the second label of the training data; the model prediction result includes: the first label probability and the second label probability of the noise monitoring data;

[0028] The updating module is used to determine whether the second label probability is higher than a first threshold value according to the prediction result of the model; if so, the noise monitoring data is saved to the noise database for data updating, and the noise event detection model is jointly trained with strong and weak supervision according to the noise database after the data update.

[0029] As a preferred solution, the present invention uses deep learning to transfer learning of pre-trained neural network parameters to obtain a noise event detection model, and continuously conducts joint strong and weak supervision training and updates to the noise event detection model through field data to improve the accuracy and efficiency of noise event detection. In addition, the joint strong and weak supervision training is to train the model according to the first label and the second label of the input data; wherein, the first initial noise event detection model is subjected to joint strong and weak supervision training, which is to conduct joint strong and weak supervision training on the first initial noise event detection model according to the first label and the second label of the input training data; the noise event detection model is subjected to joint strong and weak supervision training according to the noise database after the data is updated, which is to conduct joint strong and weak supervision training on the noise event detection model according to the first label and the second label of the training data in the noise database after the data is updated; the model has better robustness and detection performance through joint strong and weak supervision training, and the performance of data preprocessing and model training of noise event detection is improved.

[0030] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute a detection method based on a noise event detection model as described in the content of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flow chart of an embodiment of a detection method based on a noise event detection model provided by the present invention;

[0032] Figure 2 It is a structural schematic diagram of an embodiment of a detection device based on a noise event detection model provided by the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Embodiment 1

[0035] Please refer to Figure 1 , a detection method based on a noise event detection model provided by an embodiment of the present invention, comprising steps S101-S103:

[0036] Step S101: input the acquired noise monitoring data into the noise event detection model to generate a model prediction result; wherein the noise event detection model is obtained by performing joint strong and weak supervision training on a first initial noise event detection model; the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and then migrating the preset parameters in the pre-trained neural network to a second initial noise event detection model; the model prediction result includes: a first label probability and a second label probability of the noise monitoring data;

[0037] In this embodiment, the noise event detection model identifies the probability of the first label and the second label of the input data. The first label is a strong label, which means that in a segment of audio, each event has a clear start time and end time label. The second label is a weak label, which means that in a segment of audio, the audio event has no start time and end time label, only a segment-level label.

[0038] Furthermore, the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data, and then migrating preset parameters in the pre-trained neural network to the second initial noise event detection model, specifically:

[0039] Acquire training data to pre-train a pre-trained neural network, and update the parameters of the pre-trained neural network; acquire feature coding layer parameters of the updated pre-trained neural network, migrate the feature coding layer parameters to a second initial noise event detection model, assign values ​​to the feature coding layer parameters of the second initial noise event detection model, and obtain a first initial noise event detection model.

[0040] In this embodiment, the training data uses a public general audio dataset, which has 632 categories of data and 2,084,320 audio data. Compared with the transfer learning of other datasets, the transfer learning of the pre-trained model of the public general audio dataset will undoubtedly be easier to transfer to other tasks, allowing the model to adapt to new scenario tasks more quickly.

[0041] Furthermore, after assigning values ​​to the feature encoding layer parameters of the second initial noise event detection model, it also includes: adding an LSTM layer to the downstream task module of the second initial noise event detection model.

[0042] In this embodiment, the pre-trained neural network includes a first feature coding layer and a first downstream task; the first feature coding layer includes a convolutional layer and a pooling layer; the first downstream task includes a fully connected layer; the structure of the second feature coding layer of the first initial noise event detection model is the same as the first feature coding layer; after pre-training the pre-trained neural network according to the training data, the first feature coding layer parameters in the pre-trained neural network are migrated to the second feature coding layer of the second initial noise event detection model; the LSTM layer is used in the second downstream task of the first initial noise event detection model, and the LSTM layer can obtain contextual relationships to improve model performance.

[0043] The present invention utilizes the feature encoding layer parameters of the pre-trained neural network to migrate to the second initial noise event detection model, so that different categories of features form a good cluster distribution in the high-dimensional space, and then use a large number of noise data sets to perform migration learning in different scenarios, which not only accelerates the training speed of the model, but also allows the model to obtain better performance than retraining. In addition, the existing technologies all add residual networks or use fully connected layers to perform downstream task learning on feature vectors. The present invention uses a more effective LSTM layer network to perform downstream task learning on feature vectors. Compared with residual networks or fully connected layers, downstream tasks use LSTM layers to compress feature vectors, so that useful contextual relationships can be obtained to improve the sound event detection performance of the model.

[0044] Furthermore, the noise event detection model is obtained by training the first initial noise event detection model with joint strong and weak supervision, specifically:

[0045] Extracting features from the training data to obtain input feature data; inputting the input feature data into a first initial noise event detection model to output an initial recognition result; the initial recognition result includes: an initial first label probability and an initial second label probability;

[0046] According to the true probability of the first label and the true probability of the second label, the overall loss of the initial recognition result is calculated; and according to the overall loss, the parameters of the first initial noise event detection model are gradient updated to obtain the noise event detection model.

[0047] Furthermore, the noise event detection model is obtained by training the first initial noise event detection model with joint strong and weak supervision, and further includes:

[0048] The first initial noise event detection model is trained with joint strong and weak supervision using different training data to obtain several noise event detection models; the noise event detection model with the highest average precision of all class labels is selected as the final noise event detection model.

[0049] In this embodiment, the performance test of the noise event detection model is mainly based on the multi-label classification task indicators, AP (Average precision) single-class label average (the average of the maximum precision in each recall rate), mAP (Mean Average Precision) the average precision of all class labels. Finally, the noise event detection model with the best mAP performance is selected as the final noise event detection model to detect the acquired noise monitoring data. The noise monitoring data is tested for noise events through this model and the model prediction results are output.

[0050] Exemplarily, after using three different types of sound data sets, namely, noise data sets, bird sound data sets, and environmental sound data sets, as test sets to perform joint strong and weak supervision training on the first initial noise event detection model, the mAPs reached 0.97, 0.99, and 0.98, respectively. The first initial noise event detection model is obtained by pre-training the pre-trained neural network with training data, and then migrating the preset parameters in the pre-trained neural network to the second initial noise event detection model, that is, the first initial noise event detection model is a model after transfer learning, and the convergence speed is extremely fast. In addition, if the three different types of sound data sets, namely, noise data sets, bird sound data sets, and environmental sound data sets, are used as test sets to train the retrained noise event detection model, the mAPs reached 0.95, 0.98, and 0.21, respectively. Therefore, the noise detection model after transfer learning can better adapt to downstream tasks in different scenarios, and can obtain better mAP performance compared to retraining.

[0051] Finally, a noise event detection model with the highest mAP is selected as the final noise event detection model.

[0052] Furthermore, feature extraction is performed on the training data to obtain input feature data, specifically:

[0053] Acquire a time domain signal of training data; perform short-time Fourier transform on the time domain signal to obtain an amplitude spectrum of the short-time Fourier transform; compress the amplitude spectrum of the short-time Fourier transform using a Mel filter group to obtain a Mel short-time Fourier transform amplitude spectrum; take the logarithm of the obtained Mel short-time Fourier transform amplitude spectrum to obtain a logarithmic Mel time-frequency spectrum feature as input feature data.

[0054] In this embodiment, the input time domain signal is subjected to short-time Fourier transform, and an amplitude spectrum of the short-time Fourier transform is obtained;

[0055]

[0056] S(n,f)=|X(n,f)| 2 ; (2)

[0057] Where x(m) is the input time domain signal, w(nm) is the window function with an offset of n sample points, f is the frequency, and S(n,f) is the amplitude spectrum of the short-time Fourier transform.

[0058] A Mel filter bank is designed, and the short-time Fourier transform amplitude spectrum is passed through the Mel filter bank to obtain the Mel short-time Fourier transform amplitude spectrum.

[0059] MelS(n,f)=mel(S(n,f)); (3)

[0060] Where mel(·) is the Mel filter bank function and MelS(n,f) is the Mel short-time Fourier transform amplitude spectrum.

[0061] Finally, take the logarithm to obtain the logarithmic Mel short-time Fourier transform amplitude spectrum, that is, the logarithmic Mel time-frequency spectrum as the input feature data:

[0062] LogMelS(n,f)=20log 10 MelS(n,f); (4)

[0063] Among them, LogMelS(n,f) is the logarithmic Mel-time spectrum.

[0064] In this embodiment, the input feature data, i.e., the logarithmic Mel-time spectrum, is input into the noise detection model to start the model training. The Adam optimizer is used here, 100 epochs (cycles), batch_size (the number of training samples input each time) is set to 128, the learning rate is 0.0001, and the learning rate update strategy is to automatically reduce the learning rate when it is found that the loss is no longer reduced.

[0065] Furthermore, according to the true probability of the first label and the true probability of the second label, the overall loss of the initial recognition result is calculated, specifically:

[0066] The binary cross entropy loss is calculated for the true probability of the first label and the initial first label probability to obtain the first label loss; the binary cross entropy loss is calculated for the true probability of the second label and the initial second label probability to obtain the second label loss; the first label loss is added to the second label loss to obtain the overall loss of the initial recognition result.

[0067] In this example, the first label loss is calculated as:

[0068]

[0069] Among them, batch_size is the number of training samples put in each time, frame_size is the number of frames for each sample, and the length of each frame here is 32 milliseconds. zjiis the true probability of the i-th first label of the j-th frame of the z-th training sample. Similarly, The probability of the first label output by the model.

[0070] Second label loss calculation:

[0071]

[0072] Among them, batch_size is the number of training samples put in each time, and the length of each frame is 32 milliseconds. ji is the true probability of the second label of the i-th label of the j-th training sample. Similarly, y ji The second label probability output by the model.

[0073] Overall loss total calculate:

[0074] Loss total =Loss strong +Loss weak ; (7)

[0075] Among them, Loss strong is the overall loss of the first label, Loss weak is the overall loss of the second label.

[0076] In this embodiment,

[0077] Step S102: According to the model prediction result, determine whether the second label probability is higher than the first threshold; if so, save the noise monitoring data to the noise database for data update, and perform joint strong and weak supervision training on the noise event detection model according to the noise database after the data update; if not, output the model prediction result.

[0078] Furthermore, the noise event detection model is trained with joint strong and weak supervision according to the noise database after data update, specifically:

[0079] Acquire noise data of the noise database after data update, and normalize the noise data; perform data enhancement on the normalized noise data to obtain enhanced audio data;

[0080] Acquire the audio event of the enhanced audio data, construct a first tag for the enhanced audio data corresponding to the audio event with the start time and the end time; construct a second tag for the enhanced audio data corresponding to the audio event without the start time and the end time and with the segment-level tag;

[0081] The enhanced audio data with the first label or the second label is input into the noise event detection model, the recognition result is output, and the overall loss of the recognition result is calculated; the parameters of the noise event detection model are gradient updated according to the overall loss, and the noise event detection model is updated.

[0082] In this embodiment, the noise data is normalized to eliminate the signal amplitude difference caused by different data sources in the noise database and to further improve the accuracy of the final model prediction and recognition results.

[0083] This embodiment provides two optional data normalization methods:

[0084] Amplitude normalization method:

[0085]

[0086] Among them, x norm is the signal after amplitude normalization, and x input ,x max are the collected sound signal and its maximum amplitude respectively.

[0087] Data normalization method to compress the amplitude to within [a,b]:

[0088]

[0089] Among them, x norm is the signal after amplitude compression, and x input ,x min ,x max are the collected signal, the minimum value and the maximum value of the signal respectively.

[0090] In this embodiment, in order to make the subsequent training model more robust, it is necessary to perform data enhancement on the normalized sound signal in combination with the actual soundscape. The main technical methods include gain, time shift, multi-audio mixing, etc.

[0091] Gain Processing:

[0092] x gain (n) = x norm (n)*v; (10)

[0093] Among them, x gain is the audio signal after gain, x norm is the standardized audio signal, v is the volume coefficient, x gain (n) is the nth point of the audio signal after gain.

[0094] Time-Shift Processing:

[0095] x shift(n) = x norm (n+1); (11)

[0096] Among them, x shift is the time-shifted audio signal, x norm is the normalized audio signal, x shift (n) is the nth point of the audio signal after time shift, and l is the lth point of time shift.

[0097] Multi-audio mixing processing:

[0098] Considering that the sounds in actual scenarios do not exceed three categories in most cases, this embodiment only performs double-mixed and triple-mixed audio processing, and each category will mix the audio of the other categories.

[0099] Dual Hybrid Processing:

[0100] x 2mix (n) = x class1 (n)+x class2 (n); (12)

[0101] Triple Mix Treatment:

[0102] x 3mix (n) = x class1 (n)+x class2 (n)+x class3 (n); (13)

[0103] Among them, x 2mix is the audio signal after double mixing, x 3mix is the audio signal after three mixing, x class1 、x class2 、x class3 For different types of audio signals.

[0104] In this embodiment, constructing the first label and the second label to train the model simultaneously can further improve the noise detection performance. The specific method of constructing the first label and the second label, that is, the weak label and the strong label, is as follows:

[0105] Weak label construction:

[0106] y weak (i)=[0,0,…,1,…,0] T ,i∈[1,class_num]; (14)

[0107] Among them, y weak is a one-hot vector, class_num is the number of categories, i is the label index, and y weak (i) = 1 means that there is an i-th weakly labeled sound event in the entire audio segment, y weakThe position index i where (i)=0 indicates that there is no i-th weakly labeled sound event in the entire audio segment.

[0108] Strong tag construction:

[0109]

[0110] Among them, class_num is the number of categories, frame_size is the number of frames, j is the frame index, i is the label index, y is the strong (j,i) = 1 means that the jth frame of the entire audio segment has the i-th strongly labeled sound event, y strong (j,i) = 0 means that there is no i-th strong label sound event in the j-th frame of the entire audio segment. To construct a strong label, the j-th frame signal of the audio signal is randomly reset to 0, and the strong label of the label sound event corresponding to the j-th frame of the audio signal is also set to 0.

[0111] Implementing the embodiments of the present invention has the following effects:

[0112] The present invention uses deep learning to transfer learning of pre-trained neural network parameters to obtain a noise event detection model, and continuously conducts joint strong and weak supervision training and updates to the noise event detection model through field data, thereby improving the accuracy and efficiency of noise event detection. In addition, the joint strong and weak supervision training is to train the model according to the first label and the second label of the input data; wherein, the first initial noise event detection model is subjected to joint strong and weak supervision training, and the first initial noise event detection model is subjected to joint strong and weak supervision training according to the first label and the second label of the input training data; the noise event detection model is subjected to joint strong and weak supervision training according to the noise database after the data is updated, and the noise event detection model is subjected to joint strong and weak supervision training according to the first label and the second label of the training data in the noise database after the data is updated; the model has better robustness and detection performance through joint strong and weak supervision training, and the performance of data preprocessing and model training of noise event detection is improved.

[0113] Embodiment 2

[0114] Please refer to Figure 2 , a detection device based on a noise event detection model provided by an embodiment of the present invention, comprising: a detection module 201 and an update module 202;

[0115] The detection module 201 is used to input the acquired noise monitoring data into the noise event detection model to generate a model prediction result; wherein, the noise event detection model is obtained by performing joint strong and weak supervision training on the first initial noise event detection model; the first initial noise event detection model is obtained by pre-training the pre-trained neural network with training data and migrating the preset parameters in the pre-trained neural network to the second initial noise event detection model; the joint strong and weak supervision training is to train the initial noise event detection model according to the first label and the second label of the training data; the model prediction result includes: the first label probability and the second label probability of the noise monitoring data;

[0116] The updating module 202 is used to determine whether the second label probability is higher than the first threshold according to the model prediction result; if so, the noise monitoring data is saved to the noise database for data update, and the noise event detection model is jointly trained with strong and weak supervision according to the noise database after the data update.

[0117] The detection module 201 includes: a migration unit, an adding unit, a first training unit and a screening unit;

[0118] The migration unit is used to obtain training data to pre-train a pre-trained neural network and update the parameters of the pre-trained neural network; obtain the feature coding layer parameters of the updated pre-trained neural network, migrate the feature coding layer parameters to a second initial noise event detection model, assign values ​​to the feature coding layer parameters of the second initial noise event detection model, and obtain the first initial noise event detection model.

[0119] The adding unit is used to add an LSTM layer to the downstream task module of the second initial noise event detection model.

[0120] The first training unit includes: a feature extraction subunit, a recognition subunit and a loss subunit;

[0121] The feature extraction subunit is used to obtain the time domain signal of the training data; perform short-time Fourier transform on the time domain signal to obtain the amplitude spectrum of the short-time Fourier transform; compress the amplitude spectrum of the short-time Fourier transform using a Mel filter group to obtain the Mel short-time Fourier transform amplitude spectrum; take the logarithm of the obtained Mel short-time Fourier transform amplitude spectrum to obtain the logarithmic Mel time-frequency spectrum feature as the input feature data.

[0122] The recognition subunit is used to input the input feature data into the first initial noise event detection model to output an initial recognition result; the initial recognition result includes: an initial first label probability and an initial second label probability;

[0123] The loss subunit is used to perform binary cross entropy loss calculation on the true probability of the first label and the initial first label probability to obtain the first label loss; perform binary cross entropy loss calculation on the true probability of the second label and the initial second label probability to obtain the second label loss; add the first label loss to the second label loss to obtain the overall loss of the initial recognition result.

[0124] The screening unit is used to perform joint strong and weak supervision training on the first initial noise event detection model using different training data to obtain several noise event detection models; and select the noise event detection model with the highest average precision of all class labels as the final noise event detection model.

[0125] The updating module 202 includes: a judgment unit and a second training unit;

[0126] The judging unit is used to judge whether the second label probability is higher than a first threshold according to the prediction result of the model;

[0127] The second training unit is used to obtain the noise data of the noise database after the data is updated, normalize the noise data; perform data enhancement on the normalized noise data to obtain enhanced audio data;

[0128] Acquire the audio event of the enhanced audio data, construct a first tag for the enhanced audio data corresponding to the audio event with the start time and the end time; construct a second tag for the enhanced audio data corresponding to the audio event without the start time and the end time and with the segment-level tag;

[0129] The enhanced audio data with the first label or the second label is input into the noise event detection model, the recognition result is output, and the overall loss of the recognition result is calculated; the parameters of the noise event detection model are gradient updated according to the overall loss, and the noise event detection model is updated.

[0130] The above-mentioned detection device based on the noise event detection model can implement the detection method based on the noise event detection model of the above-mentioned method embodiment. The optional items in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of the present application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0131] Implementing the embodiments of the present invention has the following effects:

[0132] The present invention uses deep learning to transfer learning of pre-trained neural network parameters to obtain a noise event detection model, and continuously conducts joint strong and weak supervision training and updates to the noise event detection model through field data, thereby improving the accuracy and efficiency of noise event detection. In addition, the joint strong and weak supervision training is to train the model according to the first label and the second label of the input data; wherein, the first initial noise event detection model is subjected to joint strong and weak supervision training, and the first initial noise event detection model is subjected to joint strong and weak supervision training according to the first label and the second label of the input training data; the noise event detection model is subjected to joint strong and weak supervision training according to the noise database after the data is updated, and the noise event detection model is subjected to joint strong and weak supervision training according to the first label and the second label of the training data in the noise database after the data is updated; the model has better robustness and detection performance through joint strong and weak supervision training, and the performance of data preprocessing and model training of noise event detection is improved.

[0133] Embodiment 3

[0134] Correspondingly, the present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the detection method based on the noise event detection model as described in any one of the above embodiments.

[0135] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal device.

[0136] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0137] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0138] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the mobile terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0139] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0140] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A detection method based on a noise event detection model, characterized in that: include: The acquired noise monitoring data is input into the noise event detection model to generate a model prediction result; wherein the noise event detection model is obtained by performing joint strong and weak supervision training on a first initial noise event detection model; the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and then migrating the preset parameters in the pre-trained neural network to a second initial noise event detection model; the training data is a public and general audio data set; the model prediction result includes: a first label probability and a second label probability of the noise monitoring data; The first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and migrating preset parameters in the pre-trained neural network to the second initial noise event detection model, specifically: Acquire training data to pre-train a pre-trained neural network, and update parameters of the pre-trained neural network; acquire feature coding layer parameters of the updated pre-trained neural network, migrate the feature coding layer parameters to a second initial noise event detection model, assign values ​​to the feature coding layer parameters of the second initial noise event detection model, and obtain a first initial noise event detection model; According to the model prediction result, determine whether the second label probability is higher than the first threshold; if so, save the noise monitoring data to the noise database for data update, and perform joint strong and weak supervision training on the noise event detection model according to the noise database after data update; if not, output the model prediction result; The noise event detection model is trained with joint strong and weak supervision according to the noise database after data update, specifically: Acquire noise data of the noise database after data update, and normalize the noise data; perform data enhancement on the normalized noise data to obtain enhanced audio data; Acquire the audio event of the enhanced audio data, construct a first tag for the enhanced audio data corresponding to the audio event with the start time and the end time; construct a second tag for the enhanced audio data corresponding to the audio event without the start time and the end time and with the segment-level tag; The enhanced audio data with the first label or the second label is input into the noise event detection model, the recognition result is output, and the overall loss of the recognition result is calculated; the parameters of the noise event detection model are gradient updated according to the overall loss to update the noise event detection model.

2. A detection method based on a noise event detection model as claimed in claim 1, characterized in that: After assigning the feature encoding layer parameters of the second initial noise event detection model, the method further includes: adding an LSTM layer to the downstream task module of the second initial noise event detection model.

3. A detection method based on a noise event detection model as claimed in claim 1, characterized in that: The noise event detection model is obtained by performing joint strong and weak supervision training on the first initial noise event detection model, specifically: Extract features from the training data to obtain input feature data; Inputting the input feature data into a first initial noise event detection model to output an initial recognition result; The initial recognition result includes: an initial first label probability and an initial second label probability; According to the true probability of the first label and the true probability of the second label, the overall loss of the initial recognition result is calculated; and according to the overall loss, the parameters of the first initial noise event detection model are gradient updated to obtain the noise event detection model.

4. A detection method based on a noise event detection model as claimed in claim 3, characterized in that: The noise event detection model is obtained by performing joint strong and weak supervision training on the first initial noise event detection model, and further includes: The first initial noise event detection model is trained with joint strong and weak supervision using different training data to obtain several noise event detection models; the noise event detection model with the highest average precision of all class labels is selected as the final noise event detection model.

5. The detection method based on the noise event detection model according to claim 3, characterized in that: The feature extraction of the training data to obtain input feature data is specifically as follows: Acquire a time domain signal of training data; perform short-time Fourier transform on the time domain signal to obtain an amplitude spectrum of the short-time Fourier transform; compress the amplitude spectrum of the short-time Fourier transform using a Mel filter group to obtain a Mel short-time Fourier transform amplitude spectrum; take the logarithm of the obtained Mel short-time Fourier transform amplitude spectrum to obtain a logarithmic Mel time-frequency spectrum feature as input feature data.

6. A detection method based on a noise event detection model as claimed in claim 3, characterized in that: The overall loss of the initial recognition result is calculated according to the true probability of the first label and the true probability of the second label, specifically: The binary cross entropy loss is calculated for the true probability of the first label and the initial first label probability to obtain the first label loss; the binary cross entropy loss is calculated for the true probability of the second label and the initial second label probability to obtain the second label loss; the first label loss is added to the second label loss to obtain the overall loss of the initial recognition result.

7. A detection device based on a noise event detection model, characterized in that: include: Detection module and update module; The detection module is used to input the acquired noise monitoring data into the noise event detection model to generate a model prediction result; wherein, the noise event detection model is obtained by performing joint strong and weak supervision training on a first initial noise event detection model; the first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and then migrating the preset parameters in the pre-trained neural network to a second initial noise event detection model; the joint strong and weak supervision training is to train the initial noise event detection model according to the first label and the second label of the training data; the model prediction result includes: the training data is a public and general audio data set; the first label probability and the second label probability of the noise monitoring data; The first initial noise event detection model is obtained by pre-training a pre-trained neural network with training data and migrating preset parameters in the pre-trained neural network to the second initial noise event detection model, specifically: Acquire training data to pre-train a pre-trained neural network, and update parameters of the pre-trained neural network; acquire feature coding layer parameters of the updated pre-trained neural network, migrate the feature coding layer parameters to a second initial noise event detection model, assign values ​​to the feature coding layer parameters of the second initial noise event detection model, and obtain a first initial noise event detection model; The updating module is used to determine whether the second label probability is higher than the first threshold according to the model prediction result; if so, the noise monitoring data is saved to the noise database for data update, and the noise event detection model is jointly trained with strong and weak supervision according to the noise database after data update; The noise event detection model is trained with joint strong and weak supervision according to the noise database after data update, specifically: Acquire noise data of the noise database after data update, and normalize the noise data; perform data enhancement on the normalized noise data to obtain enhanced audio data; Acquire the audio event of the enhanced audio data, construct a first tag for the enhanced audio data corresponding to the audio event with the start time and the end time; construct a second tag for the enhanced audio data corresponding to the audio event without the start time and the end time and with the segment-level tag; The enhanced audio data with the first label or the second label is input into the noise event detection model, the recognition result is output, and the overall loss of the recognition result is calculated; the parameters of the noise event detection model are gradient updated according to the overall loss to update the noise event detection model.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute a detection method based on a noise event detection model as described in any one of claims 1 to 6.

Images