Method for training model and environmental sound classification based on data mixing and label smoothing

Abstract

The present invention discloses a method for training a model based on data mixing and label smoothing and classifying environmental sounds, and relates to the technical field of speech or sound processing; the training method comprises step S1: obtaining batch data; step S2: time domain data enhancement and time domain preprocessing; step S3: mixed audio and label smoothing; step S4: frequency domain preprocessing and frequency domain enhancement; step S5: updating deep learning model parameters; by mixing audio and label smoothing, etc., the training of the deep learning model is more efficient; the classification method comprises using the trained deep learning model to classify the environmental sounds to obtain a classification result, and the classification result has higher accuracy.

Description

Technical Field

[0001] The present invention relates to the field of speech or sound processing technology, and in particular to a method for training a model and classifying environmental sounds based on data mixing and label smoothing. Background Art

[0002] Sound is one of the ways people receive and transmit information. At the same time, sound data has the advantage of non-contact measurement, so the application of sound classification is very extensive. Sound classification can be divided into environmental sound classification and human sound classification. Environmental sound classification can be applied to species detection, industrial sound anomaly detection, noise mapping, smart cities, and healthcare.

[0003] Since environmental sounds are high-dimensional data, deep learning models that can process complex high-dimensional data are the most commonly used environmental sound classification methods. However, since deep learning models are data-driven models, they require a large amount of sound data to train, which is time-consuming and labor-intensive. Summary of the invention

[0004] The present invention provides a method for training a model and environmental sound classification based on data mixing and label smoothing, which solves the technical problem of low efficiency in training deep learning models.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] A method for training a model based on data mixing and label smoothing, comprising the following steps:

[0007] Step S1: obtaining batch data;

[0008] Step S2: performing conventional time domain data enhancement and time domain preprocessing on the batch data to obtain batch data after time domain data enhancement and time domain preprocessing;

[0009] Step S3: randomly initialize a floating point number of a batch of data between 0 and 1. When the floating point number is less than the mixing probability, step S31 is executed to mix the audio and label smoothing for data enhancement. Otherwise, step S32 is executed without mixing the audio and label smoothing to obtain the batch data before frequency domain processing.

[0010] Step S31: Perform mixed audio and label smoothing on the batch data after time domain data enhancement and time domain preprocessing in step S2 to obtain smoothed batch data and labels, i.e., batch data before frequency domain processing, and execute step S4;

[0011] Step S32: without performing audio mixing and label smoothing, the batch data after the time domain data enhancement and time domain preprocessing in step S2, that is, the batch data before frequency domain processing, execute step S4;

[0012] Step S4: performing frequency domain preprocessing and frequency domain enhancement on the batch data before frequency domain processing obtained in step S3 to obtain batch data after frequency domain preprocessing and frequency domain enhancement;

[0013] Step S5: When the batch data before the frequency domain processing of step S4 is the smoothed batch data and labels obtained in step S31, the deep learning model is back-propagated based on the loss function of mixed audio and label smoothing to update the training parameters of the deep learning model; when the batch data before the frequency domain processing of step S4 is the batch data after time domain data enhancement and time domain preprocessing in step S2, the training parameters of the deep learning model are updated based on the cross entropy loss function.

[0014] A further technical solution is that in step S1, batch data is repeatedly selected from the total training samples to obtain batch data and used for training.

[0015] A further technical solution is that in step S2, conventional time domain data enhancement and time domain preprocessing include resampling, speech rate enhancement, decibel normalization and random cropping.

[0016] A further technical solution is that: the step S31 is divided into the following steps:

[0017] Step S311: randomly selecting two data from the batch data after time domain data enhancement and time domain preprocessing, respectively calculating the sound pressure level of each data, and obtaining the sound pressure levels of the two data;

[0018] Step S312: According to the random mixed decibel difference, the two data in step S311 are added to obtain mixed data x mixup ;

[0019] Step S313: Add all data except the two data in the batch data after time domain data enhancement and time domain preprocessing in step S311 according to the smoothed decibel difference to obtain new time domain sound signal data x mixupLabelSmoothing ;

[0020] Step S314: The time domain sound signal data x obtained in step S313 is mixupLabelSmoothing Perform decibel normalization to obtain the normalized mixed data x dbNormMixupLabelSmoothing ;

[0021] Step S315: Modify the loss function of the deep learning model to a loss function based on Mixup and label smoothing.

[0022] A further technical solution is that: the specific calculation steps of step S311 are: from s data (x1, y1), (x2, y2), ..., (x s ,y s) are randomly selected from the batch data, (x a ,y a ) and (x b ,y b ), s is the batch size batch_size, x is the acoustic signal sample, y is the label of the sample; calculate x a and x b The sound pressure level decibel value spl, spl formula is as follows:

[0023] (1)

[0024] In formula (1), spl(x) is the sound pressure level of x; x is the sound signal sample; n is the number of points of the sound signal sample x; x i is the i-th sampling point in x, i∈[1, n ];

[0025] The specific calculation steps of step S312 are: according to the random mixed decibel difference db_mixup, x a and x b Add together to get mixed data x mixup , the formula is as follows:

[0026] (2)

[0027] In formula (2), x mixup is mixed data; x a is the acoustic signal sample; x b is the sound signal sample; db_mixup is the mixed decibel difference;

[0028] The specific calculation steps of step S313 are: adding the remaining data of the batch data according to the smoothed decibel difference db_labelSmoothing, thereby obtaining the new time domain sound signal data x mixupLabelSmoothing , the formula is as follows:

[0029] (3)

[0030] In formula (3), x mixupLabelSmoothing is the new time domain sound signal data, x j For all the data except x in the batch a and x b Samples of

[0031] The specific calculation steps of step S314 are: mixupLabelSmoothing The decibel is normalized to the same decibel db_target to obtain the normalized mixed data x dbNormMixupLabelSmoothing , the formula is as follows:

[0032] (4)

[0033] In formula (4), x dbNormMixupLabelSmoothing is the normalized mixed data; db_target is the decibel normalization threshold;

[0034] At the same time a and b The label vector is modified based on label smoothing. The formula is as follows:

[0035] (5)

[0036] In formula (5), y k for y a and b An element of a vector; c is y a and b The category to which the vector belongs; n_c is the number of categories, and the ε smoothing coefficient is generally 0.01;

[0037] The specific calculation steps of step S315 are as follows: the formula of the loss function based on the mixed audio Mixup and label smoothing LabelSmoothing is as follows:

[0038] (6)

[0039] In formula (6), y pred The normalized mixed data x for the deep learning model dbNormMixupLabelSmoothing Output; y a_smooth for y a The label vector after label smoothing; y b_smooth for y b The label vector after label smoothing; CrossEntropy is the cross entropy loss function; L is the loss function model formula based on mixed audio Mixup and label smoothing Label Smoothing.

[0040] A further technical solution is that in step S4, the batch data after frequency domain preprocessing and frequency domain enhancement is the smoothed batch data and labels obtained in step S31 or the batch data after time domain data enhancement and time domain preprocessing in step S2.

[0041] A method for classifying environmental sounds, comprising classifying environmental sounds using a trained deep learning model to obtain classification results, and obtaining a trained deep learning model according to the above-mentioned method of training a model based on data mixing and label smoothing.

[0042] The beneficial effects of adopting the above technical solution are:

[0043] A method for training a model based on data mixing and label smoothing, comprising the following steps: step S1: obtaining batch data; step S2: performing conventional time domain data enhancement and time domain preprocessing on the batch data to obtain batch data after time domain data enhancement and time domain preprocessing; step S3: randomly initializing a floating point number of a batch data between 0 and 1, when the floating point number is less than the mixing probability, executing step S31 to perform mixed audio and label smoothing for data enhancement, otherwise executing step S32 without performing mixed audio and label smoothing to obtain batch data before frequency domain processing; step S31: performing mixed audio and label smoothing on the batch data after time domain data enhancement and time domain preprocessing in step S2 to obtain smoothed batch data and labels, i.e., batch data before frequency domain processing, and executing step S4; step S32 : Without mixed audio and label smoothing, the batch data after time domain data enhancement and time domain preprocessing in step S2 is the batch data before frequency domain processing, and step S4 is executed; step S4: the batch data before frequency domain processing obtained in step S3 is subjected to frequency domain preprocessing and frequency domain enhancement to obtain the batch data after frequency domain preprocessing and frequency domain enhancement; step S5: when the batch data before frequency domain processing in step S4 is the smoothed batch data and label obtained in step S31, the deep learning model is back-propagated based on the loss function of mixed audio and label smoothing, and the training parameters of the deep learning model are updated; when the batch data before frequency domain processing in step S4 is the batch data after time domain data enhancement and time domain preprocessing in step S2, the training parameters of the deep learning model are updated based on the loss function of cross entropy. It is more efficient to train deep learning models through mixed audio and label smoothing, etc.

[0044] A method for classifying environmental sounds includes classifying environmental sounds using a trained deep learning model to obtain classification results, and obtaining a trained deep learning model according to the above-mentioned method of training a model based on data mixing and label smoothing. The environmental sound classification results are obtained by using a trained deep learning model such as mixed audio and label smoothing, and the accuracy is higher.

[0045] Please refer to the detailed description of the specific implementation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of Embodiment 1 of the present invention;

[0047] Figure 2 yes Figure 1 Flow chart of step S3;

[0048] Figure 3 is a flow chart of Embodiment 2 of the present invention;

[0049] Figure 4 This is the ResNet-SE model training diagram without data mixing and label smoothing data enhancement;

[0050] Figure 5 This is the ResNet-SE model training diagram with data mixing and label smoothing data enhancement;

[0051] Figure 6 This is a test image of the ResNet-SE model without data mixing and label smoothing data enhancement;

[0052] Figure 7 This is a test image of the ResNet-SE model with data mixing and label smoothing data enhancement. DETAILED DESCRIPTION

[0053] Data augmentation techniques can enhance data diversity and data volume, thereby improving the generalization ability and robustness of the model and saving annotation costs. Sound data augmentation techniques include speech rate enhancement, noise enhancement, time slicing, time shifting, mixed audio Mixup, and spectrogram enhancement SpecAugment. These data augmentation methods have successfully improved the accuracy of the model.

[0054] Among them, mixed audio mixes two audios together, allowing the model to learn two labels, increasing the diversity of training samples. However, compared with the one hot vector label, label smoothing improves the generalization ability of the model by introducing a certain degree of uncertainty in the target label. Therefore, this application proposes an environmental sound classification method for sound data enhancement based on data mixing and label smoothing, which increases label uncertainty while mixing data, effectively improving the accuracy of the deep learning model.

[0055] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0057] First aspect: Data augmentation methods of data mixing and label smoothing are applied to deep learning model training.

[0058] Mixup is a data enhancement method that shows higher accuracy and better generalization performance in both image classification and speech recognition. Mixup generates new samples by randomly adding two samples and their label combinations during the training process of the deep learning model. This method significantly increases the diversity of samples, thereby suppressing the overfitting of the model. At the same time, the new samples generated by the Mixup method contain information from multiple labels, making it easier for the model to learn the similarities and differences between different labels.

[0059] However, after the mixed audio Mixup is combined, the label is one-hot encoded, which is prone to overfitting. Compared with the one-hot encoded label, label smoothing improves the generalization ability of the model by adding some errors to the target label, thereby introducing a certain degree of uncertainty. Therefore, this application proposes a method for sound data enhancement based on data mixing and label smoothing.

[0060] The main steps of applying data augmentation methods of data mixing and label smoothing to deep learning model training are described as follows.

[0061] Embodiment 1:

[0062] like Figure 1 and Figure 2 As shown, the present invention discloses a method for training a model based on data mixing and label smoothing, comprising step S1: obtaining batch data; step S2: time domain data enhancement and time domain preprocessing; step S3: mixing audio and label smoothing; step S4: frequency domain preprocessing and frequency domain enhancement; step S5: updating deep learning model parameters. The details are as follows.

[0063] like Figure 1 As shown in Figure 1, data augmentation methods of data mixing and label smoothing are applied to the steps of deep learning model training.

[0064] Step S1: Obtain batch data.

[0065] Batch data is obtained by repeatedly selecting from the total training samples and used for training.

[0066] Since the training samples of deep learning models are often very large, it is not possible to load the training samples into the video memory at one time. In order to more reasonably utilize the parallel computing power of the GPU and the allocation of video memory, only a batch of batch data of batch_size needs to be loaded during deep learning model training.

[0067] Step S2: time domain data enhancement and time domain preprocessing.

[0068] Conventional time domain data enhancement and time domain preprocessing are performed on the batch data to obtain batch data after time domain data enhancement and time domain preprocessing.

[0069] Time domain data enhancement is to perform operations on the time domain dimension of the sound signal to increase the amount of data of the sound signal, thereby increasing the diversity of the sound signal in the time domain. Time domain preprocessing is to perform operations on the time domain dimension of the sound signal to unify the signal specifications, such as sampling rate and decibel.

[0070] Conventional time domain data enhancement and time domain preprocessing include: resampling, speech rate enhancement, decibel normalization, and random cropping.

[0071] Time domain resampling is to unify the sampling rate of the sound signal of the batch data to the same sampling rate sample_rate. Since batch data is often obtained by various different acquisition methods, it has different sampling rates. In order to make the batch data have a uniform sampling rate, the batch data needs to be resampled first.

[0072] Time domain speech rate enhancement is to fix the sampling rate and use interpolation to increase / decrease the number of time domain points of batch data according to a random proportion of speech_rate, thereby reducing / increasing the speech rate of the sound signal, thereby increasing the diversity of the speech rate of the data.

[0073] Decibel normalization is to unify the decibel of the sound signal to the same decibel db_target. Since the decibels of the batch data are different, in order to make the batch data have a uniform decibel, the decibel of the batch data needs to be normalized first.

[0074] Random time-domain cropping is to randomly crop the sound signal in the time domain to obtain different time-domain signals. If a data in the batch data is lower than the shortest duration min_duration, it will be discarded because it is not conducive to training. If a data in the batch data is between the shortest duration min_duration and the longest duration max_duration, it does not need to be cropped. If a data in the batch data is greater than the longest duration max_duration, it will be randomly cropped to the longest duration max_duration. Finally, in order to facilitate GPU acceleration calculations, the time domain of this batch of data will be padded with zeros to the same size.

[0075] Step S3: Mixed audio and label smoothing.

[0076] A floating point number of a batch of data is randomly initialized between 0 and 1. When the floating point number is less than the mixing probability, step S31 is executed to mix the audio and label smoothing for data enhancement. Otherwise, step S32 is executed without mixing the audio and label smoothing to obtain the batch data before frequency domain processing.

[0077] like Figure 2As shown in the figure, the steps of audio mixing Mixup and label smoothing are described in detail as follows.

[0078] Step S31: Perform mixed audio and label smoothing on the batch data after time domain data enhancement and time domain preprocessing in step S2 and update the loss function to obtain the smoothed batch data and labels, i.e., the batch data before frequency domain processing, and execute step S4.

[0079] The selection range of the mixing probability mixup_probability is ∈[0, 1].

[0080] The specific steps of the step S31 audio mixing Mixup and label smoothing data enhancement are as follows:

[0081] Step S311: randomly select two data from the batch data after time domain data enhancement and time domain preprocessing, calculate the sound pressure level of each data respectively, and obtain the sound pressure levels of the two data.

[0082] The specific calculation steps are: from s data (x1, y1), (x2, y2), ..., (x s ,y s ) are randomly selected from the batch data, (x a ,y a ) and (x b ,y b ), s is the batch size batch_size, x is the acoustic signal sample, and y is the label of the sample. Calculate x a and x b The sound pressure level decibel value SPL, the abbreviation of sound pressure level is SPL, and the SPL formula is as follows:

[0083] (1)

[0084] In formula (1), spl(x) is the sound pressure level of x; x is the sound signal sample; n is the number of points of the sound signal sample x; x i is the i-th sampling point in x, i∈[1, n ].

[0085] Step S312: According to the random mixed decibel difference, the two data in step S311 are added to obtain mixed data x mixup .

[0086] The specific calculation steps are as follows: According to the random mixed decibel difference db_mixup, x a and x b Add together to get mixed data x mixup , the formula is as follows:

[0087] (2)

[0088] In formula (2), x mixup is mixed data; x a is the acoustic signal sample; x b is the sound signal sample; db_mixup is the mixed decibel difference.

[0089] Mix the data x like this mixup The sample contains x a and x b Information, x a and x b The difference is db_mixup decibels.

[0090] Step S313: Add all data except the two data in the batch data after time domain data enhancement and time domain preprocessing in step S311 according to the smoothed decibel difference to obtain new time domain sound signal data x mixupLabelSmoothing .

[0091] The specific calculation steps are: add the remaining data of the batch data according to the smoothed decibel difference db_labelSmoothing to obtain the new time domain sound signal data x mixupLabelSmoothing , the formula is as follows:

[0092] (3)

[0093] In formula (3), x mixupLabelSmoothing is the time domain sound signal data, x j For all the data except x in the batch a and x b of the sample.

[0094] Mix the data x like this mixup The sample contains all the data except x. a and x b Information about other categories of samples, other categories and mixed data x mixup Difference db_labelSmoothing in dB.

[0095] Step S314: The time domain sound signal data x obtained in step S313 is mixupLabelSmoothing Perform decibel normalization to obtain the normalized mixed data x dbNormMixupLabelSmoothing .

[0096] The specific calculation steps are: Then the new time domain sound signal data x mixupLabelSmoothing The decibel is normalized to the same decibel db_target to obtain the normalized mixed data x dbNormMixupLabelSmoothing , the formula is as follows:

[0097] (4)

[0098] In formula (4), x dbNormMixupLabelSmoothing is the normalized mixed data; db_target is the decibel normalization threshold.

[0099] At the same time a and b The label vector is modified based on label smoothing. The formula is as follows:

[0100] (5)

[0101] In formula (5), y k for y a and b An element of a vector; c is y a and b The category to which the vector belongs; n_c is the number of categories, and the ε smoothing coefficient is generally 0.01.

[0102] Step S315: modify the loss function.

[0103] The specific calculation steps are as follows: Finally, the loss function of the deep learning model needs to be modified. The loss function model based on mixed audio Mixup and label smoothing Label Smoothing is the weighted multi-classification cross entropy loss Cross-EntropyLoss. The formula of the loss function based on mixed audio Mixup and label smoothing Label Smoothing is as follows:

[0104] (6)

[0105] In formula (6), y pred The normalized mixed data x for the deep learning model dbNormMixupLabelSmoothing Output; y a_smooth for y a The label vector after label smoothing; y b_smooth for y b The label vector after label smoothing; CrossEntropy is the cross entropy loss function; L is the loss function based on mixed audio Mixup and label smoothing Label Smoothing.

[0106] Step S32: Without performing mixed audio and label smoothing, the batch data after the time domain data enhancement and time domain preprocessing in step S2, that is, the batch data before frequency domain processing, execute step S4.

[0107] Step S4: frequency domain preprocessing and frequency domain enhancement.

[0108] The batch data before frequency domain processing obtained in step S3 is subjected to frequency domain preprocessing and frequency domain enhancement to obtain batch data after frequency domain preprocessing and frequency domain enhancement. The batch data after frequency domain preprocessing and frequency domain enhancement includes the smoothed batch data and labels obtained in step S31 and the batch data after time domain data enhancement and time domain preprocessing in step S2.

[0109] Converting time domain signals into spectrogram features is more in line with human perception of sound and can also achieve the purpose of data dimensionality reduction. Filter bank, abbreviated as Fbank, is a feature extraction method commonly used in speech processing and speech recognition. It converts the time domain audio signal into spectrogram features. The main steps are as follows: pre-emphasis, short-time Fourier transform, abbreviated as STFT, power spectrum transform, weighted summation of Mel-scale triangular filters, and finally logarithmic transform.

[0110] After the frequency domain transformation, the spectrogram features need to be enhanced in the frequency domain. Common methods are frequency domain masking and time domain masking. Frequency domain masking is to randomly mask the frequency axis of the spectrogram features, and time domain masking is to randomly mask the time axis of the spectrogram features, thereby increasing the diversity of features.

[0111] Step S5: Update deep learning model parameters.

[0112] When the batch data after frequency domain preprocessing and frequency domain enhancement in step S4 is the smoothed batch data and labels obtained in step S31, the deep learning model backpropagates based on the loss function of mixed audio Mixup and label smoothing, and the training parameters of the deep learning model are updated; when the batch data after frequency domain preprocessing and frequency domain enhancement in step S4 is the batch data after time domain data enhancement and time domain preprocessing in step S2, the training parameters of the deep learning model are updated based on the cross entropy loss function.

[0113] Backpropagation is a key algorithm in deep learning model training, which is used to calculate and propagate gradients to update model parameters. Its core idea is to apply the chain rule, propagate errors from the output layer to the input layer, calculate gradients layer by layer, and update parameters. Backpropagation is usually used in conjunction with the optimization function Adam to improve the efficiency of training.

[0114] Second aspect: In Example 1, after the deep learning model is trained, the accuracy of the model needs to be evaluated through an evaluation process. The accuracy of the model can be evaluated by inputting the test set into the trained model. In the accuracy evaluation stage of the model, the data enhancement method is not used, only the data preprocessing method is used.

[0115] Embodiment 2:

[0116] like Figure 3 As shown, the present invention discloses a method for evaluating the accuracy of a deep learning model, including step S6: evaluating the accuracy of a deep learning model, which is described in detail as follows.

[0117] Step S6 specifically includes the following steps:

[0118] Step S601: repeatedly selecting batch data from the total test samples for testing.

[0119] Step S602: Batch data resampling.

[0120] Step S603: Batch data decibel normalization.

[0121] Step S604: Cut to the same length to facilitate GPU acceleration.

[0122] Step S605: frequency domain preprocessing, Fbank spectrogram feature extraction.

[0123] Step S606: Input the Fbank spectrogram features into the model and evaluate the accuracy.

[0124] Embodiment 3:

[0125] The present invention discloses a method for classifying environmental sounds, comprising adopting the method based on data mixing and label smoothing training model described in Example 1 to obtain a trained deep learning model; and classifying the environmental sounds using the trained deep learning model to obtain a classification result.

[0126] Embodiment 4:

[0127] The present invention discloses a model training device based on data mixing and label smoothing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the steps of Example 1 are implemented when the processor executes the computer program.

[0128] Embodiment 5:

[0129] The present invention discloses a device for classifying environmental sounds, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory and the processor form an electronic terminal, and the processor implements the steps of Example 3 when executing the computer program.

[0130] Embodiment 6:

[0131] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in Embodiment 1 are implemented.

[0132] Embodiment 7:

[0133] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in Embodiment 3 are implemented.

[0134] In order to make the content, purpose and advantages of the present application clearer and more detailed, the present application is described in more detail in conjunction with the following embodiments.

[0135] This application uses the public dataset Urban8K to verify the superiority of data enhancement with data mixing and label smoothing. Urban8K is an open dataset for environmental sound classification and recognition, which is widely used in machine learning and sound processing research. It contains common sound samples in various urban environments. The characteristics of the dataset are as follows:

[0136] 1) Number of samples: The Urban8K dataset contains 8732 sound samples.

[0137] 2) Categories: These sound samples are divided into 10 categories, including car horns, dog barking, drilling, engine idling, gunshots, children playing, ambulances, street music, police sirens, and wheels sliding.

[0138] 3) Sampling rate: The sampling rate of each sound sample is 44.1 kHz, mono.

[0139] 4) Duration: The duration of each sample is 4 seconds.

[0140] 5) Data format: Sound samples are stored in WAV format.

[0141] Therefore, the Urban8K dataset can be applied to a variety of environmental sound classification scenarios.

[0142] In the selection of deep learning models, this application uses the residual network deep learning model ResNet-SE with an attention mechanism that has good results in both image and speech classification. ResNet with Squeeze-and-Excitation is abbreviated as ResNet-SE.

[0143] This application splits the training set and test set in a ratio of 9:1 for model training and model evaluation respectively.

[0144] The parameters of the data augmentation method based on data mixing and label smoothing applied to model training are as follows:

[0145] 1) Get batch data: batch_size is 32 to get batch data.

[0146] 2) Time domain data enhancement and time domain preprocessing: the sample_rate of the resampling is 16000; the speech_rate of the speech rate enhancement is randomly set to 0.9, 1.0, and 1.1; the db_target of the decibel normalization is -20; the min_duration of the random clipping is 0.5 seconds, and the max_duration is 3 seconds.

[0147] 3) Mixup and label smoothing: the data mixing probability mixup_probability is 0.3; the mixed decibel difference db_mixup is randomly set between -5 and 5; the smoothing decibel difference db_labelSmoothing is 80.

[0148] 4) Frequency domain preprocessing and frequency domain enhancement: The frame length of Fbank is 25 milliseconds and the frame shift is 10 milliseconds.

[0149] 5) Model update parameters, back propagation: The learning rate of the optimizer Adam is 0.001.

[0150] Experimental results show:

[0151] In this experiment, the accuracy of the ResNet-SE model with data augmentation based on data mixing and label smoothing and the ResNet-SE model without data augmentation based on data mixing and label smoothing were compared on the Urban8K dataset, with accuracy as the indicator for evaluating the quality of the method.

[0152] During the training process, every time a batch of data from the training set is taken out 10 times, the accuracy of this batch of data under the updated model will be recorded to generate a training accuracy graph.

[0153] like Figure 4 The following figure shows the training diagram of the ResNet-SE model without data enhancement methods based on data mixing and label smoothing. The horizontal axis is the number of batches of training data, and the vertical axis is the accuracy of the batch data of training, that is, the accuracy rate.

[0154] like Figure 5 The figure shows the training diagram of the ResNet-SE model based on data mixing and label smoothing data enhancement method. The horizontal axis is the number of batch data for training, and the vertical axis is the accuracy of the batch data for training, that is, the accuracy rate.

[0155] from Figure 4 It can be seen that due to the lack of data enhancement based on data mixing and label smoothing, the model quickly converges to the highest accuracy of the training data after a certain number of iterations. Figure 5In the data enhancement method based on data mixing and label smoothing, after a certain number of iterations, the training accuracy of the model still fluctuates significantly. This is because the data enhancement method based on data mixing and label smoothing increases the number and difficulty of training, so the training accuracy fluctuates greatly.

[0156] During the training process, each time the model has trained all the batches of data in the training set, that is, an epoch, it will record the accuracy of the test set and generate a test accuracy graph.

[0157] like Figure 6 The figure shows the test graph of the ResNet-SE model without data enhancement methods based on data mixing and label smoothing. The horizontal axis is the number of epochs of the test, and the vertical axis is the epoch accuracy of the test, that is, the accuracy rate.

[0158] like Figure 7 The figure shows the test graph of the ResNet-SE model based on the data enhancement method of data mixing and label smoothing. The horizontal axis is the number of epochs of the test, and the vertical axis is the accuracy of the test epoch, that is, the accuracy rate.

[0159] from Figure 6 and Figure 7 It can be seen that the accuracy of the test set of the ResNet-SE model without data augmentation based on data mixing and label smoothing and the ResNet-SE model with data augmentation based on data mixing and label smoothing gradually converges with continuous iterative training. The best accuracy of the ResNet-SE model without data augmentation based on data mixing and label smoothing is 96.4%; the best accuracy of the ResNet-SE model with data augmentation based on data mixing and label smoothing is 97.1%. Therefore, it is concluded that the ResNet-SE model with data augmentation based on data mixing and label smoothing is better than the ResNet-SE model without data augmentation based on data mixing and label smoothing, and the data augmentation based on data mixing and label smoothing is an effective way to improve the accuracy of the model.

[0160] Technical effect:

[0161] This application proposes a data enhancement method of data mixing and label smoothing, using the ResNet-SE model. Through the verification of the Urban8K dataset, the data enhancement method of data mixing and label smoothing can effectively increase the data volume, increase the data diversity, improve the performance of the model, and increase the accuracy of sound classification.

Claims

1. A method for training a model based on data mixing and label smoothing, characterized in that: The following steps are included: Step S1: obtaining batch data; Step S2: performing conventional time domain data enhancement and time domain preprocessing on the batch data to obtain batch data after time domain data enhancement and time domain preprocessing; Step S3: randomly initialize a floating point number of a batch of data between 0 and 1. When the floating point number is less than the mixing probability, step S31 is executed to mix the audio and label smoothing for data enhancement, otherwise step S32 is executed without mixing the audio and label smoothing; Step S31: Perform mixed audio and label smoothing on the batch data after time domain data enhancement and time domain preprocessing in step S2 to obtain smoothed batch data and labels, i.e., batch data before frequency domain processing, and execute step S4; The step S31 is divided into the following steps: Step S311: randomly selecting two data from the batch data after time domain data enhancement and time domain preprocessing, respectively calculating the sound pressure level of each data, and obtaining the sound pressure levels of the two data; The specific calculation steps of step S311 are: from s data (x1, y1), (x2, y2), ..., (x s ,y s ) are randomly selected from the batch data, (x a ,y a ) and (x b ,y b ), s is the batch size batch_size, x is the acoustic signal sample, y is the label of the sample; calculate x a and x b The sound pressure level decibel value spl, spl formula is as follows: (1) In formula (1), spl(x) is the sound pressure level of x; x is the sound signal sample; n is the number of points of the sound signal sample x; x i is the i-th sampling point in x, i∈[1, n ]; Step S312: According to the random mixed decibel difference, the two data in step S311 are added to obtain mixed data x mixup ; The specific calculation steps of step S312 are: according to the random mixed decibel difference db_mixup, x a and x b Add together to get mixed data x mixup , the formula is as follows: (2) In formula (2), x mixup is mixed data; x a is the acoustic signal sample; x b is the sound signal sample; db_mixup is the mixed decibel difference; Step S313: Add all data except the two data in the batch data after time domain data enhancement and time domain preprocessing in step S311 according to the smoothed decibel difference to obtain new time domain sound signal data x mixupLabelSmoothing ; The specific calculation steps of step S313 are: adding the remaining data of the batch data according to the smoothed decibel difference db_labelSmoothing, thereby obtaining the new time domain sound signal data x mixupLabelSmoothing , the formula is as follows: (3) In formula (3), x mixupLabelSmoothing is the new time domain sound signal data, x j For the batch data except x a and x b Samples of Step S314: The time domain sound signal data x obtained in step S313 is mixupLabelSmoothing Perform decibel normalization to obtain the normalized mixed data x dbNormMixupLabelSmoothing， At the same time, the label vector is modified according to the label smoothing; The specific calculation steps of step S314 are: mixupLabelSmoothing The decibel is normalized to the same decibel db_target to obtain the normalized mixed data x dbNormMixupLabelSmoothing , the formula is as follows: (4) In formula (4), x dbNormMixupLabelSmoothing is the normalized mixed data; db_target is the decibel normalization threshold; At the same time a and b Modify the label vector based on label smoothing. The formula is as follows: (5) In formula (5), y k for y a_smooth and b_smooth An element of a vector; c is y a and b The category to which the vector belongs; n_c is the number of categories, ε is the smoothing coefficient; Step S315: modifying the loss function of the deep learning model to a loss function based on Mixup and label smoothing; The specific calculation steps of step S315 are as follows: the formula of the loss function based on the mixed audio Mixup and label smoothing Label Smoothing is as follows: (6) In formula (6), y pred The normalized mixed data x for the deep learning model dbNormMixupLabelSmoothing Output; y a_smooth for y a The label vector after label smoothing; y b_smooth for y b The label vector after label smoothing; CrossEntropy is the cross entropy loss function; L is the loss function model formula based on mixed audio Mixup and label smoothing Label Smoothing; Step S32: without performing audio mixing and label smoothing, the batch data after the time domain data enhancement and time domain preprocessing in step S2, that is, the batch data before frequency domain processing, execute step S4; Step S4: performing frequency domain preprocessing and frequency domain enhancement on the batch data before frequency domain processing obtained in step S3 to obtain batch data after frequency domain preprocessing and frequency domain enhancement; Step S5: When the batch data before the frequency domain processing of step S4 is the smoothed batch data and labels obtained in step S31, the deep learning model is back-propagated based on the loss function of mixed audio and label smoothing to update the training parameters of the deep learning model; when the batch data before the frequency domain processing of step S4 is the batch data after time domain data enhancement and time domain preprocessing in step S2, the training parameters of the deep learning model are updated based on the cross entropy loss function.

2. The method for training a model based on data mixing and label smoothing according to claim 1, characterized in that: In step S1, batch data is repeatedly selected from the total training samples to obtain batch data for training.

3. The method for training a model based on data mixing and label smoothing according to claim 1, characterized in that: In step S2, conventional time domain data enhancement and time domain preprocessing include resampling, speech rate enhancement, decibel normalization and random cropping.

4. The method for training a model based on data mixing and label smoothing according to claim 1, characterized in that: In step S4, the batch data before frequency domain processing is the smoothed batch data and labels obtained in step S31 or the batch data after time domain data enhancement and time domain preprocessing in step S2.

5. A method for classifying environmental sounds, comprising classifying environmental sounds using a trained deep learning model to obtain a classification result, characterized in that: According to the method for training a model based on data mixing and label smoothing according to any one of claims 1 to 4, a trained deep learning model is obtained.

Images