A data augmentation method for audio, a real-time sound event detection system and method

Abstract

The present invention discloses a method for data augmentation of audio and a real-time sound event detection system and method, including: establishing two deep learning models, where the deep learning models include a pre-awakening model and a detection model; deforming audio data through a series of audio transformations to obtain deformed audio; extracting spectral features from the deformed audio; randomly masking the spectral features to obtain data after data augmentation; using the data after data augmentation to train the deep learning models in the sound event detection method and saving the trained models; using a microphone to record an audio stream in real time, slicing the audio stream and sending it into the trained models for detection to obtain detection results. According to the present invention, the sound events to be detected can be quickly and accurately fed back, and the number of model parameters and the memory occupancy in this method are small, meeting the conditions for mobile use.

Description

Technical Field

[0001] The present invention relates to the technical field of sound event detection, and particularly relates to a data augmentation method for audio, a real-time sound event detection system, and a method thereof. Background Art

[0002] Sound Events Detection (SED) refers to analyzing different sound signals to identify the categories and start and end times of the information contained in each sound source. For example, detecting the honking of cars on the road, the coughing of patients in the hospital, and the abnormal noise of machine operation. Currently, sound event detection has been widely used in devices such as smart furniture, medical care, and environmental monitoring, and has high application prospects. However, for the same sound event in the real world, its characteristics vary greatly in different environments, and there are interference factors such as the cocktail party effect in an environment with multiple sounds mixed, resulting in certain limitations of traditional machine learning methods in sound event detection tasks. For example, when the number of sound event categories is large, the performance of the Support Vector Machine (SVM) model will decrease accordingly; the characteristics of sound signals are highly non-linear, so the performance of traditional machine learning models such as the Hidden Markov Model (HMM) and the Guassian Mixed Model (GMM) is inferior to that of deep learning models.

[0003] Currently, many scholars have started to use deep learning models for sound event detection. Although its accuracy has exceeded 80%, the computational complexity is high, and the detection results cannot be quickly fed back. Since it is costly to obtain sound data in different environments in the real world, and the training of neural network models depends relatively on the amount of data, too little data is likely to cause the model to be unable to effectively learn or overfit, resulting in weak generalization ability of the model in different environments. Summary of the Invention

[0004] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a data augmentation method for audio, a real-time sound event detection system, and a method thereof, which can quickly and accurately feedback the sound events to be detected, and the number of parameters and the memory occupied by the model in this method are small, meeting the conditions for use on mobile devices. To achieve the above object and other advantages according to the present invention, there is provided a data augmentation method for audio and a real-time sound event detection system, including:

[0005] A deep learning model, the deep learning model includes a processing module, a pre-wake-up module and a detection module. The processing module is used to transform audio data, extract spectral features from the transformed audio, and convert the original audio data, extract the spectral features of the converted audio and perform random spectral masking;

[0006] The pre-wake-up module and the detection module need to be trained respectively. After the pre-wake-up module is trained, the pre-wake-up module is saved. The pre-wake-up module is a lightweight deep learning model composed of a multi-layer convolutional neural network;

[0007] After the detection module is trained, the detection module is saved. The detection module is composed of a multi-layer convolutional neural network.

[0008] An audio data augmentation method and a real-time sound event detection method, including the following steps:

[0009] S1. Establish a deep learning model, including a pre-wake-up model and a detection model;

[0010] S2. Transform the audio data through a series of audio transformations to obtain the transformed audio;

[0011] S3. Extract spectral features from the transformed audio and mask the spectrum to obtain the result after data augmentation. Extracting spectral features includes framing and windowing the audio, and then using the short-time Fourier transform to obtain spectral features;

[0012] S4. Use the result after data augmentation to train the pre-wake-up model and the detection model, and save the trained pre-wake-up model and detection model;

[0013] S5. Record the audio stream through a microphone, slice the audio stream, and input it into the trained pre-wake-up model to obtain the output probability of the pre-wake-up model;

[0014] S6. When the output probability of the pre-wake-up model is greater than the preset wake-up threshold, or when a volume fluctuation is detected, activate the detection model for detection to obtain the output probability of the detection model;

[0015] S7. When the probability of a certain category output by the detection model is greater than the preset detection threshold, return the detection result.

[0016] Preferably, both the pre-wake-up model and the detection model are lightweight convolutional neural network models, composed of a multi-layer convolutional neural network, and equipped with batch normalization and skip connections to facilitate optimizing model parameters.

[0017] Preferably, the series of audio transformations in step S2 includes applying a single or multiple audio data transformation operations, such as volume transformation, pitch transformation, audio stretching and noise addition, to the original audio data in a combined form.

[0018] Preferably, the spectral features in step S3 use Mel spectrum features, and masking the spectrum refers to randomly masking the spectrum from the frequency domain and time domain dimensions. The masking width and number are manually set hyperparameters. The masked value is the minimum value of the area, and the logarithm of the masked Mel spectrum is taken to obtain the logarithmic Mel spectrum feature.

[0019] Preferably, the training process of the pre-wake-up model and the detection model in step S4 includes setting a smaller learning rate and a medium-sized batch size as hyperparameters, using an adaptive optimizer, adopting L2 regularization to prevent overfitting, adopting an interval adjustment method to attenuate the learning rate, using a cross-entropy loss function to calculate the loss value, and training the pre-wake-up model and the detection model through a back-propagation algorithm until the performance of the pre-wake-up model and the detection model reaches the best and is saved.

[0020] Preferably, recording the audio stream with a microphone in step S5 refers to recording a mono audio stream with a lower sampling rate using a microphone. The audio stream slicing refers to dividing the audio stream into a segment every few seconds, and the segment slides every few seconds to obtain real-time audio segments.

[0021] Preferably, the preset awakening threshold in step S6 is obtained by analyzing a receiver operating characteristic (ROC) curve. When the probability of any category in the output probability of the pre-awakening model is greater than the preset threshold, the detection model will be activated.

[0022] Preferably, the preset detection threshold in step S7 sets a threshold for each category separately, and when the detection model outputs a probability of a certain category greater than the preset threshold, the label information and timestamp of the category are returned.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] (1) The real-time sound event detection method of the present invention has a response speed of less than 0.5 seconds, providing timely and accurate feedback on the sound events to be detected. Furthermore, the model in this method occupies a relatively small memory footprint, totaling only 19MB, and the number of model parameters is also relatively small, totaling 4.82MB, making it suitable for mobile device use.

[0025] (2) The pre-wake-up model in the real-time sound event detection method of the present invention has low computational complexity and consumes only a small amount of computational cost in the standby state, which can ensure long-term battery life of the mobile device.

[0026] (3) The audio data augmentation method in the present invention can greatly improve the performance and generalization ability of the deep learning model, and effectively avoid the phenomena of training collapse or overfitting of the model. Description of the Drawings

[0027] Figure 1 It is a schematic flowchart of the audio data augmentation method, real-time sound event detection system and method according to the present invention;

[0028] Figure 2 It is a schematic flowchart of the audio data augmentation method of the audio data augmentation method, real-time sound event detection system and method according to the present invention;

[0029] Figure 3 It is a schematic flowchart of the deep learning model training of the audio data augmentation method, real-time sound event detection system and method according to the present invention;

[0030] Figure 4 It is a comparison diagram of the effects of the audio data augmentation method of the audio data augmentation method, real-time sound event detection system and method according to the present invention;

[0031] Figure 5 It is a schematic flowchart of the real-time sound event detection method of the audio data augmentation method, real-time sound event detection system and method according to the present invention. Detailed Embodiments

[0032] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0033] Refer to Figures 1-5 , an audio data augmentation method and a real-time sound event detection system, including: a deep learning model, the deep learning model includes a processing module, a pre-awakening module and a detection module, the processing module is used to deform the audio data, extract spectral features from the deformed audio, and the processing module is used to perform audio conversion on the original audio data, extract spectral features of the converted audio and perform random spectral masking;

[0034] The pre-awakening module and the detection module need to be trained respectively. After the pre-awakening module is trained, it is saved, and the pre-awakening model is a lightweight deep learning model composed of a multi-layer convolutional neural network;

[0035] After the detection module is trained, it is saved. The detection module is composed of a multi-layer convolutional neural network.

[0036] An audio data augmentation method and a real-time sound event detection method include the following steps:

[0037] S1. Establish a deep learning model, including a pre-awakening model and a detection model;

[0038] S2. Deform the audio data through a series of audio transformations to obtain the deformed audio;

[0039] S3. Extract spectral features from the deformed audio and mask the spectrum to obtain the result after data augmentation;

[0040] S4. Use the result after data augmentation to train the pre-awakening model and the detection model, and save the trained pre-awakening model and detection model;

[0041] S5. Record an audio stream using a microphone, slice the audio stream, and input it into the trained pre-awakening model to obtain the output probability of the pre-awakening model;

[0042] S6. When the output probability of the pre-awakening model is greater than a preset awakening threshold, or when a volume fluctuation is detected, activate the detection model for detection to obtain the output probability of the detection model;

[0043] S7. When the probability of a certain category output by the detection model is greater than a preset detection threshold, return the detection result.

[0044] Furthermore, both the pre-awakening model and the detection model are lightweight convolutional neural network models, composed of a multi-layer convolutional neural network, and batch normalization and skip connections are used to facilitate the optimization of model parameters.

[0045] Furthermore, the series of audio transformations in step S2 include applying single or multiple audio data transformation operations, such as volume transformation, pitch transformation, audio stretching, and adding noise, etc., to the original audio data in a combined form.

[0046] Furthermore, in step S3, the spectral features use Mel spectral features. The masking of the spectrum refers to randomly masking the spectrum from the frequency domain and time domain dimensions. The width and number of masks are both hyperparameters set manually. The masked value is the minimum value in that area, and the logarithm of the masked Mel spectrum is taken to obtain the logarithmic Mel spectral features.

[0047] Further, the training process of the pre-awakening model and the detection model in step S4 includes setting a relatively small learning rate, a medium-sized batch size as hyperparameters, and using an adaptive optimizer. L2 regularization is adopted to prevent overfitting, and the learning rate is decayed by adjusting the learning rate at intervals. The cross-entropy loss function is used to calculate the loss value, and the pre-awakening model and the detection model are trained through the backpropagation algorithm until the performance of the pre-awakening model and the detection model reaches the best, and then they are saved.

[0048] Further, in step S5, an audio stream is recorded using a microphone and sliced. A monophonic audio stream is recorded using an external microphone at a sampling rate of 16 kHz. The audio stream is sliced into a segment every 2 seconds, and the segment slides every 0.5 seconds. The segment is input into the pre-detection model for detection.

[0049] Further, the preset awakening threshold in step S6 is obtained by analyzing the ROC curve. When the output probability of any category of the pre-awakening model is greater than the preset threshold, the detection model is activated.

[0050] Further, for the preset detection threshold in step S7, the threshold for each category is set separately. When the probability of a certain category output by the detection model is greater than the preset threshold, the label information and timestamp of that category are returned.

[0051] First, the relevant terms and related concepts such as deep learning involved in the embodiments of the present application are introduced below.

[0052] [[ID=tmp15]](1) Feedforward neural network

[0053] A feedforward neural network (FNN) is a unidirectional multi-layer structure, including an initial input layer, intermediate hidden layers, and a final output layer, where the hidden layers can be single-layer or multi-layer. There are several neurons in each layer, and the neurons between layers are all connected. The neurons in each layer can receive the signals from the neurons in the previous layer and generate outputs to the next layer.

[0054] [[ID=tmp21]](2) Convolutional neural network

[0055] The convolutional neural network (CNN) is a feedforward neural network with a convolutional structure. The CNN uses filters (composed of convolutional kernels and bias vectors) to continuously slide and perform Hadamard product operations on data of the corresponding window size, and then sum with the bias vector to obtain a new value. The calculation process is shown in Equation (1). CNN has the characteristic of weight sharing. It has fewer learning parameters, can effectively avoid overfitting, and has high computational efficiency. In addition, due to the existence of local receptive fields, CNN can combine the representation information of surrounding data, and as the convolutional layer deepens, it can abstract higher-dimensional features of the data.

[0056]

[0057] Where ⊙ is the Hadamard product.

[0058] (3) Loss function

[0059] The loss function is used to measure the degree of difference between the predicted value f(x) of the model and the true value Y. It is a non-negative value function, usually denoted by L(Y, f(x)). The smaller the loss function, the closer the predicted value of the model is to the true value. The training process of the neural network model is to use the optimizer and backpropagation algorithm to continuously update the model parameters through iteration, so that the value of the loss function continuously decreases until the loss value completely converges. At this time, the training process of the model ends.

[0060] (4) Backpropagation algorithm

[0061] The backpropagation algorithm (BP) is a learning algorithm suitable for multi-layer neural networks. It is based on the gradient descent method. The learning process of the BP algorithm consists of a forward propagation process and a backpropagation process. In the forward propagation process, the input information passes through the input layer, through the hidden layer, and is processed layer by layer and transmitted to the output layer. Calculate the degree of difference between the model output and the true value as the objective function, transfer to the backpropagation, and layer by layer find the partial derivative of the objective function with respect to the weights of each neuron, forming the gradient of the objective function with respect to the weight vector as the basis for modifying the weights. The training of the network is completed during the process of modifying the weights. When the error reaches the desired range, the training process ends.

[0062] Refer to Figure 1 , a data augmentation method for audio and a real-time sound event detection method, including: S1. Establish two deep learning models, and the deep learning models include a pre-awakening model and a detection model;

[0063] S2. Deform the audio data through a series of audio transformations to obtain the deformed audio;

[0064] S3. Extract spectral features from the deformed audio and mask the spectrum to obtain the result after data augmentation;

[0065] S4. Use the result after data augmentation to train the pre-awakening model and the detection model, and save the trained pre-awakening model and detection model;

[0066] S5. Use a microphone to record an audio stream. After slicing the audio stream, input it into the trained pre-awakening model to obtain the output probability of the pre-awakening model;

[0067] S6. When the output probability of the pre-awakening model is greater than a preset threshold, or when a volume fluctuation is detected, activate the detection model for detection to obtain the output probability of the detection model;

[0068] S7. When the probability of a certain category output by the detection model is greater than a preset threshold, return the detection result.

[0069] Furthermore, the pre-awakening model in step S1 is a lightweight deep learning model composed of a multi-layer convolutional neural network. First, use 1×1 pointwise convolution for dimensionality increase. Secondly, use a 3×3 convolutional kernel for depthwise separable convolution, which can greatly reduce the computational complexity compared with traditional convolution. In addition, use a pooling operation to further reduce the data volume. Introduce a lightweight SE attention mechanism to adjust the weights of each channel. Finally, use pointwise convolution for dimensionality reduction. In addition, when the input dimension and output dimension are equal, a skip connection can be made. In addition, use an optimized activation function to reduce the computational complexity of the model.

[0070] Furthermore, the detection model in step S1 is composed of a multi-layer convolutional neural network. The Ghost module is used. By performing a simple linear transformation on a small amount of original feature data as the features of the middle layer, the bias vector is deleted to reduce the computational complexity. Use two Ghost modules to form a Ghost bottleneck (G-bneck). The first Ghost module is used to expand the number of channels, and the second Ghost module is used to reduce the number of channels. When the number of channels is the same, a skip connection can be made. After each Ghost module, there is batch normalization (Batch Normalization, BN). The second Ghost module is not activated with ReLU. In addition, when the convolution stride is 2, a depthwise separable convolution is embedded in the Ghost module.

[0071] Further, in step S2, the audio data is deformed through a series of audio transformations, and single or multiple audio data transformation operations, such as volume transformation, pitch transformation, audio stretching, and adding noise, are applied to the original audio data in a combined form. Manually set the threshold range of the parameters in the audio transformation, and randomly generate parameters for transformation. Set a lower audio sampling rate and change the audio channel to mono, and save the deformed audio.

[0072] Further, in step S3, spectral features are extracted, and the spectrum is masked. The frame length for audio framing is set to 32 ms, and the frame shift is 16 ms. After framing, windowing is performed, and the Hamming window is selected as the window function. Next, short-time Fourier transform is performed to convert the windowed audio signal from the time domain to the frequency domain. Random cropping is performed on the feature data, and random masking is performed in the frequency domain and time domain dimensions. The width and number of masks are hyperparameters manually set, and the masked value is the minimum value in that area.

[0073] Further, in step S4, the pre-wake-up model and the detection model are trained. The initial learning rate is set to 0.001, and the batch size is 64. The Adam optimizer is used, and L2 regularization is adopted, with the weight set to 5×10 -4 . In addition, after each pass through the dataset during the training process, the learning rate is adjusted to 0.85 times the current value. The cross-entropy loss function is used to calculate the loss value, and the pre-wake-up model and the detection model are trained through the backpropagation algorithm until the performance of the pre-wake-up model and the detection model reaches the best, and then they are saved.

[0074] Further, in step S6, a preset wake-up threshold is set, which is obtained by analyzing the ROC curve, and a relatively low preset wake-up threshold is manually set. When the output probability of any class of the pre-wake-up model is greater than the preset threshold, or when the instantaneous volume is 5 decibels higher than the average ambient volume in the past 60 seconds, the detection model is activated.

[0075] Further, for the preset detection threshold in step S7, the optimal threshold for each class is obtained by analyzing the ROC curve. When the output probability of a certain class of the detection model is greater than the preset threshold, the label information and timestamp of that class are returned.

[0076] Embodiment 1

[0077] Please refer to Figure 2 , Figure 2 , which is a schematic structural diagram of the audio data augmentation method used in the embodiments of the present application. The audio data augmentation method used in the embodiments of the present application mainly includes three parts. The first part is audio transformation, the second part is feature extraction, and the third part is spectrum masking.

[0078] The first part transforms the audio data through a series of audio transformations, and applies single or multiple audio data transformation operations, such as volume transformation, pitch transformation, audio stretching, and adding noise, etc., to the original audio data in a combined form.

[0079] The greater the amplitude of the sound wave, the louder the sound, and vice versa. The volume transformation changes the loudness by adjusting the amplitude of the audio. When performing the volume transformation, the gain amplitude of the original audio is randomly set between -10 dB and 10 dB.

[0080] The pitch of the sound depends on the vibration frequency. The higher the frequency, the higher the pitch. The pitch transformation adjusts the pitch of the sound. When performing the pitch transformation, the pitch of the original audio is randomly changed within the range of plus or minus 4 semitones, and the audio speed remains unchanged.

[0081] Audio stretching refers to changing the audio speed in the time dimension. During the stretching process, while keeping the pitch unchanged, the audio is randomly stretched to between 0.9 and 1.1 times its original length.

[0082] Noise is added in two different ways. One is to add white noise, and the other is real noise. When adding Gaussian white noise, white noise that conforms to the Gaussian normal distribution is used. When adding real noise, environmental noise recorded in the real world is used. The decibel value of the noise is multiplied by a random coefficient and added to the original audio, and the random coefficient ranges from 0.1 to 0.3.

[0083] The second part extracts the log Mel spectrogram from the transformed audio, and the process is as follows: First, the audio is framed, the frame length is set to 32 ms, and the frame shift is 16 ms. After framing, windowing is performed, and the Hamming window is selected as the window function. The formula is as follows:

[0084]

[0085] Next, the short-time Fourier transform is performed to convert the windowed audio signal from the time domain to the frequency domain, and then the frequency scale is converted to the Mel scale. The conversion process is as follows:

[0086]

[0087] Filtering is performed using 128 equally - area triangular filters on the Mel scale to obtain the Mel spectrogram features.

[0088] The third part is spectral masking. Random cropping is performed on the Mel spectrogram in the time domain in the training set. The cropping length is 128, and the training feature data with a size of (128, 128) is obtained. Random cropping can effectively improve the diversity of training data, making the training data used in each iteration different, thus avoiding overfitting. Then, random masking is performed in the frequency domain and time domain dimensions. The upper limit of the masking width is set to 3, and the width is randomly taken from 1 to 3. The number of masks is set to 7, and the masked value is the minimum value in this area. Then, the Mel spectrogram is logarithmically mapped to the db scale as shown in formula (3) to obtain the logarithmic Mel spectrogram.

[0089] y = 20 * log 10 (x) (3)

[0090] After that, normalization is performed on the data of each frequency domain dimension respectively to obtain the result of data augmentation, which is used as the input of the model.

[0091]

[0092] Where is the value at time point i, μ is the average value, and σ is the standard deviation.

[0093] Example 2

[0094] Please refer to Figure 3 , Figure 3 , which is a schematic diagram of the training process of the deep learning model in the real-time sound event detection method. First, a deep learning model is established, including a pre-awakening model and a detection model.

[0095] The overall structure of the pre-awakening model is similar to Google MobileNetV3-Small (MobNet-S). MobNet-S is a lightweight convolutional neural network model for image classification published by Google in 2019. It has stronger performance and lower computational complexity than MobileNetV2. In the ImageNet classification task, MobNet-S has an accuracy about 3.2% higher than MobileNetV2 and a reduced operation time of 15%. MobNet-S uses pointwise convolution for dimensionality increase. Secondly, depthwise separable convolution is performed using a 3×3 convolutional kernel, which can significantly reduce the computational amount compared to traditional convolution. Pooling is used to further reduce the data volume. A lightweight attention mechanism is introduced to adjust the weights of each channel. Finally, pointwise convolution is used for dimensionality reduction. In addition, when the input dimension and output dimension are equal, skip connections can be made.

[0096] Skip connections were first proposed by He Kaiming et al. The formula for skip connections is shown in equation (5). It creates a shortcut between convolutional layers, effectively avoiding the gradient vanishing and network degradation problems existing in deep convolutional neural networks.

[0097] H(x) = F(x) + x (5)

[0098] Where F(x) is the weight of the convolutional layer. When F(x) = 0, an identity mapping H(x) = x is formed, which helps the model determine which convolutional layers belong to the redundant part and mask them.

[0099] MobNet-S also uses the ReLU6 activation function to replace the traditional ReLU function, which enables it to be better fixed-point quantized on mobile devices using float8. The following is the formula for ReLU6:

[0100] ReLU6(x) = min(max(0, x), 6) (6)

[0101] Due to the highly non-linear characteristics of audio features, the performance of traditional activation functions such as ReLU and Sigmoid will decline when the number of neural network layers is relatively deep. The Swish activation function has higher non-linearity, but its computational complexity is relatively high. The formula for the Swish function is as follows:

[0102] Swish = x · σ(x) (7)

[0103] Where σ is the Sigmoid function, and the formula is as follows:

[0104]

[0105] To simplify the computational complexity, the hard-swish (hswish) activation function is used to replace the swish activation function, and the Sigmoid function in it is replaced by the Hard-Sigmoid function, reducing the computational complexity. The formula for hswish is as follows:

[0106]

[0107] However, there are significant differences between audio feature data and image feature data. Therefore, some adjustments have been made to the MobNet-S model applied to image classification. The input channels of the model are changed from the original three channels to a single channel. In addition, the last 5 convolutional layers are removed, reducing the number of model parameters. Since MobNet-S was originally used for a 1000-class image classification task, the number of output channels of the penultimate layer is changed from 1024 to 50, and the output of the last one-dimensional convolutional layer is changed to the number of classification categories. The adjusted model is called MobileNet-Tiny (MobNet-T). The overall structure of the model is shown in Table 1:

[0108]

[0109] Table 1 MobNet-T Structure Table

[0110] Where Input is the input size, Operator is the convolution operation, exp size is the dilation dimension, out is the output dimension, SE represents Squeeze-Excite, NL represents the non-linear activation function, Stride is the convolution stride, AvgPool is the average pooling, and k is the number of categories for the final classification of the model.

[0111] The Softmax activation function is used to activate and classify the output results of the model. The formula is as follows:

[0112]

[0113] Where z i is the output value of the i-th node, and C is the number of output nodes, that is, the number of categories for classification.

[0114] The cross-entropy loss function is used as the loss function. The formula is as follows:

[0115]

[0116] The detection model is trained using the GhostNet structure. GhostNet is a lightweight network released by Huawei in 2020. Its performance on the CIFAR image recognition dataset exceeds that of MobileNetV3-Large (MobNet-L), and it also has higher computing efficiency.

[0117] Most of the current mainstream lightweight networks use depthwise separable convolutions or pointwise convolutional neural networks. Although they simplify the computational complexity compared to ordinary CNNs, they still occupy a large amount of memory and floating-point operations per second. In addition, traditional convolutional neural networks are prone to generating many similar features in the intermediate layers, forming redundant data, and even reducing the performance of the model to a certain extent. GhostNet proposes a new type of Ghost module to solve this problem. The Ghost module uses a small amount of original feature data after simple linear transformation as the features in the intermediate layer. In addition, the bias vector is removed to reduce the computational complexity. The convolution process is shown in Equation (12). The linear transformation process is shown in Equation (13):

[0118] Y′ = X * f′ (12)

[0119] Where f′ is the convolution kernel. Padding is used to keep the data size unchanged.

[0120]

[0121] Where Φ is the linear operation

[0122] In addition, GhostNet is improved based on the bottleneck, which is called Ghost bottleneck (G-bneck). G-bneck is mainly composed of two stacked Ghost modules. The first Ghost module is used to expand the number of channels, and the second Ghost module is used to reduce the number of channels. When the number of channels is the same, skip connections can be made. After each Ghost module, batch normalization (BN) is accompanied, and ReLU is not used for activation after the second Ghost module. Additionally, when the convolution stride is 2, a depthwise separable convolution is embedded in the Ghost module. Similar to MobNet-T, the input channels of GhostNet are adjusted from three channels to a single channel. The GhostNet structure is shown in Table 2.

[0123]

[0124]

[0125] Table 2 GhostNet Structure Table

[0126] Among them, FNN is a feedforward neural network, and k is the number of categories for the final classification of the model.

[0127] The hyperparameters used for training the pre-awakening model and the detection model are the same. The initial learning rate is set to 0.001, and the batch size is 64. The Adam optimizer is used, and L2 regularization is adopted with the weight set to 5×10 -4 . The formula is as follows:

[0128]

[0129] where E e is the error between the training data labels, and w i represents the weight value.

[0130] In addition, an approach of adjusting the learning rate at intervals is adopted. After each traversal of the training set data, that is, after 1 Epoch, the current learning rate lr will be multiplied by the decay coefficient γ. The formula is shown in Equation (15), where γ is set to 0.8 and the step size is set to 1.

[0131] new lr = initial lr ×γ epoch / / stepsize (15)

[0132] The cross-entropy loss function is used to calculate the loss value. The parameters of the pre-awakening model and the detection model are updated by the backpropagation algorithm in the way of gradient descent until the prediction accuracy of the pre-awakening model and the detection model reaches the best, and then the model is saved. The calculation process of the accuracy is as follows:

[0133]

[0134] Among them, TP represents the number of correctly predicted positive example samples, FP represents the number of incorrectly predicted positive example samples, TN represents the number of correctly predicted negative example samples, and FN represents the number of incorrectly predicted negative example samples.

[0135] The performance of different models is compared by looking at the accuracy of the models on the ESC-10 dataset. ESC-10 is a small environmental sound classification dataset made by Harvard University. It contains 10 sound categories, such as barking dogs, rain sounds, ocean wave sounds, etc. Each category has 40 samples, for a total of 400 samples, with a total duration of 33 minutes. The length of each audio sample is about 5 seconds. The dataset is pre-segmented into five folds, where the training set accounts for 80% and the test set accounts for 20%. That is, 32 samples per category are used for training and 8 samples are used for testing. The ESC dataset has only 40 samples per category, which is more in line with practical applications.

[0136] To ensure the accuracy of the experimental data, five-fold cross-validation is performed using the preset segmentation method of the dataset. Five different data segmentation forms are used for training and testing respectively, and the average value is taken to obtain the final average prediction accuracy.

[0137] In the development and model training stages, a platform with a relatively high configuration was used. The experimental environment is as follows: The operating environment is Windows 10 pro, the CPU is AMD Ryzen 2700 @ 3.2 GHz, the GPU uses Nvidia Geforce GTX1070Ti 8G VRAM, the memory uses dual-channel 32g 3000MHz ddr4 c18, 512G NVMe M.2 and 1TB 7200rpm HDD. The deep learning development environment is Pytorch 1.9.0 + cuda11.2, and the toolkits used are librosa 0.9.0, soundfile 0.10.3, pyaudio 0.2.11, numpy 1.20.3, and the development tool is Pyhcharm.

[0138] During the testing process, a laptop with a relatively low configuration was selected to simulate the performance of a mobile phone. The experimental environment is as follows: The operating environment is Windows 10, the CPU is Intel Core i3 380M

[0139] @2.53Hz, integrated graphics, the memory uses dual-channel 4g 1066MHz ddr3, the hard disk is 256G HDD, and the software configuration is the same as that of the development platform.

[0140]

[0141] Table 3 Comparison Table of Comprehensive Model Performance

[0142] Table 3 is the comparison table of the comprehensive performance of the model. The results of the comparative experiment show that using data augmentation can greatly improve the performance of the model, and the accuracy rate on different models has increased by 9-16%. MobNet-T has the largest performance improvement after data augmentation, with an increase of about 16%. The performance of the enhanced MobNet-T exceeds that of the initial GhostNet. Followed by GhostNet with a performance improvement of about 15%. MobNet-S and MobNet-L have relatively similar performances, both increasing by about 11.5%. The improvement of ResNet-18 is slightly smaller because its initial starting point is already very high, reaching 82.91%.

[0143] Since this algorithm needs to promptly feedback the detected sound events, in addition to the performance of the model, the operation efficiency is also a very important indicator. Check the time consumed by the model to calculate a feature of size (128, 128) once to judge its computational complexity. In addition, considering the use of the algorithm on mobile devices such as mobile phones or single-chip microcomputers, the number of model parameters and the parameter size also need to be taken into account. The comprehensive performance of the model is shown in Table 3, where the input channels of the baseline models are all single channels, and the inputs are all (1, 128, 128). ResNet-18 ranks the highest among the comparison models, with an accuracy rate of 91.67%, but its model calculation amount is also the largest, and the time consumption is close to 1 s. GhostNet ranks second, at 88.03%, but its calculation amount is much lower than that of ResNet-18, only 41% of ResNet. The performance difference between MobNet-S and MobNet-L on the dataset is small, only differing by 2.34%. The MobNet-T proposed in the present invention has an accuracy rate of 73.97%. Although it is somewhat lower than the other comparison models to a certain extent, it has greatly reduced the number of parameters and the calculation time. It has reduced the number of parameters and the calculation time by about 80% compared to MobNet-L. The wake-up model Siri of the Apple mobile phone is a multi-layer perceptron composed of a five-layer feedforward neural network with 192 nodes. Although its model complexity is low and the calculation time is almost the same as that of MobNet-T, its performance is too low to have practical use value.

[0144] Please refer to Figure 4 , Figure 4The following is a comparison chart of the effects of the audio data augmentation method according to the present invention. Experimental results show that the use of spectral masking reduces the fluctuation of the model's accuracy on the test set. Although it cannot significantly improve the performance of the model, it reduces the variance of the model's accuracy on different folds and improves the stability of the model. Audio transformation uses a series of combined transformations to amplify the data to 15 times the original size, which increases the accuracy by approximately 16%. Of course, if more types of changes are made, the model performance may be further improved. In summary, the combination of audio transformation and spectral masking can achieve the best performance of the model, while also improving the model's robustness and generalization capabilities.

[0145] Example 3

[0146] See also Figure 5 , Figure 5 It is a structural diagram of the real-time sound event detection method according to the present invention. First, an external microphone is used to record the audio stream. The microphone adopts Rode VideoMicro, and the sound card adopts the built-in sound card of the laptop. Use Pyaudio to record the mono audio stream, and the sampling rate is set to 16kHz. The audio stream is divided into a segment every 2s. The segment slides once every 0.5s, and the segment is input into the model for detection. The time spent on each part of the detection method was checked, among which the time spent on extracting audio features and data preprocessing is 46.88ms, the time required for pre-wake-up model detection is 93.75ms, the time spent on the detection model is 351.06ms, and the total time spent on the algorithm theory is 491.69ms, about 0.5 seconds. The actual time will fluctuate to a certain extent due to factors such as hardware performance and working conditions.

[0147] There are two prerequisites for activating the detection model. One is that when the probability value α of any category in the output result of the pre-wake-up model is greater than the preset probability threshold value τ, it is considered that a possible sound event is detected. Or when the volume of the instantaneous audio in the environment is 5 decibels higher than the average volume of the environment in the past 60 seconds. Similarly, the probability threshold value θ of the detection model is set, and the value of θ for each classification category is different. When the output probability β of the detection model is greater than θ, it is considered that the corresponding sound event is detected, and the current classification result and timestamp will be returned. The pseudo code of the real-time sound event detection method is shown in Algorithm 1:

[0148]

[0149]

[0150] The probability threshold values τ and θ in the method are both obtained by analyzing the ROC curve. Among them, a lower τ value is set to improve the recall rate of the pre-awakening model, while the θ value is selected as the value when the accuracy is the best. When the number of classification categories is large, the over-strong generalization ability of the model will lead to an increase in the recall rate, and the false awakening rate will also increase accordingly. If you want to reduce the recall rate, you can abandon the data augmentation module, replace the one-dimensional convolutional layer in the model with an FNN, and then train the model to overfitting, which can greatly reduce the recall rate of the model.

[0151] The number of devices and the processing scale described here are used to simplify the description of the present invention, and the application, modification, and variation of the present invention are obvious to those skilled in the art.

[0152] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily achieved. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrated and described examples here.

Claims

1. A method for audio data augmentation and a real-time sound event detection method, characterized in that it includes a detection system, the detection system includes a deep learning model, the deep learning model includes a processing module, a pre-awakening module and a detection module, the processing module is used to deform audio data, extract spectral features from the deformed audio, and the processing module is used to perform audio conversion on the original audio data, extract the spectral features of the converted audio and perform random spectral masking; The pre-awakening module and the detection module need to be trained respectively. After the pre-awakening module is trained, it is saved, and the pre-awakening module passes through a lightweight deep learning model composed of a multi-layer convolutional neural network. First, 1×1 pointwise convolution is used for dimensionality increase; secondly, 3×3 convolutional kernels are used for depthwise separable convolution, and pooling operations are used to further reduce the data volume; a lightweight SE attention mechanism is introduced to adjust the weights of each channel, and pointwise convolution is used for dimensionality reduction; in addition, when the input dimension and the output dimension are equal, skip connections can be made; After the detection module is trained, it is saved. The detection module is composed of a multi-layer convolutional neural network; by performing a simple linear transformation on a small amount of original feature data as the features of the middle layer, the bias vector is deleted to reduce the computational complexity; two Ghost modules are used to form a Ghost bottleneck (G-bneck). The first Ghost module is used to expand the number of channels, and the second Ghost module is used to reduce the number of channels. When the number of channels is the same, skip connections can be made; after each Ghost module, there is batch normalization (Batch Normalization, BN), and the second Ghost module is not activated using ReLU; in addition, when the convolution stride is 2, a depthwise separable convolution is embedded in the Ghost module; The method specifically includes the following steps: S1. Establish a deep learning model, including a pre-awakening module and a detection module; S2. Deform the audio data through a series of audio transformations to obtain the deformed audio; S3. Extract spectral features from the deformed audio and mask the spectrum to obtain the result after data augmentation; S4. Use the result after data augmentation to train the pre-awakening module and the detection module, and save the trained pre-awakening module and detection module; S5. Record an audio stream using a microphone, slice the audio stream, and input it into the trained pre-awakening module to obtain the output probability of the pre-awakening module; S6. When the output probability of the pre-awakening module is greater than a preset awakening threshold, or when a volume fluctuation is detected, activate the detection module for detection to obtain the output probability of the detection module; S7. When the probability of a certain category output by the detection module is greater than a preset detection threshold, return the detection result.

2. The data augmentation method and real-time sound event detection method for audio according to claim 1, characterized in that, Both the pre-awakening module and the detection module are lightweight convolutional neural network models, composed of a multi-layer convolutional neural network, and the model parameters are optimized using the backpropagation algorithm.

3. The data augmentation method for audio and the real-time sound event detection method according to claim 2, wherein In step S2, the series of audio transformations include applying single or multiple audio data transformation operations, such as volume transformation, pitch transformation, audio stretching, and adding noise, etc., to the original audio data in a combined form.

4. The data augmentation method and real-time sound event detection method for audio according to claim 3, characterized in that, In step S3, the spectral features use Mel spectral features. The masking of the spectrum is performed by double random masking in both the frequency domain and the time domain. The width and number of the masks are hyperparameters set manually. The masked value is the minimum value of the Mel spectral features in the area where the masking operation is located. The logarithm of the masked Mel spectrum is taken to obtain the logarithmic Mel spectral features.

5. The data augmentation method and real-time sound event detection method for audio according to claim 4, characterized in that, The training process of the pre-wake-up module and the detection module in step S4 includes setting a relatively small learning rate, a medium-sized batch size, and using an adaptive optimizer. L2 regularization is adopted to prevent overfitting. The learning rate is decayed by adjusting the learning rate at intervals. The cross-entropy loss function is used to calculate the loss value. The pre-wake-up module and the detection module are trained through the backpropagation algorithm until the performance of the pre-wake-up module and the detection module reaches the best, and they are saved.

6. A method for data augmentation of audio and a method for real-time sound event detection according to claim 5, characterized in that, In step S5, using a microphone to record an audio stream means using a microphone to record a mono audio stream at a relatively low sampling rate. The slicing of the audio stream means slicing the audio stream into a segment every few seconds, and the segment slides every few seconds to obtain real-time audio segments.

7. A method for data augmentation of audio and a method for real-time sound event detection according to claim 1, characterized in that, The preset wake-up threshold in step S6 is obtained by analyzing the receiver operating characteristic curve. When the output probability of any category of the pre-wake-up module is greater than the preset threshold, the detection module will be activated.

8. A method for data augmentation of audio and a method for real-time sound event detection according to claim 1, characterized in that, The preset detection threshold in step S7 sets the threshold for each category separately. When the output probability of a certain category of the detection module is greater than the preset threshold, the label information and timestamp of that category are returned.

Images