Sound event classification method based on frequency band division to reduce reverberation

Abstract

The application provides a sound event classification method based on frequency band division and reverb reduction, which can effectively reduce the influence of background noise such as reverb and improve the accuracy of sound classification and recognition. The method converts the to-be-processed audio signal from the time domain to the frequency domain to obtain a to-be-processed frequency spectrum signal, controls the frequency band width of the to-be-processed frequency spectrum signal through a frequency ratio, reduces the reverb of the selected frequency band, obtains a frequency spectrum signal after reverb reduction, combines the frequency spectrum signal with other signals to obtain a final output frequency spectrum signal after reverb reduction, and converts the frequency spectrum signal into a time domain signal for output; and based on the final output time domain signal, a data enhanced training set is formed to train a sound event classification model, thereby indirectly enhancing the feature representation capability of the sound event classification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sound event classification, in particular to a sound event classification method based on frequency band reduction of reverberation. BACKGROUND

[0002] In the natural environment, vision, hearing and other senses are important for people to perceive the surrounding environment. Through numerous senses, people can obtain rich information about the surrounding environment. At present, with the development of deep learning technology, people's research on machine vision is becoming more and more perfect, and corresponding sound-related events have been studied in various directions. Sound event classification refers to the classification of sounds in daily environment and surroundings. Sound event classification research is the key to developing intelligent applications in monitoring systems, robot navigation systems, audio multimedia search and other fields. Many current models can easily identify voiceprint information and semantic information, such as the patent with application number CN201910539745.8, which discloses a city sound event classification method. However, in practical applications, when identifying events in complex scenes in reality, the accuracy of sound event classification is not high due to reverberation, echo and other background noise. SUMMARY

[0003] In order to solve the problem that the sound classification method in the prior art has low accuracy when applied in complex scenes due to reverberation, echo and other background noise, the present application provides a sound event classification method based on frequency band reduction of reverberation, which can effectively reduce the influence of reverberation and other background noise and improve the accuracy of sound classification and recognition.

[0004] The technical solution of the present application is as follows: a sound event classification method based on frequency band reduction of reverberation, comprising the following steps:

[0005] S1: collecting original audio sample information and performing preprocessing operation to obtain preprocessed audio signal to be processed;

[0006] characterized in that it further comprises the following steps:

[0007] S2: converting the audio signal to be processed into frequency domain from time domain S(t) through short-time Fourier transform to obtain the frequency spectrum signal S(t,ω) to be processed, wherein t is time and ω is frequency;

[0008] S3: performing frequency component analysis on the frequency spectrum signal S(t,ω) to be processed to obtain the highest frequency ω h and the lowest frequency ω l of the signal, and further obtain the input signal bandwidth B i corresponding to the frequency spectrum signal S(t,ω) to be processed;

[0009] Bi = ω h - ω l ;

[0010] S4: setting a frequency division ratio r based on the collection scene of the original audio sample information, selecting a frequency band signal and controlling the output bandwidth B of the selected frequency band o ;

[0011] B o = B i *r;

[0012] S5: based on the output bandwidth B0 of the selected frequency band, sending the to-be-processed frequency spectrum signal S(t, ω) into a Butterworth filter, based on a Butterworth frequency domain response, obtaining a Butterworth amplitude function |H(ω)| in the frequency domain and a selected frequency spectrum signal S1(t, ω) corresponding to B0;

[0013]

[0014] wherein N is an order, ω c is a cutoff frequency;

[0015] S6: weakening reverberation of the selected frequency spectrum signal S1(t, ω) by using a weighted prediction error algorithm WPE to obtain a frequency spectrum signal output S2(t, ω) after reducing reverberation;

[0016]

[0017] wherein, obtained by a prediction filter estimation in WPE, H is a conjugate transpose operator, D is a prediction delay, D divides a reverberation environment into early reverberation and late reverberation, is a late reverberation part;

[0018] S7: subtracting the selected frequency spectrum signal S1(t, ω) from the to-be-processed frequency spectrum signal S(t, ω) to obtain other signals S3(t, ω);

[0019] combining S2(t, ω) and the other signals S3(t, ω) to obtain a final output frequency spectrum signal S4(t, ω) after reducing reverberation, and then using an inverse short-time Fourier transform to convert the signal S4(t, ω) from the frequency domain to the time domain to obtain S4(t);

[0020]

[0021] S8: constructing a frequency domain residual shrinkage network F-DRSN based on a deep residual shrinkage network DRSN;

[0022] The frequency domain residual shrinkage network F-DRSN comprises, in sequence, a convolutional layer, a residual shrinkage unit, global average pooling, and a fully connected layer; wherein the number of residual shrinkage units is greater than 1; each residual shrinkage unit comprises a frequency domain self-correction algorithm;

[0023] S9: constructing a sound event classification model based on the frequency domain residual shrinkage network F-DRSN; training the sound event classification model to obtain a trained sound event classification model;

[0024] S10: obtaining sound signal data to be detected, extracting acoustic features from the sound signal data to be detected to obtain acoustic features to be recognized;

[0025] The acoustic features to be recognized are input into the trained sound event classification model to obtain a final classification result.

[0026] It is further characterized in that:

[0027] In step S5, the order N and the cutoff frequency ω c are calculated as follows:

[0028]

[0029] wherein δ1 is the maximum attenuation in the passband, and δ2 is the minimum attenuation in the stopband;

[0030] ω p is the passband cutoff frequency, ω s is the stopband cutoff frequency, ω p and ω s are calculated as follows:

[0031]

[0032] When the filtered frequency band is a high frequency band, ω p is denoted as ω s is denoted as When the filtered frequency band is a low frequency band, ω p is denoted as ω s is denoted as

[0033] ω h is the highest frequency of the spectrum signal S(t, ω) to be processed, ω l is the lowest frequency of the spectrum signal S(t, ω) to be processed; B o is the output bandwidth of the selected frequency band;

[0034] and Bandwidth of the high-frequency transition band and bandwidth of the low-frequency transition band, respectively;

[0035] The residual shrinkage unit comprises a frequency domain self-correction module, two convolution layers and a soft thresholding module connected in sequence, and an attention module, the output of the last convolution layer is sent to the soft thresholding module and the attention module to extract typical features in the channel, and then sent to the soft thresholding module to remove noise;

[0036] The operation in the frequency domain self-correction module comprises:

[0037] a1: compress the frequency domain dimension of the input feature x by MaxPooling;

[0038] a2: extract features from the compressed result by using two convolution layers;

[0039] a3: perform up-sampling to restore the size to the same size as the initial input feature, and sum the extracted frequency domain features and the original feature map to obtain the output feature x1;

[0040] a4: compress x1 into a one-dimensional vector by global average pooling, and obtain the recalibrated result x2 through a fully connected layer;

[0041] a5: multiply the result of the original feature passing through two convolution layers with x2 to obtain the final output x3;

[0042] In step S9, the sound classification model is trained, specifically comprising the following steps:

[0043] b1: select a data set with original audio data samples, divide the data set to obtain a training set and a test set;

[0044] b2: calculate the time domain signal S4(t) after frequency band reduction and reverberation reduction corresponding to the training set;

[0045] b3: fully mix the original time domain signal S(t) and the time domain signal S4(t) after frequency band reduction and reverberation reduction corresponding to the training set to form a data enhanced training set;

[0046] b4: extract acoustic features from the enhanced training set to obtain: log-mel spectrum, gamma pass spectrum and gamma pass frequency cepstral coefficient, and record the three acoustic features as: training acoustic feature set;

[0047] b5: send the training acoustic feature set into the sound event classification model to train the sound event classification model, and obtain the trained sound event classification model;

[0048] b6: extracting acoustic features from the data of the test set respectively, obtaining: log-mel-spectrogram, gammatone-spectrogram and gammatone-frequency cepstral coefficient, and recording the three acoustic features as: verification acoustic feature set;

[0049] b7: sending the verification acoustic feature set into the trained sound event classification model, and judging the model training result according to the classification result;

[0050] In step S10, the method for extracting acoustic features from the to-be-detected sound signal data is as follows:

[0051] c1: confirming the classification result obtained by sending the verification acoustic feature set into the trained sound event classification model, and finding the acoustic feature with the highest classification accuracy, recorded as: recognition acoustic feature;

[0052] c2: extracting the recognition acoustic feature from the to-be-detected sound signal data, recorded as: to-be-recognized acoustic feature;

[0053] In step S1, the original audio sample information includes: audio duration and sampling frequency.

[0054] The preprocessing operation includes: pre-emphasis, framing and windowing.

[0055] The application provides a sound event classification method based on frequency band division and reverberation reduction. After processing the to-be-processed audio signal, the method converts from time domain S(t) to frequency domain to obtain a to-be-processed frequency spectrum signal S(t, ω). The to-be-processed frequency spectrum signal is controlled by a frequency band width through a frequency division ratio r, and the reverberation of the selected frequency band is reduced to obtain a frequency spectrum signal output S2(t, ω) after reverberation reduction. Then, S2(t, ω) and other signals S3(t, ω) are combined to obtain a final output frequency spectrum signal S4(t, ω) after reverberation reduction, which is then converted into a time domain signal S4(t). Based on the final output time domain signal S4(t), a data enhanced training set is formed, a sound event classification model is trained, and a trained sound event classification model is obtained. The method uses the frequency band division and reverberation reduction method to calculate the final output signal after reverberation reduction, effectively reduces the reverberation and other background noise in the original audio, and as much as possible reduces the sound distortion and improves the intelligibility of the overall speech, thereby indirectly enhancing the feature representation ability of the sound event classification model, and further improving the classification accuracy of the sound event classification model. At the same time, the frequency domain self-correction algorithm is introduced into the DRSN model to construct a sound event classification model. The frequency domain self-correction module improves the attention to the frequency domain features of the input sound data, fully extracts the frequency domain features of the sound data, and combines the final output time domain signal S4(t) obtained after the frequency band division and reverberation reduction method, to further improve the classification accuracy of the sound event classification model. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The schematic diagram of the principle of reducing reverberation by frequency band in the application;

[0057] Figure 2 The schematic diagram of the network structure of the sound event classification model in the application;

[0058] Figure 3 The schematic diagram of the network structure of the frequency domain self-correction module in the application. DETAILED DESCRIPTION

[0059] As Figure 1 shown, the application includes a sound event classification method based on reducing reverberation by frequency band, which includes the following steps:

[0060] S1: Collecting original audio sample information, performing preprocessing operation, improving signal quality, and obtaining preprocessed audio signal to be processed.

[0061] The original audio sample information includes audio duration and sampling frequency. The preprocessing operation includes pre-emphasis, framing and windowing; wherein the pre-emphasis enhances the high frequency components in the speech signal, the framing processes the signal into a stationary signal, and the windowing reduces the spectral leakage problem caused by the signal stage.

[0062] S2: The audio signal to be processed is converted into frequency domain from time domain S(t) by short-time Fourier transform, and the signal to be processed is obtained. S(t, ω), wherein t is time, and ω is frequency.

[0063] S3: Frequency component analysis is performed on the signal to be processed S(t, ω) to analyze various frequency components in the signal, and the highest frequency ω h and the lowest frequency ω l in the signal are obtained, and then the input signal bandwidth B i corresponding to the signal to be processed S(t, ω) is obtained.

[0064] B i = ω h - ω l .

[0065] S4: Based on the collection scene of the original audio sample information, set the frequency division ratio r, select the frequency band signal and control the output bandwidth B o of the selected frequency band.

[0066] B o = B i *r.

[0067] In specific implementation, the setting of the frequency division ratio is obtained through historical experience or through experimental data test according to the sound collection scene.

[0068] S5: Based on the output bandwidth B0 of the selected frequency band, the spectrum signal S(t,ω) to be processed is sent into the Butterworth filter. Based on the Butterworth frequency domain response, the selected spectrum signal S1(t,ω) corresponding to the Butterworth amplitude function |H(ω)| and B0 in the frequency domain is obtained.

[0069]

[0070] Where N is the order, ω c This is the cutoff frequency.

[0071] Depending on the original audio acquisition scenario, the resulting spectral signal after Butterworth filter processing may be either a high-frequency or low-frequency signal. For example, sound data acquired in an indoor environment contains a majority of low-frequency signals, while signals acquired in an outdoor urban environment are more likely to be classified as high-frequency signals. This method, based on the setting of the division ratio r, uses a Butterworth filter to achieve the effect of dividing the spectral signal to be processed into frequency bands. Figure 1 In the middle, S h1 (t,ω) indicates that the Butterworth filter produces a high-frequency signal, S l1 (t,ω) indicates that the Butterworth filter produces a low-frequency signal. Similarly, S h3 (t,ω) and S l3 (t, ω) represent the values ​​based on S. h1 (t,ω) and S l1 Other high-frequency signals and other low-frequency signals are obtained by (t,ω) calculation.

[0072] In step S5, the order N and cutoff frequency ω are known based on existing technology. c The calculation method is as follows:

[0073]

[0074] Where δ1 is the maximum attenuation in the passband and δ2 is the minimum attenuation in the stopband. In practical applications, δ1 and δ2 are set based on experiments; δ1 is generally smaller to ensure signal quality within the passband, while δ2 is generally larger to suppress interference signals.

[0075] ω p ω is the passband cutoff frequency. s ω is the stopband cutoff frequency; p to ω s The frequency band between ω and ω is a transition band. p and ω s The calculation method is as follows:

[0076]

[0077] wherein, when the filtered frequency band is the high frequency band, ω p is denoted as ω s is denoted as wherein, when the filtered frequency band is the low frequency band, ω p is denoted as ω s is denoted as ω h is the highest frequency of the to-be-processed spectral signal S(t, ω), and ω l is the lowest frequency of the to-be-processed spectral signal S(t, ω); B o is the output bandwidth of the selected frequency band; wherein, and are the bandwidth of the high frequency band transition band and the bandwidth of the low frequency band transition band, respectively. In a specific implementation, and are selected according to actual scene requirements; for example, in the embodiment, in the high frequency band filtering scene, a larger transition band range is expected, and may be set to 1000 Hz, or in the low frequency band filtering scene, may be set to 300 Hz; after the passband cutoff frequency ω p and the stopband cutoff frequency ω s of the high or low frequency band are determined, the cutoff frequency ω c and the order N can be obtained.

[0078] S6: The selected spectral signal S1(t, ω) is subjected to a dereverberation operation using a weighted prediction error algorithm WPE to obtain a dereverberated spectral signal output S2(t, ω).

[0079]

[0080] wherein, is estimated by a prediction filter in the WPE, H is a conjugate transpose operator, and D is a prediction delay, which divides the reverberation environment into early reverberation and late reverberation, is the late reverberation part. Figure 1 wherein, S h2 (t, ω) and S l2 (t, ω) represent the dereverberated spectral signal outputs calculated based on the high frequency band signal and the low frequency band signal, respectively.

[0081] The voice signal often contains three parts, which are direct signal, early reverberation and late reverberation. Since early reverberation has a positive effect on voice intelligibility, but late reverberation reduces voice intelligibility, it is often focused on eliminating late reverberation. Therefore, the S1(t,ω) signal is reduced by using the weighted prediction error method (WPE) to reduce late reverberation, and a high-frequency reduced or low-frequency reduced spectral signal output S2(t,ω) of the S1(t,ω) signal can be obtained.

[0082] S7: subtract the selected spectral signal S1(t,ω) from the to-be-processed spectral signal S(t,ω) to obtain other signal S3(t,ω); combine the spectral signal S2(t,ω) after reducing reverberation and the other signal S3(t,ω) to obtain the final output signal S4(t,ω) after reducing reverberation, and then convert the signal from the frequency domain to the time domain by using the inverse short-time Fourier transform to obtain the time-domain output S4(t); this method effectively reduces the reverberation and other background noise of high frequency or low frequency in the original audio, causes less damage to the sound quality, and enhances the intelligibility of the voice at the same time;

[0083]

[0084] That is, the time-domain output signal S4(t) is obtained by removing the reverberation data in the specified frequency band from the to-be-processed time-domain signal S(t), Figure 1 wherein S h4 (t) and S l4 (t) represent the final output signals calculated based on the high-frequency band signal and the low-frequency band signal, respectively.

[0085] S8: based on the deep residual shrinkage network DRSN, a frequency domain self-correction algorithm is introduced to construct a frequency domain residual shrinkage network F-DRSN;

[0086] As shown in Figure 2 , the frequency domain residual shrinkage network F-DRSN includes a convolutional layer Conv, a residual shrinkage block Residual Shrinkage Block, a global average pooling Global Average Pooling and a fully connected layer FC connected in sequence, and the fully connected layer FC further includes a dropout function and a softmax function (not labeled in the figure) connected in sequence; the input is the acoustic feature Sound Feature of the to-be-detected sound signal data, and the output is the sound classification result SoundClassification.

[0087] In the frequency domain residual shrinkage network F-DRSN, the first convolutional layer Conv is a Conv2D, the size of the convolution kernel is 3*3, the number of convolution kernels is 128, and the initial features are extracted. The fully connected layer FC is processed by global average pooling to pool the multi-dimensional array after the convolution operation into a one-dimensional array, and the Dropout function is connected to prevent overfitting. Finally, the sound event classification model outputs the classification prediction through the softmax function, and the calculation formula of the softmax function is:

[0088]

[0089] Si represents the softmax function, i is a positive integer, and Zi is the output value of the fully connected layer.

[0090] The number of residual shrinkage units included in the frequency domain residual shrinkage network F-DRSN is greater than 1, Figure 2 In the embodiment shown, three residual shrinkage units are included, and each residual shrinkage unit is followed by a BN+Relu layer. The frequency domain self-correction algorithm is introduced in the residual shrinkage unit in the present application, which enhances the ability of the model to collect frequency domain information of the input feature Sound Feature.

[0091] The residual shrinkage unit includes: a frequency domain self-correction module, two convolutional layers and a soft thresholding module connected in sequence, and an attention module, the output of the last convolutional layer is sent to the soft thresholding module, and at the same time is sent to the attention module to extract typical features in the channel, and then is sent to the soft thresholding module to remove noise. In the attention module, Abs represents taking the absolute value, GAP represents global average pooling, FC represents the fully connected layer, Maxpooling represents the maximum pooling, and Sigmoid and Softmax are activation functions.

[0092] As shown in Figure 3 The operations in the frequency domain self-correction module include:

[0093] a1: compress the frequency domain dimension of the input feature x by MaxPooling;

[0094] a2: extract features from the compressed result by using two convolutional layers, effectively extracting frequency domain features;

[0095] a3: upsample UpSampling to restore the size to the same size as the initial input feature, and sum the extracted frequency domain features and the original feature map to obtain the output feature x1;

[0096] a4: compress x1 into a one-dimensional vector by global average pooling, and obtain the recalibrated result through the fully connected layer, denoted as x2;

[0097] a5: the result of the original feature through two convolution layers is multiplied by x2 to obtain the final output x3;

[0098] In the present application, the frequency domain self-correction algorithm is introduced into the deep residual shrinkage network DRSN to form F-DRSN, thereby enhancing the attention of the classification model to the frequency domain; the frequency domain self-correction can more effectively extract the context information in the frequency domain by compressing the frequency dimension, and the residual structure and the channel attention are used to aggregate the effective information in the shallow features and the deep features, and the typical features in the channel are extracted according to the channel attention map.

[0099] The structure of the convolution layer in the frequency domain self-correction module is: Conv2D+BatchNormalization+Relu, the size of the convolution kernel is 3*3, and the number of the convolution kernel is 128. The soft thresholding is to put the soft threshold denoising method into the deep learning network, so that the network adaptively adjusts the threshold of the potential noise in the input signal, and reduces the more obvious noise signal to close to 0, successfully eliminating most of the noise features in the input signal.

[0100] The soft threshold function can be expressed as follows:

[0101]

[0102] Where |x| is the coefficient of wavelet transform, and t is a pre-selected threshold. The threshold quantization adopts a fixed threshold method, which is defined as: t=σ2log(N), where N is the number of signal samples, and σ is the standard deviation of noise.

[0103] S9: constructing a sound event classification model based on the frequency domain residual shrinkage network F-DRSN; training the sound event classification model to obtain a trained sound event classification model.

[0104] In step S9, the sound classification model is trained, specifically including the following steps:

[0105] b1: selecting a data set with original audio data samples, dividing the data set to obtain a training set and a test set;

[0106] b2: obtaining the original time domain signal S(t) of the training set data, and calculating the time domain signal S4(t) after reducing the reverberation in the frequency band;

[0107] b3: mixing the original time domain signal S(t) of the training set and the final time domain signal S4(t) after reducing the reverberation to form a data enhanced training set;

[0108] b4: extracting acoustic features from the enhanced training set to obtain: log-mel spectrum, gamma-pass spectrum and gamma-pass frequency cepstral coefficient, and the three acoustic features are denoted as: acoustic feature set for training;

[0109] b5: input the training acoustic feature set into the sound event classification model, train the sound event classification model, obtain the trained sound event classification model, and train the sound event classification model;

[0110] b6: extract acoustic features from the data of the test set respectively, obtain: log-mel spectrogram, gamma pass spectrogram and gamma pass frequency cepstral coefficient, and record the three acoustic features as: verification acoustic feature set;

[0111] b7: input the verification acoustic feature set into the trained sound event classification model, judge the model training result according to the classification result, if the effect is not good, retrain, if the effect reaches the expectation, stop training.

[0112] S10: obtaining the sound signal data to be detected, extracting acoustic features from the sound signal data to be detected, obtaining the acoustic features to be recognized, inputting the acoustic features to be recognized into the trained sound event classification model, and obtaining the final classification result.

[0113] In step S10, the method for extracting acoustic features from the sound signal data to be detected is as follows:

[0114] c1: confirming the classification result obtained by inputting the verification acoustic feature set into the trained sound event classification model, finding the acoustic feature with the highest classification accuracy, and recording it as: recognition acoustic feature;

[0115] For example, in the training, it is found that the acoustic feature with the highest classification accuracy in the low frequency band is: log-mel spectrogram, then the log-mel spectrogram is recorded as the recognition acoustic feature;

[0116] c2: after extracting the recognition acoustic feature from the sound signal data to be detected, record it as: acoustic feature to be recognized;

[0117] In actual classification, the log-mel spectrogram feature is extracted from the sound signal data to be detected, and is input into the trained sound event classification model for classification.

[0118] In this method, according to different types of sound signal data to be detected, the frequency division ratio r is set accordingly, the corresponding frequency division band data feature is extracted for reverberation noise elimination, and then the final time domain signal S4(t) after reverberation reduction is used to train the sound event classification model, so as to ensure that the classification model trained is especially suitable for the sound signal data to be detected; at the same time, in the training, the acoustic feature with the highest classification accuracy of the sound signal data to be detected in the sound event classification model is observed, recorded as the recognition acoustic feature, and then in the actual test, the recognition acoustic feature corresponding to the sound signal data to be detected is used for classification, which further improves the classification accuracy.

[0119] To verify the performance of the sound event classification method based on frequency band reverberation reduction in the technical solution of this invention, an experimental environment was set up using Windows 11 system, RTX 3080 Ti (12G VRAM), and 32G RAM. Keras + TensorFlow was used as the deep learning framework, and the UrbanSound8K standard dataset for urban sound event classification was employed. The dataset consisted of folds 1-9 as the training set (7895 samples), which was augmented to 15790 samples. The test set consisted of WAV audio files from fold 10 (838 samples). The following experiments were conducted:

[0120] (1) Performance comparison of different network models;

[0121] (2) Comparison of classification accuracy of reverberation reduction models in different frequency bands;

[0122] (3) Reduce the optimal frequency division ratio of the high-frequency reverberation model;

[0123] (4) Classification accuracy of different data augmentation methods.

[0124] We used CNN, ResNet-18 (a common residual network), and F-DRSN (the model in this application) as classification models, respectively, and Log-Mel, Gamma, and GFCC as features to conduct sound event classification experiments on the Urbansound8K dataset. The accuracy of the three models was compared, and the specific results are shown in Table 1.

[0125] Table 1 Performance Comparison of Different Network Models

[0126] Network model Log-Mel Gamma GFCC CNN 83.37％ 81.58％ 80.38％ ResNet-18 84.81％ 83.49％ 83.25％ F-DRSN 85.17％ 84.57％ 83.85％

[0127] As shown in Table 1, compared with Convolutional Neural Networks (CNNs), the F-DRSN network of this patent improves performance on the three features by 1.8%, 2.99%, and 3.47%, respectively; compared with the common ResNet-18 residual network, the F-DRSN network of this patent improves performance on the three features by 0.36%, 1.08%, and 0.6%, respectively. In summary, the F-DRSN model has stronger representation capabilities than CNNs and ResNet-18, thus significantly improving classification accuracy.

[0128] Using the sound event classification model (F-DRSN) in this application as the classification model, and with Log-Mel, Gamma, and GFCC as features, sound event classification experiments were conducted on the Urbansound8K dataset, and the accuracy was compared. The specific results are shown in Tables 2, 3, and 4.

[0129] Table 2 Comparison of classification accuracy of reverberation reduction models in different frequency bands (r = 0.3)

[0130] Frequency band Log-Mel Gamma GFCC Without data augmentation 85.17％ 84.57％ 83.85％ Low frequency band 85.65％ 87.32％ 85.89％ High frequency band 87.80％ 87.68％ 86.12％ Full frequency band 86.72％ 86.36％ 86.01％

[0131] From Table 2, (1) Compared with the experiment without data enhancement, the network model classification accuracy is improved to varying degrees in low frequency, high frequency and full frequency when reducing reverberation. Among them, the performance of reducing reverberation in high frequency is the best, and the accuracy rates are 87.80%, 87.68% and 86.12% respectively. At the same time, in the high frequency, the network with Log-Mel feature as input performs better than other features as input. (2) In terms of inputting various features, the model mostly performs well on Log-Mel feature, except that in the low frequency, the accuracy rates of Gamma and GFCC features are higher than that of Log-Mel feature, which are 1.67% and 0.24% higher respectively.

[0132] Table 3 Best frequency ratio of high frequency reverberation reduction model

[0133] Frequency ratio r 0.1 0.2 0.3 0.35 0.4 0.5 0.6 0.7 0.9 Accuracy 85.41％ 86.48％ 87.80％ 89.12％ 86.00％ 87.92％ 85.77％ 85.65％ 85.41％

[0134] From Table 3, (1) When r is from 0.1 to 0.35, reducing high frequency reverberation has good effect and causes less distortion to the original sound, so the accuracy rate gradually increases; (2) Since reducing high frequency reverberation inevitably causes some distortion to the original sound, when r is from 0.35 to 0.4, the degree of reducing high frequency reverberation gradually increases, and the proportion of distortion also increases. When r is from 0.35 to 0.4, the advantage of reducing high frequency reverberation to enhance speech intelligibility dominates, and the accuracy rate rises a little. (3) When r is from 0.5 to 0.9, the larger r is, the higher the proportion of original audio distortion is, which reduces the effective information of the audio and leads to a decrease in the accuracy rate of the model. (4) The experiment verifies that when the frequency ratio is 0.35, the accuracy rate of the model is the highest, indicating that this parameter is the optimal parameter.

[0135] Table 4 Classification accuracy of different data enhancement methods

[0136] Data augmentation method Log-Mel Gamma GFCC Baseline 85.17％ 84.57％ 83.85％ Dynamic range compression 87.44％ 87.32％ 86.84％ Time stretch 88.16％ 86.01％ 86.96％ Pitch conversion 87.20％ 87.32％ 85.65％ Reducing high frequency reverberation 89.12％ 88.28％ 87.80％

[0137] From Table 4, compared with traditional data enhancement methods, the data enhancement method of reducing high frequency reverberation performs best among the three acoustic features. The classification accuracy of Log-Mel feature, which performs best in reducing high frequency reverberation, is improved by 0.96%, 0.96% and 0.84% respectively compared with time stretching, pitch conversion or dynamic range compression, time stretching, which are the best among the three acoustic features after data enhancement.

[0138] In summary, based on the comparison of classification accuracy in Table 1, the feature expression ability of F-DRSN is better than that of common network CNN and ResNet-18;

[0139] Based on the classification accuracy data in Table 2, it can be concluded that reducing high-frequency reverberation performs better in the frequency band reduction reverberation method, and the sound classification model in the basic embodiment is a classification model for high-frequency band data; because high-frequency band sounds account for the majority in urban sounds, the experimental data and the collection environment of the original sound data are matched;

[0140] Based on the classification accuracy in Table 3, the best frequency division ratio r = 0.35 can be obtained, that is, when the sound classification model in the embodiment is used to classify the high-frequency band spectrum signal to be processed, the frequency division ratio r is set to 0.35;

[0141] Based on Table 4, compared with the traditional data enhancement method, the data enhancement method for reducing high-frequency reverberation significantly improves the network representation ability.

[0142] The sound event classification method provided in the application reduces the high-frequency reverberation or low-frequency reverberation and the like background noise in the original audio before processing the audio data, and the mixed audio of the enhanced audio and the original audio is used as the input, and the acoustic features are extracted and input into the model, the method can effectively reduce the high-frequency reverberation or low-frequency reverberation and the like background noise in the original audio, the sound quality damage is small, and the intelligibility of the speech is enhanced, thereby improving the representation ability of the model and the classification accuracy of the model, so that the accuracy of the sound event classification model is improved.

Claims

1. A sound event classification method based on frequency band division and reverberation reduction, comprising the following steps: S1: Collecting original audio sample information, and performing preprocessing operation to obtain preprocessed audio signal to be processed; characterized in that it further comprises the following steps: S2: Transforming the audio signal to be processed by short-time Fourier transform to convert the signal from time domain S(t) to frequency domain to obtain the frequency spectrum signal S(t, ω) to be processed, wherein t is time and ω is frequency; S3: Perform frequency component analysis on the spectrum signal S(t,ω) to be processed, and obtain the highest frequency ω in the signal. h and lowest frequency ω l This allows us to obtain the input signal bandwidth B corresponding to the spectral signal S(t,ω) to be processed. i ; B i = ω h - ω l ; S4: based on the original audio sample information of the collection scene, set the frequency division ratio r, select the frequency band signal and control the output bandwidth B of the selected frequency band o ; B o = B i *r; S5: Based on the output bandwidth B0 of the selected frequency band, the frequency spectrum signal S(t, ω) to be processed is sent to a Butterworth filter, and based on the Butterworth frequency domain response, the Butterworth amplitude function |H(ω)| in the frequency domain and the selected frequency spectrum signal S1(t, ω) corresponding to B0 are obtained; where N is the order, ω c is the cutoff frequency; S6: The selected frequency spectrum signal S1(t, ω) is subjected to reverberation reduction by using a weighted prediction error algorithm WPE to obtain a frequency spectrum signal output S2(t, ω) after reverberation reduction; wherein, estimated by a prediction filter in the WPE, H is a conjugate transpose operator, D is a prediction delay, D separates the reverberant environment into early and late reverberation, is the late reverberation part; S7: The frequency spectrum signal S(t, ω) to be processed is subtracted from the selected frequency spectrum signal S1(t, ω) to obtain other signal S3(t, ω); S2(t, ω) and other signal S3(t, ω) are combined to obtain the final output frequency spectrum signal S4(t, ω) after reverberation reduction, and the signal S4(t, ω) is converted from frequency domain to time domain by using inverse short-time Fourier transform to obtain S4(t); S8: Based on a deep residual shrinkage network DRSN, a frequency domain residual shrinkage network F-DRSN is constructed; The frequency domain residual shrinkage network F-DRSN comprises a convolution layer, a residual shrinkage unit, a global average pooling and a fully connected layer connected in sequence; wherein the number of the residual shrinkage units is greater than 1; each of the residual shrinkage units comprises a frequency domain self-correction algorithm; S9: Based on the frequency domain residual shrinkage network F-DRSN, a sound event classification model is constructed; The original audio data sample is selected as the data set, the data set is divided to obtain a training set; the original time domain signal S(t) of the training set data is calculated, and the time domain signal S4(t) after frequency band division and reverberation reduction is calculated; the original time domain signal S(t) of the training set and the final time domain signal S4(t) after reverberation reduction are mixed to form a training set after data enhancement; the acoustic features of the enhanced training set are extracted to train the sound event classification model, and the trained sound event classification model is obtained; S10: Acquiring sound signal data to be detected, and extracting acoustic features of the sound signal data to be detected to obtain acoustic features to be recognized; The acoustic features to be recognized are sent into the trained sound event classification model to obtain the final classification result.

2. The sound event classification method based on frequency band division and reverberation reduction according to claim 1, characterized in that: In step S5, the order N and the cut-off frequency ω c are calculated by the following method. Wherein, δ1 is the maximum attenuation of the passband, and δ2 is the minimum attenuation of the stopband; ω p is the passband cutoff frequency, ω s is the stopband cutoff frequency, ω p and ω s The calculation method is as follows: wherein, when the filtered frequency band is a high frequency band, ω p is written as ω s is written as when the filtered frequency band is a low frequency band, ω p is written as ω s is written as ω h is the highest frequency of the spectrum signal S(t, ω) to be processed, ω l is the lowest frequency of the spectrum signal S(t, ω) to be processed; B o is the output bandwidth of the selected frequency band; and respectively, the bandwidth of the high frequency band transition zone, the bandwidth of the low frequency band transition zone.

3. The sound event classification method based on frequency band division and reverberation reduction according to claim 1, characterized in that: The residual shrinkage unit comprises a frequency domain self-correction module, two convolution layers and a soft thresholding module connected in sequence, and a attention module, and the output of the last convolution layer is sent into the soft thresholding module and the attention module to extract typical features in the channel, and then sent into the soft thresholding module to remove noise.

4. The sound event classification method based on frequency band division and reverberation reduction according to claim 3, characterized in that: The operation in the frequency domain self-correction module comprises: a1: compress the frequency domain dimension of the input feature x by MaxPooling; a2: extract features from the compressed result by using two convolutional layers; a3: perform up-sampling to restore the size to the same size as the initial input feature, and sum the extracted frequency domain features and the original feature map to obtain the output feature x1; a4: compress x1 into a one-dimensional vector by global average pooling, and obtain the recalibrated result through a fully connected layer, denoted as x2; a5: multiply the result of the original feature after two convolutional layers with x2 to obtain the final output x3.

5. The sound event classification method based on frequency band division and reverberation reduction according to claim 1, characterized in that: In step S9, the sound classification model is trained, specifically including the following steps: b1: selecting a data set with original audio data samples, dividing the data set to obtain a training set and a test set; b2: calculating the time domain signal S4(t) after reducing reverberation in each frequency band corresponding to the training set; b3: fully mixing the original time domain signal S(t) and the time domain signal S4(t) after reducing reverberation in each frequency band corresponding to the training set to form an enhanced training set; b4: extracting acoustic features from the enhanced training set to obtain: log-mel spectrum, gamma-pass spectrum and gamma-pass frequency cepstral coefficient, and recording the three acoustic features as: training acoustic feature set; b5: inputting the training acoustic feature set into the sound event classification model to train the sound event classification model, and obtaining the trained sound event classification model; b6: extracting acoustic features from the data of the test set respectively to obtain: log-mel spectrum, gamma-pass spectrum and gamma-pass frequency cepstral coefficient, and recording the three acoustic features as: verification acoustic feature set; b7: inputting the verification acoustic feature set into the trained sound event classification model, and judging the model training result according to the classification result.

6. The sound event classification method based on frequency band division and reverberation reduction according to claim 5, characterized in that: In step S10, the method for extracting acoustic features from the to-be-detected sound signal data is as follows: c1: confirming the classification result obtained by inputting the verification acoustic feature set into the trained sound event classification model, finding the acoustic feature with the highest classification accuracy, and recording it as: recognition acoustic feature; c2: extracting the recognition acoustic feature from the to-be-detected sound signal data, and recording it as: to-be-recognized acoustic feature.

7. The sound event classification method based on frequency band division and reverberation reduction according to claim 1, characterized in that: In step S1, the original audio sample information includes: audio duration and sampling frequency; The preprocessing operation includes: pre-emphasis, framing and windowing.

Images