A sound scene classification method based on multi-scale residual attention network

Abstract

The present invention provides a sound scene classification method based on a multi-scale residual attention network, comprising the following steps: extracting features from collected audio data, extracting a log-Mel spectrogram and its first-order and second-order differences as input features; constructing a multi-scale residual attention network, inputting the extracted log-Mel spectrogram into the network for training and establishing a classification model; employing a mixup method to enhance data diversity; employing a focal loss to focus on samples that are difficult to classify; acquiring new sound scene speech, and using the classification model to classify the speech into a sound scene, thereby obtaining a sound scene classification result. The present invention employs a log-Mel spectrogram and its first-order and second-order differences, and uses a multi-scale residual attention network model to classify sound scenes, thereby being able to mine more rich and comprehensive feature information, thereby improving sound scene classification performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more particularly to a sound scene classification method based on a multi-scale residual attention network. Background Art

[0002] Humans have an inherent ability to identify sound scenes. This means they can use past experience to determine the context of a piece of audio, such as a subway or bus. With the continuous advancement of signal processing and artificial intelligence technologies, it's becoming possible for machines to understand and identify the source of sound. Acoustic scene classification (ASC) is a multi-category classification task that aims to identify the context of an audio clip. Currently, ASC has been widely used in smart wearable devices, audio archiving, interactive robots, security monitoring, and other fields.

[0003] There are two main categories of sound scene classification methods: one is the sound scene classification method based on traditional machine learning, such as Gaussian mixture model, hidden Markov model, support vector machine, etc., but its classification effect is not high and its generalization ability is poor; the other is the sound scene classification method based on deep learning, such as deep neural network, convolutional neural network, recurrent neural network, etc., but it often only contains a single-scale convolution kernel, the mined features are not rich and comprehensive enough, and does not consider that the features of different regions have different importance.

[0004] Therefore, how to comprehensively mine data features and improve the accuracy of sound scene classification is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a sound scene classification method based on a multi-scale residual attention network. In order to solve the problems in the current sound scene classification task, such as the single and insufficient feature scale of feature extraction and the failure to consider the different importance of different regions of the extracted features, a multi-scale residual network is adopted to mine more details and overall information, and channel attention is introduced at the same time to highlight important features and suppress minor features.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A sound scene classification method based on a multi-scale residual attention network includes the following steps:

[0008] Step 1: Collect audio data and input it into the feature extraction module for feature extraction. The logarithmic Mel spectrum graph and its first-order difference and second-order difference are extracted as input features.

[0009] Step 2: Construct a multi-scale residual attention network and input the input features into the network for training to establish a classification model;

[0010] Step 3: Use the mixup method to process the audio data to obtain data samples and enhance data diversity;

[0011] Step 4: Input the data samples into the classification model for classification, use focal loss to focus on samples that are difficult to classify, and optimize the classification model;

[0012] Step 5: Obtain new sound scene speech, input it into the optimized classification model for sound scene classification, and obtain the sound scene classification result.

[0013] Preferably, the specific process of feature extraction in step 1 is:

[0014] Step 1.1: Perform pre-emphasis processing on the collected voice data to make the high-frequency and low-frequency parts of the sound signal more balanced;

[0015] Step 1.2: Divide the pre-emphasized voice data into frames, into several frames of voice signals;

[0016] Step 1.3: Use the Hanning window function to perform windowing processing on each frame of speech signal to obtain a short-time windowed speech signal;

[0017] Step 1.4: Perform Fourier transform on the short-time windowed speech signal to convert it from the time domain to the frequency domain to obtain a frequency domain signal;

[0018] Step 1.5: Pass the obtained frequency domain signal through the Mel filter to obtain a Mel spectrogram of appropriate size;

[0019] Step 1.6: Take the logarithm of the Mel-spectrogram to obtain the logarithmic Mel-spectrogram;

[0020] Step 1.7: Calculate the first-order and second-order differences of the log-Mel spectrum to obtain the dynamic characteristics of the speech signal. Then, stack the log-Mel spectrum and its first-order and second-order differences to obtain the final input features.

[0021] Preferably, in step 1, the frame overlap rate is 50% when the voice data is framed; the number of FFT points in the Fourier transform process is 2048; and the number of Mel filters is 128.

[0022] Preferably, the specific process of step 2 is:

[0023] Step 2.1: Separate the input features consisting of the logarithmic Mel-frequency spectrum and its first-order difference and second-order difference into high-frequency and low-frequency parts;

[0024] Step 2.2: Input the high-frequency part and the low-frequency part into the channel attention module of the multi-scale residual attention network respectively, assign different weights according to the importance of the features, highlight the important features, suppress the minor features, and thus generate new features;

[0025] Step 2.3: Input the new features extracted by the channel attention module into the multi-scale residual module of the multi-scale residual attention network to extract feature information of different precisions and depths, and obtain high-frequency feature maps and low-frequency feature maps;

[0026] Step 2.4: Concatenate the two feature maps obtained by the multi-scale residual module in the frequency dimension to obtain all the features;

[0027] Step 2.5: All features are sequentially passed through a convolutional block consisting of a batch normalization layer, a rectified linear unit, and a 1×1 convolutional layer, a convolutional block consisting of a batch normalization layer and a 1×1 convolutional layer, a batch normalization layer, a global average pooling layer, and a softmax layer for classification.

[0028] Preferably, the specific process of generating new features through the channel attention module in step 2.2 includes:

[0029] Step 2.2.1: Perform maximum pooling and average pooling operations on the high-frequency input features and the low-frequency input features respectively to obtain two feature maps;

[0030] Step 2.2.2: Send the two feature maps obtained by pooling processing into the multi-layer perceptron respectively to obtain two perception results;

[0031] Step 2.2.3: Add the two perception results obtained by the multi-layer perceptron to obtain the result;

[0032] Step 2.2.4: Apply sigmoid activation to the result of the addition to obtain the weight parameters of the input features.

[0033] Step 2.2.5: Finally, multiply the weight parameter by the input feature to generate a new feature.

[0034] Preferably, the new features in step 2.3 pass through the batch normalization layer and the convolution layer in sequence, and then pass through the residual block Residual 01 consisting of two 1×1, two 3×3 and two 5×5 convolution kernels twice, and then pass through the residual block Residual 02 consisting of two 1×1, two 3×3 and two 5×5 convolution kernels, maximum pooling, average pooling and zero padding three times, and the residual block Residual 01 consisting of a combined block, thereby obtaining a high-frequency feature map and a low-frequency feature map.

[0035] Preferably, the formula for obtaining data samples using the mixup method adopted in step 3 is:

[0036] x=λx i +(1-λ)x j

[0037] y=λy i +(1-λ)y j

[0038] Among them, (x i ,y i ) and (x j ,y j ) are two random samples randomly selected from the training set of the collected speech data, x i and x j is the original input vector, y i and y j is the corresponding label encoding; λ is a hyperparameter, λ∈[0,1].

[0039] Preferably, the acquired new sound scene speech is tested to obtain a real classification result, and the classification accuracy is calculated based on the sound scene classification result obtained in step 5. The classification model can be optimized and modified based on the classification accuracy to improve the classification accuracy.

[0040] It can be seen from the above technical solution that compared with the prior art, the present invention discloses a sound scene classification method based on a multi-scale residual attention network, which utilizes convolution kernels of multiple scales to mine more details and overall information, and combines the improved residual network structure to extract more semantic information at different levels; introduces a channel attention mechanism, and assigns different weights to features on different channels according to the different importance of features, so as to learn key features, suppress secondary features, and enhance the network's ability to learn features; adopts a mixup method to enhance data diversity, and uses focal loss to focus on samples that are difficult to classify, thereby improving the classification effect of sound scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 The accompanying figure is a flow chart of the sound scene classification method based on the multi-scale residual attention network provided by the present invention;

[0043] Figure 2 The accompanying drawing is a schematic diagram of the feature extraction process provided by the present invention;

[0044] Figure 3 The accompanying drawing is a schematic diagram of the structure of the channel attention module provided by the present invention;

[0045] Figure 4 The accompanying drawing is a schematic diagram of the structure of the multi-scale residual module provided by the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, the present invention provides a sound scene classification method based on a multi-scale residual attention network, comprising the following steps:

[0048] S1: Input the collected audio data into the feature extraction module, extract the logarithmic Mel spectrum and its first-order difference and second-order difference as input features; the specific process of the feature extraction module is as follows Figure 2 As shown, the following steps are included:

[0049] S1.1: Perform pre-emphasis processing on the collected audio data to make the high-frequency and low-frequency parts of the sound signal more balanced. The pre-emphasis formula is as follows:

[0050] H(z)=1-az -1 (1)

[0051] S1.2: Divide the pre-emphasized audio data into several frames of speech signals;

[0052] S1.3: Use the Hanning window function to perform windowing processing on each frame of the speech signal to obtain a short-time windowed speech signal. The formula of the Hanning window is as follows:

[0053]

[0054] S1.4: Perform Fourier transform on the short-time windowed speech signal to convert it from the time domain to the frequency domain to obtain a frequency domain signal. The conversion process is as follows:

[0055]

[0056] S1.5: Pass the frequency domain signal obtained in the previous step through the Mel filter to obtain a Mel spectrum of appropriate size. The center frequency of each triangular filter in the Mel filter is shown in formula (4), and the frequency response is shown in formula (5):

[0057]

[0058]

[0059] Where f(m) is the center frequency of the mth filter, f l and f h are the upper and lower limits of the triangular filter, N is the number of sampling points in the filter, and f s is the sampling frequency; M -1 and M are inverse functions of each other, and M is defined as shown in formula (6):

[0060]

[0061] in,

[0062] S1.6: Take the logarithm of the mel-spectrogram to obtain a logarithmic mel-spectrogram;

[0063] S1.7: Calculate the first-order and second-order differences of the log-mel spectrogram to obtain the dynamic characteristics of the speech signal. Then, stack the log-mel spectrogram and its first-order and second-order differences to obtain the final input features.

[0064] S2: Construct a multi-scale residual attention network, input the extracted input features into the multi-scale residual attention network for training, and obtain a classification model;

[0065] S2.1: Separate the input features consisting of the logarithmic Mel-frequency spectrum and its first-order difference and second-order difference into high-frequency and low-frequency parts;

[0066] S2.2: The high-frequency part and the low-frequency part are input into the channel attention module of the multi-scale residual attention network respectively. Different weights are assigned according to the importance of the features, highlighting the important features and suppressing the minor features. The specific process can be expressed as formulas (7) and (8):

[0067] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) (7)

[0068]

[0069] Where F is the input feature map, size is (H×W×C); AvgPool(F) and MaxPool(F) are average pooling and maximum pooling operations respectively; M c (F) is the weight parameter, F' is the feature obtained by the channel attention module; σ represents the sigmoid function, Represents the product operation;

[0070] The channel attention module structure is as follows Figure 3 As shown, the following steps are included:

[0071] S2.2.1: Perform max pooling and average pooling on the input features to obtain two feature maps;

[0072] S2.2.2: Send the two feature maps obtained by pooling processing into the multi-layer perceptron respectively to obtain two perception results;

[0073] S2.2.3: Add the two perception results obtained by the multilayer perceptron;

[0074] S2.2.4: Apply a sigmoid activation operation to the summed result to obtain the weight parameters of the input features;

[0075] S2.2.5: Finally, the weight parameter is multiplied by the input feature to generate a new feature.

[0076] S2.3: The new features proposed by the channel attention module are input into the multi-scale residual module (MSRM) of the multi-scale residual attention network to extract feature information of different precisions and depths, and obtain high-frequency feature maps and low-frequency feature maps; the structure of the multi-scale residual module is as follows Figure 4 As shown, the steps are as follows:

[0077] S2.3.1: The generated new features are passed through the batch normalization layer (BN) and the convolution layer;

[0078] S2.3.2: Pass the residual block Residual 01 twice, which consists of three convolution kernels of different scales: 1×1, 3×3, and 5×5.

[0079] S2.3.3: The network is then passed through three convolution blocks consisting of Residual 02, which consists of three convolution kernels of different scales: 1×1, 3×3, and 5×5, max pooling, average pooling, and zero padding, and Residual 01.

[0080] S2.3.4: Finally, obtain the high-frequency feature map and the low-frequency feature map.

[0081] S2.4: Concatenate the two feature maps obtained by the multi-scale residual module in the frequency dimension to obtain the full features.

[0082] S2.5: All features are sequentially passed through a convolutional block consisting of a batch normalization layer, a rectified linear unit (ReLU), and a 1×1 convolutional layer, a convolutional block consisting of a batch normalization layer and a 1×1 convolutional layer, a batch normalization layer, a global average pooling layer, and a softmax layer for classification.

[0083] S3: Use the mixup method to enhance data diversity. Mixup can be specifically expressed as:

[0084] x=λx i +(1-λ)x j (9)

[0085] y=λy i +(1-λ)y j (10)

[0086] Where (x i ,y i ) and (x j ,y j ) are two random samples randomly selected from the training set of the collected speech data, x i and x j is the original input vector, y i and y j Encode the corresponding label; λ is a hyperparameter, λ∈[0,1], which can control the degree of confusion between two random samples;

[0087] S4: Focus loss is used to focus on samples that are difficult to classify. The focus loss function can be specifically expressed as:

[0088]

[0089] Where n represents the number of categories; y i represents the true label encoding of the i-th sample; p i represents the probability that the i-th sample is predicted to be the true label; α is the weight factor; λ is the hyperparameter;

[0090] S5: Acquire new sound scene speech, use the trained classification model to perform sound scene classification on the new sound scene speech, and obtain a sound scene classification result.

[0091] Example

[0092] The public dataset TAU ​​UrbanAcoustic Scenes 2020Mobile Development dataset (TAU for short) in the DCASE2020 (Acoustic Scene and Event Detection and Classification Challenge) Challenge Task 1A is used for sound scene classification. The dataset contains recordings from 10 different acoustic scenes recorded by 9 different devices in 10 European cities. The dataset contains 10 sound scenes, including airports (Airport), shopping malls (Shopping mall), subway stations (Metro station), sidewalks (Street pedestrian), public squares (Public square), congested streets (Streettraffic), trams (Tram), buses (Bus), subways (Metro) and parks (Park). The experiment uses the classification accuracy of the sound scene category as the criterion for judging the model. The training set is used to train the model parameters, and the test set is used to compare the performance of the model. The experimental results are shown in Table 1 Sound scene category classification results:

[0093] Table 1 Classification results of sound scene categories

[0094]

[0095] From the experimental results, it can be seen that the performance of the multi-scale residual attention network proposed in this invention is significantly better than the baseline system of DCASE2020 Task1A.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0097] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sound scene classification method based on multi-scale residual attention network, characterized by: The following steps are involved: Step 1: Collect audio data for feature extraction, and extract the logarithmic Mel-spectrogram and its first-order difference and second-order difference as input features; Step 2: Construct a multi-scale residual attention network and input the input features into the network for training to establish a classification model; Step 3: Use the mixup method to process the audio data to obtain data samples; Step 4: Input the data sample into the classification model for classification, use focal loss to focus on samples that are difficult to classify, and optimize the classification model; Step 5: Obtain new sound scene speech, input it into the optimized classification model to perform sound scene classification, and obtain the sound scene classification result; The specific process of feature extraction in step 1 is as follows: Step 1.1: pre-emphasize the collected voice data; Step 1.2: Divide the pre-emphasized voice data into frames, into several frames of voice signals; Step 1.3: Use the Hanning window function to perform windowing processing on each frame of speech signal to obtain a short-time windowed speech signal; Step 1.4: Perform Fourier transform on the short-time windowed speech signal to convert it from the time domain to the frequency domain to obtain a frequency domain signal; Step 1.5: Pass the obtained frequency domain signal through the Mel filter to obtain the Mel spectrum; Step 1.6: Take the logarithm of the Mel-spectrogram to obtain the logarithmic Mel-spectrogram; Step 1.7: Calculate the first-order difference and second-order difference of the log-Mel spectrum graph, and then stack the log-Mel spectrum graph and its first-order difference and second-order difference to obtain the final input features; The specific process of step 2 is: Step 2.1: Divide the input features into high-frequency and low-frequency parts; Step 2.2: Input the high-frequency part and the low-frequency part into the channel attention module of the multi-scale residual attention network respectively, assign different weights according to the importance of the features, and generate new features; the specific process of generating new features through the channel attention module in step 2.2 includes: Step 2.2.1: Perform maximum pooling and average pooling operations on the high-frequency input features and the low-frequency input features respectively to obtain two feature maps; Step 2.2.2: Send the two feature maps obtained by pooling processing into the multi-layer perceptron respectively to obtain two perception results; Step 2.2.3: Add the two perception results obtained by the multi-layer perceptron to obtain the result; Step 2.2.4: Apply sigmoid activation to the result of the addition to obtain the weight parameters of the input features. Step 2.2.5: Finally, multiply the weight parameter by the input feature to generate a new feature; Step 2.3: Input the new features into the multi-scale residual module of the multi-scale residual attention network, extract feature information of different precisions and different depths, and obtain high-frequency feature maps and low-frequency feature maps; the new features in step 2.3 pass through the batch normalization layer and the convolution layer in sequence, pass through the residual block Residual 01 composed of two 1×1, two 3×3 and two 5×5 convolution kernels twice, and then pass through the residual block Residual 02 composed of two 1×1, two 3×3 and two 5×5 convolution kernels, maximum pooling, average pooling and zero padding three times, and the residual block Residual 01 composed of zero padding, thereby obtaining high-frequency feature maps and low-frequency feature maps; Step 2.4: Concatenate the high-frequency feature map and the low-frequency feature map in the frequency dimension to obtain all the features; Step 2.5: All features are sequentially classified through a convolutional block consisting of a batch normalization layer, a rectified linear unit, and a 1×1 convolutional layer, a convolutional block consisting of a batch normalization layer and a 1×1 convolutional layer, a batch normalization layer, a global average pooling layer, and a softmax layer to obtain a classification model.

2. The sound scene classification method based on multi-scale residual attention network according to claim 1 is characterized in that In step 1, the frame overlap rate is 50% when the voice data is framed; the number of FFT points in the Fourier transform process is 2048; and the number of Mel filters is 128.

3. The sound scene classification method based on multi-scale residual attention network according to claim 1 is characterized in that The formula for obtaining data samples using the mixup method in step 3 is: x=λx i +(1-λ)x j y=λy i +(1-λ)y j Among them, (x i ,y i ) and (x i ,y j ) are two random samples randomly selected from the training set of the collected speech data, x i and x j is the original input vector, y i and y j is the corresponding label encoding; λ is a hyperparameter, λ∈[0,1].

4. The sound scene classification method based on multi-scale residual attention network according to claim 1 is characterized in that The acquired new sound scene speech is tested to obtain a true classification result, and the classification accuracy is calculated based on the sound scene classification result obtained in step 5.

Images