A network model for acoustic scene classification based on hierarchical information fusion

Abstract

The present invention provides an acoustic scene classification network model based on hierarchical information fusion, comprising an acoustic scene feature learning module and a hierarchical information fusion module, wherein the acoustic scene feature learning module extracts features from the audio and inputs them into the hierarchical information fusion module; a parent class classifier and a child classifier are provided in the hierarchical information fusion module, and the hierarchical information fusion module first performs 1D convolution on the input features in two ways to obtain a parent class representation and a child class representation; then the parent class representation is input into the parent class classifier to obtain a predicted parent class category, and the parent class representation and the child class representation are spliced ​​and fused and then input into the child class classifier to obtain a predicted child class category. In the acoustic scene classification network model of the present invention, a parent class classifier and a child classifier are provided, and in the process of subclass classification, the parent class information is integrated into the child class information, thereby forcing the network model to learn the hierarchical structure relationship between categories, and ultimately effectively improving the classification accuracy of the network model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an acoustic scene classification network model based on hierarchical information fusion. Background Art

[0002] With the development of computer hearing, acoustic scene classification has become an important research field. Acoustic scene classification involves identifying the recording environment based on audio. Currently, acoustic scene classification technology has found important practical applications in smart wearable devices, assistive devices for the disabled, audio monitoring systems, and audio file management.

[0003] Acoustic scene classification technology is difficult to implement because an audio file typically contains a vast amount of information, often including content irrelevant to the scene. Furthermore, unlike images, acoustics is difficult to visualize. These factors have resulted in some existing acoustic scene classification technologies having limited accuracy. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides an acoustic scene classification network model based on hierarchical information fusion to solve the acoustic scene classification problem and improve the accuracy of classification.

[0005] The present invention achieves the above technical objectives through the following technical means.

[0006] An acoustic scene classification network model based on hierarchical information fusion includes an acoustic scene feature learning module and a hierarchical information fusion module, wherein the acoustic scene feature learning module extracts features from the audio and inputs them into the hierarchical information fusion module; the hierarchical information fusion module is provided with a parent class classifier and a child classifier, and the hierarchical information fusion module first performs 1D convolution on the input features in two ways to obtain a parent class representation and a child class representation, respectively; then, the parent class representation is input into the parent class classifier to obtain a predicted parent class category, and the parent class representation and the child class representation are concatenated and fused and then input into the child class classifier to obtain a predicted child class category.

[0007] Furthermore, a loss function layer is provided in the hierarchical information fusion module, and its loss function is:

[0008]

[0009] in is the joint loss, is the cross entropy loss of the parent class, is the cross entropy loss of subclasses, is the hierarchical dependency loss, α is the weight of the hierarchical dependency loss; the hierarchical dependency loss function is:

[0010]

[0011] in Indicates whether the affiliation between the parent class and the child class is correct. If it is correct, The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the parent class is correct. If it is correct The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the subclass is correct. If it is correct The value is 1, which is incorrect. The value is 0; ploss is the dependency penalty and is set to a constant.

[0012] Furthermore, the acoustic scene feature learning module adopts Trident ResNet.

[0013] Furthermore, the Trident ResNet contains three residual network paths; in each residual path: a convolution is performed first, with a step size of 1 in the frequency dimension and a step size of 2 in the time dimension, followed by 12 residual blocks, each of which contains two convolutions, batch normalization is performed before each convolution, and a linear rectification function is used for activation, and the convolution kernel size is 3×3.

[0014] Furthermore, when training the network model, the weight α is set to 0.8, each training lasts for 510 cycles, and the learning rate is reset to the maximum value of 0.1 after 2, 6, 14, 30, 62, 126 and 254 cycles, and then decayed to 1×10 -5 .

[0015] Furthermore, the input of the acoustic scene feature learning module is a logarithmic Mel spectrogram.

[0016] Furthermore, for the audio to be recognized, the audio data is first pre-emphasized, framed, and windowed. Then, a short-time Fourier transform with 2048 FFT points is used to compress each spectrum to a 256-Mel frequency scale. Finally, the logarithm is taken to obtain a logarithmic Mel spectrogram. Before the logarithmic Mel spectrogram is input into the acoustic scene feature learning module, deltas and delta-deltas are first calculated and superimposed along the channel dimension.

[0017] The beneficial effects of the present invention are:

[0018] (1) The present invention provides an acoustic scene classification network model based on hierarchical information fusion, which is used to identify the acoustic scene category behind the audio and solve the acoustic scene classification problem. The designed hierarchical information fusion module includes a parent classifier and a child classifier; and in the process of subclass classification, the parent class information is fused into the child class information, so that the parent class information can be shared among its parallel subclasses, forcing the network model to learn the hierarchical structure relationship between categories, and ultimately effectively improving the classification accuracy of the network model.

[0019] (2) In the acoustic scene classification network model of the present invention, a hierarchical dependency loss is also used to penalize the class mismatch predicted by the two classifiers, thereby improving the classification accuracy affected by this problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a framework diagram of the acoustic scene classification network model of the present invention. DETAILED DESCRIPTION

[0021] The following describes embodiments of the present invention in detail. Examples of the illustrated embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0022] 1. Project Description

[0023] There are certain differences between the acoustic scene classification problem and the traditional classification problem. In traditional classification problems, there is no relationship between categories, and each is independent of each other. In acoustic scene classification, there is a hierarchical relationship between categories, that is, the relationship between parent and child classes. For example, the urban acoustic scene 2020 mobile development dataset officially provided by DCASE (Detection and Classification of AcousticScenes and Events) contains 10 specific acoustic scene categories (subcategories), such as airports, shopping malls, subway stations, and pedestrian streets; and these 10 subcategories belong to 3 parent categories. For example, "airports", "shopping malls", and "subway stations" in the subcategories belong to the parent category "indoor". Existing acoustic scene classification technologies often ignore this hierarchical relationship when designing models. The present invention notes that how to take this hierarchical relationship into account in model design is an important technical breakthrough direction for improving the accuracy of acoustic scene classification.

[0024] like Figure 1The figure shows the framework of the acoustic scene classification network model of the present invention, which mainly includes an acoustic scene feature learning module and a hierarchical information fusion module. The acoustic scene feature learning module serves as the backbone network for extracting features from audio.

[0025] In this embodiment, the backbone network adopts the Trident ResNet (high, medium and low frequency path separation and late fusion network) proposed by Suh et al. For the audio file to be recognized, the FBank acoustic features in the audio file are extracted and used as the input of the backbone network. The backbone network contains three parallel residual networks; in the backbone network: first, the input FBank acoustic features are divided into three parts: low frequency, medium frequency and high frequency according to the frequency axis; then the above three parts are respectively sent to the three residual networks; in each residual network, the frequency axis is not downsampled to ensure that the frequency axis dimension remains unchanged; in addition, each residual network also adopts hollow convolution to expand the receptive field; finally, the outputs of the three residual networks are spliced ​​in a frequency axis split manner to obtain the features. The above Trident ResNet can be found in the literature: Suh S, Park S, Jeong Y, et al. Designing acoustic scene classification models with CNN variants [J]. Tech. Rep., DCASE2020 Challenge, 2020.

[0026] The hierarchical information fusion module is connected after the acoustic scene feature learning module, and the extracted features are sent to the hierarchical information fusion module as input. Two parallel convolution layers are provided on the input side of the hierarchical information fusion module, namely the parent class convolution layer and the child class convolution layer. Both the parent class convolution layer and the child class convolution layer are 1D convolution layers. The input features are subjected to 1D convolution by the parent class convolution layer and the child class convolution layer respectively to obtain the parent class representation and the child class representation. A parent class classifier and a child classifier are provided after the parent class convolution layer and the child class convolution layer; wherein the parent class representation is sent to the parent class classifier to obtain the predicted parent class category, and the parent class representation and the child class representation are spliced ​​and fused and then sent to the child class classifier together to obtain the predicted subclass category. In the above manner, the network model is forced to learn the hierarchical structural relationship information between acoustic scene categories.

[0027] The last layer of the hierarchical information fusion module is equipped with a loss function layer. During network model training, the predicted parent and child categories and the corresponding true category labels are fed into the loss function layer, and the network model is optimized by minimizing the loss function value. The loss function is:

[0028]

[0029] in is the joint loss, is the cross entropy loss of the parent class, is the cross entropy loss of subclasses, is the layer-dependent loss, and α is the weight of the layer-dependent loss.

[0030] The network model of the present invention is provided with two classifiers (parent class classifier and child classifier), so that two classification results (parent class category and child class category) will be obtained. Obviously, the affiliation between the parent class and the child class is fixed, that is, when a specific child class category is known, the corresponding parent class category is also uniquely determined. However, the prediction of the network model cannot guarantee such affiliation, so there will be a situation where the predicted parent class category does not match the predicted child class category. To address this problem, the present invention introduces hierarchical dependency loss to punish the "mismatch situation", thereby forcing the parent class category and child class category predicted by the network model to conform to their original affiliation. The hierarchical dependency loss function is:

[0031]

[0032] in Indicates whether the affiliation between the parent class and the child class is correct. If it is correct, The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the parent class is correct. The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the subclass is correct, where correct The value is 1, which is incorrect. The value is 0; ploss is the dependency penalty and is set to a constant.

[0033] 2. Testing

[0034] 1. Dataset Description

[0035] Testing was conducted using the Urban Acoustic Scenes 2020 Mobile Development Dataset provided by DCASE. This dataset contains 23,040 recordings from 10 different acoustic scenes in 10 European cities. These include: airports, shopping malls, subway stations, pedestrian streets, public squares, traffic streets, parks, trams, buses, and subways. All recordings were made using three different devices: devices A, B, and C. Device A is the primary recording device, consisting of a Soundman OKM II Klassik / studio A3, electret binaural microphones, and a Zoom F8 recorder with a 48kHz sampling rate and 24-bit resolution. Device B is a Samsung Galaxy S7, and device C is an iPhone SE. Based on the data recorded with these three devices, synthetic data was created for six mobile devices: S1, S2, S3, S4, S5, and S6. Data from devices S4, S5, and S6 will not be used for training. Of the 23,040 recordings, device A has the most data, with 14,400. The remaining eight devices each have 1,080 recordings. Each audio data entry is 10 seconds long, and all data is resampled to 44.1 kHz.

[0036] 2. Audio Data Processing

[0037] The logarithmic Mel spectrogram (FBank acoustic features) is extracted from the audio file as the input of the network model of the present invention. First, each audio data is pre-emphasized, framed, and windowed. Then a short-time Fourier transform of 2048 FFT points is used. Each spectrum is compressed to a 256 Mel frequency scale, and then the logarithm is taken to obtain a logarithmic Mel spectrogram. In addition, deltas and delta-deltas are calculated from the logarithmic Mel spectrogram and superimposed along the channel dimension to obtain the input of the final model.

[0038] 3. Backbone network settings

[0039] The backbone network consists of three residual network paths. The specific network structure of each path is shown in Table 1. Each residual path first performs a convolution with a stride of 1 in the frequency dimension and a stride of 2 in the time dimension. It is then connected to 12 residual blocks. Each residual block contains two convolutions, each preceded by batch normalization and activation with a linear rectification function. The convolution kernel size is 3×3.

[0040] In the time dimension, a stride of 2 is used in the first convolution of the fourth, seventh, and tenth residual blocks to downsample the time dimension. No downsampling is performed in the frequency dimension; all convolutions have a stride of 1. Furthermore, in the frequency dimension, dilated convolutions with a dilation rate of 2 are used in the first convolution of the second, third, fifth, sixth, eighth, ninth, eleventh, and twelfth residual blocks. Dilated convolutions can expand the receptive field without changing the size of the feature map, thereby maintaining the size of the frequency dimension within the entire embedding network and ensuring that the outputs of the three residual paths can be reassembled according to the original splitting method.

[0041] Table 1: Residual network structure configuration

[0042]

[0043] 4. Other Experimental Settings

[0044] In the test, the network model was built using the Tensorflow and Keras deep learning frameworks. Backpropagation and stochastic gradient descent with momentum of 0.9 were used, and the batch size was set to 16. The weight α was set to 0.8. Each training cycle lasted 510 cycles, and the learning rate was reset to the maximum value of 0.1 after 2, 6, 14, 30, 62, 126, and 254 cycles, and then decayed to 1×10 according to the cosine mode. -5 .

[0045] 5. Test Results

[0046] The network model of this invention achieved a classification accuracy of 93.2% for the parent class. Regarding the classification accuracy of the child class, the ablation test in this test showed that without the hierarchical dependency loss, the accuracy was 74.2%, while with the hierarchical dependency loss, the accuracy increased to 74.5%. This shows that the mismatch between the classification results of the two classifiers does affect the performance of the model, and the introduction of the hierarchical dependency loss can improve this problem.

[0047] Table 2 below shows the comparative test results of the network model of the present invention and other existing network models. The official baseline system is the acoustic scene classification network model officially provided by DCASE, with an average classification accuracy of 54.1%. The best effect is achieved for device A because device A is the main recording device and has the most audio data (approximately 75%). In addition, device A is a professional recording device and has the best quality recorded audio data. However, the official baseline system has a serious performance degradation on other devices, especially on the invisible devices S4, S5, and S6. The network model proposed by Suh et al. can be found in the literature Suh S, Park S, Jeong Y, et al. Designing acoustic scene classification models with CNN variants [J]. Tech. Rep., DCASE2020 Challenge, 2020; the network model proposed by Liu et al. can be found in the literature Liu J. Acoustic scene classification with residual networks and attention mechanism [J]. Tech. Rep., DCASE2020 Challenge, 2020; the network model proposed by Gao et al. can be found in the literature

[24] . Gao W, McDonnell M, UniSAS. Acoustic scene classification using deep residual networks with focal loss and mild domain adaptation [J]. Tech. Rep., DCASE2020 Challenge, 2020.

[0048] Based on the comparative test results with the existing network models, the network model proposed in the present invention has the highest average classification accuracy, and the classification accuracy for each specific recording device is also higher than that of other network models.

[0049] Table 2: Comparative test of the present invention and other methods

[0050]

[0051] The present invention is not limited to the above-mentioned embodiments. Any obvious improvement, replacement or modification that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. An acoustic scene classification network model based on hierarchical information fusion, characterized by: It includes an acoustic scene feature learning module and a hierarchical information fusion module. The acoustic scene feature learning module extracts features from the audio and inputs them into the hierarchical information fusion module. The hierarchical information fusion module is equipped with a parent classifier and a child classifier. The hierarchical information fusion module first performs 1D convolution on the input features in two ways to obtain parent class representation and child class representation respectively. The parent class representation is then input into the parent class classifier to obtain the predicted parent class category. The parent class representation and child class representation are concatenated and fused and then input into the child class classifier to obtain the predicted child class category. The hierarchical information fusion module is provided with a loss function layer, and its loss function is: in is the joint loss, is the cross entropy loss of the parent class, is the cross entropy loss of subclasses, is the hierarchical dependency loss, α is the weight of the hierarchical dependency loss; the hierarchical dependency loss function is: in Indicates whether the affiliation between the parent class and the child class is correct. If it is correct, The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the parent class is correct. If it is correct The value is 1, which is incorrect. The value is 0; Indicates whether the prediction result of the subclass is correct. If it is correct The value is 1, which is incorrect. The value is 0; ploss is the dependency penalty and is set to a constant.

2. The acoustic scene classification network model according to claim 1, characterized in that: The acoustic scene feature learning module adopts Trident ResNet.

3. The acoustic scene classification network model according to claim 2, characterized in that: The Trident ResNet contains three residual network paths; in each residual path: a convolution is performed first, with a step size of 1 in the frequency dimension and a step size of 2 in the time dimension, followed by 12 residual blocks, each of which contains two convolutions. Batch normalization is performed before each convolution and activated with a linear rectification function. The size of the convolution kernel is 3×3.

4. The acoustic scene classification network model according to claim 1, characterized in that: The network model was trained with a weight α set to 0.

8. Each training session lasted 510 cycles, and the learning rate was reset to a maximum value of 0.1 after 2, 6, 14, 30, 62, 126, and 254 cycles, and then decayed to 1×10 -5 .

5. The acoustic scene classification network model according to claim 1, characterized in that: The input of the acoustic scene feature learning module is a logarithmic Mel spectrogram.

6. The acoustic scene classification network model according to claim 5, characterized in that: For the audio to be recognized, the audio data is first pre-emphasized, framed, and windowed. Then, a short-time Fourier transform with 2048 FFT points is used to compress each spectrum to a 256-Mel frequency scale. Finally, the logarithm is taken to obtain a logarithmic Mel spectrogram. Before the logarithmic Mel spectrogram is input into the acoustic scene feature learning module, deltas and delta-deltas are first calculated and superimposed along the channel dimension.

Images