Acoustic scene classification method and system based on model fusion, and electronic equipment

Abstract

The invention relates to the technical field of acoustic scene classification scheme design, in particular to an acoustic scene classification method and system based on model fusion and electronic equipment. According to the method, three classification models are built, the three models built by the method are fused for use, ten types of acoustic scenes are further divided into three major types, and a three-classifier is built for learning upper-layer shared features among similar scenes. Data enhancement and model fusion strategies are introduced into a three-classifier and a very-class classifier, a multi-level acoustic scene classification model based on model fusion is established, and the output of the very-class classifier and the output of the three-classifier are fused in the later period to improve the overall performance of the model. The accuracy and the reliability of acoustic scene classification are improved to a great extent, and the generalization performance and the classification accuracy of the neural network model are improved under the cross-equipment condition for the acoustic scene classification problem.

Description

Technical Field

[0001] The present invention relates to the technical field of acoustic scene classification scheme design, and in particular to an acoustic scene classification method and system based on model fusion, and electronic equipment. Background Art

[0002] The purpose of acoustic scene classification is to enable the machine to judge the scene through the audio collected by its own microphone. Acoustic scene classification technology can be applied to devices such as intelligent monitoring systems, smart phones, and smart headphones. Early acoustic scene classification work relied on traditional machine learning methods. In recent years, with the continuous research and progress of deep learning algorithms, a variety of deep learning methods have been used to learn high-level representations of audio features and achieved better performance than traditional machine learning methods. However, the current research field of acoustic scene classification also faces some problems. The scale and diversity of existing scene audio datasets are limited, and there is a lack of large-scale comprehensive datasets. In order for neural networks to achieve high system performance, large-scale datasets are often required for model training. In addition, the existing datasets are collected by multiple devices, and there are differences in the characteristics of the devices themselves. The amount of data for different devices is also unbalanced. Even unknown devices that are not included in the training set may appear in the test set, which places high demands on the generalization performance of the model. How to improve the generalization performance and classification accuracy of neural network models in cross-device situations for acoustic scene classification problems is an urgent problem to be solved in existing technologies.

[0003] Therefore, the prior art needs to be further developed. Summary of the invention

[0004] The purpose of the present invention is to overcome the above technical deficiencies and provide an acoustic scene classification method and system, and electronic device based on model fusion to solve the problems existing in the prior art.

[0005] To achieve the above technical objectives, according to a first aspect of the present invention, the present invention provides an acoustic scene classification method based on model fusion, comprising: S100, obtaining a Mel feature spectrogram of a noise source speech signal; dividing the obtained Mel feature spectrogram into a low frequency band and a high frequency band according to a preset frequency division threshold, and then building a Resnet model based on high and low frequency separation; building a FCNN model based on channel attention based on the obtained Mel feature spectrogram; in the process of building the FCNN model based on channel attention, limiting the downsampling of the frequency axis, and then building a FCNN model with limited frequency downsampling; S200, the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention are fused, and then the three classifiers of the acoustic scene are trained; the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling are fused, and then the ten classifiers of the acoustic scene are trained; S300, fusing the output layers of the three classifiers and the ten classifiers to obtain a target acoustic scene classification model, and performing acoustic scene classification using the target acoustic scene classification model.

[0006] Specifically, the classification results of the three classifiers of the acoustic scene include: Indoor scenes, outdoor scenes and traffic scenes.

[0007] Specifically, the classification results of the ten classifiers of the acoustic scene include: Airports, shopping malls, subway stations, pedestrian streets, public squares, street traffic, trams, buses, subways, city parks.

[0008] Specifically, the output layers of the three classifiers and the ten classifiers are fused to obtain a target acoustic scene classification model, and the target acoustic scene classification model is used to perform acoustic scene classification, including: set up and Respectively represent the three major categories and ten subcategories of acoustic scene sets, and are respectively used and Represents the classification result output of the three-classifier and the ten-classifier, then for the input , the final prediction classification result can be expressed as follows: in, express yes is a superset of ; therefore, the entire classification system predicts that the input audio segment belongs to The probability of is equal to the probability given by the three classifiers The probability of The product of the probabilities of .

[0009] Specifically, the acquired Mel feature spectrogram is divided into a low frequency band and a high frequency band according to a preset frequency division threshold, and then a Resnet model based on high and low frequency separation is built, including: The Mel feature spectrograms of the low-frequency band and the high-frequency band are respectively input into the first residual network and the second residual network, and the outputs of each channel of the first residual network and the second residual network are cascaded on the frequency axis. After the cascading, the contribution value of each channel of the first residual network and the second residual network is weighted using the global average pooling layer and the Softmax activation function, thereby obtaining a Resnet model based on high and low frequency separation.

[0010] Specifically, the first residual network and the second residual network both include a first stack layer, a second stack layer, a third stack layer and a fourth stack layer connected in sequence, the first stack layer includes a first residual block, the second stack layer includes a second residual block and a first residual block connected in sequence, the second residual block includes a downsampling path, the downsampling path includes an average pooling layer and a zero padding layer, which are used to perform a downsampling operation. After the downsampling operation, the output result of the downsampling path and the output result of the residual path are spliced ​​to match the output dimension during subsequent convolution operations.

[0011] Specifically, the FCNN model based on the channel attention is built based on the acquired Mel feature spectrogram, including: The Mel feature spectrum is input into the first convolution block, which contains two convolutional layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 2×2, followed by zero padding. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each layer is ReLU, and batch normalization is performed after each layer. The last step of the first convolution block is to perform a maximum pooling operation, and the output result of the first convolution block is input into the second convolution block. The second convolution block contains two convolutional layers. The difference between the second convolution block and the first convolution block is that the second convolution block Both convolutional layers use a convolution kernel of size 3×3 and a stride of 1×1. The output of the second convolutional block is input into the third convolutional block. The third convolutional block has four convolutional layers. Each convolutional layer uses a convolution kernel of size 3×3 and a stride of 1×1. Each convolutional layer uses zero padding, batch normalization, and ReLU function operations. A dropout layer is used after the first three convolutional layers to prevent overfitting. The last step of the third convolutional block performs a 2×2 maximum pooling operation, and finally undergoes a single-layer convolution operation. Its output feature map will be used as the input of the channel attention module.

[0012] Specifically, the FCNN model for limiting frequency downsampling includes: The Mel feature spectrum is input into the first convolution block, which includes two convolution layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 1×2. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each convolution layer is ReLU, and batch normalization is performed after each convolution layer. The output result of the first convolution block is input into the second convolution block, which includes two convolution layers. The size of the convolution kernel of the two convolution layers in the second convolution block is 3×3 and the step size is 1×1. The activation function used after each convolution layer is also ReLU, and batch normalization is also performed after each layer. The output result of the second convolution block is input into the third convolution block. The third convolution block has four convolution layers, and each convolution layer uses a convolution kernel of size 3×3 and a 1× 1, and batch normalization and ReLU function are used. Dropout of 0.3 is used after the first and third convolutional layers to prevent overfitting. The output result of the third convolutional block is input to the fourth convolutional block. The fourth convolutional block includes two convolutional layers. Both convolutional layers of the fourth convolutional block use convolution kernels with a size of 3×3 and a step size of 1×1. Batch normalization, ReLU function activation and a dropout layer with a value of 0.5 are also performed after each convolutional layer of the fourth convolutional block. In the ten convolutional layers of the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block arranged in the order of calculation, the second and fourth convolutional layers are followed by maximum pooling of 3×3 and a step size of 2×2, while the sixth and eighth convolutional layers are followed by maximum pooling of 3×3 and a step size of 1×2.

[0013] According to a second aspect of the present invention, there is provided an acoustic scene classification system based on model fusion, comprising: An acquisition module, used for acquiring a Mel feature spectrum of a noise source speech signal; A control module is used to divide the acquired Mel feature spectrogram into a low frequency band and a high frequency band according to a preset frequency division threshold, and then build a Resnet model based on high and low frequency separation; build an FCNN model based on channel attention based on the acquired Mel feature spectrogram; in the process of building the FCNN model based on channel attention, limit the downsampling of the frequency axis, and then build an FCNN model with limited frequency downsampling; used to fuse the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention, and then train to obtain a three-classifier of the acoustic scene; fuse the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling, and then train to obtain a ten-classifier of the acoustic scene; used to fuse the output layers of the three classifiers and the ten-classifiers to obtain a target acoustic scene classification model, and use the target acoustic scene classification model to classify the acoustic scene.

[0014] According to a third aspect of the present invention, there is provided an electronic device, comprising: a memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned acoustic scene classification method based on model fusion is implemented.

[0015] Beneficial effects: The present invention builds three classification models, and fuses the three models built by the present invention for use, further dividing the ten types of acoustic scenes into three categories, and constructing a three-classifier for learning upper-level shared features between similar scenes. Data enhancement and model fusion strategies are introduced in both the three-classifier and the ten-classifier, and a multi-level acoustic scene classification model based on model fusion is built. The present invention fuses the outputs of the ten-classifier and the three-classifier in the later stage to improve the overall performance of the model, which greatly improves the accuracy and reliability of acoustic scene classification, and realizes the improvement of the generalization performance and classification accuracy of the neural network model for acoustic scene classification problems in cross-device situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Schematic diagram of the Resnet model structure based on high and low frequency separation.

[0017] Figure 2 Schematic diagram of the fully connected neural network model structure based on channel attention.

[0018] Figure 3 Schematic diagram of the fully connected neural network structure with limited frequency downsampling.

[0019] Figure 4 It is the hierarchical relationship between ten types of acoustic scenes.

[0020] Figure 5 Schematic diagram of the multi-level acoustic scene classification model structure.

[0021] Figure 6 Schematic diagram of a multi-level acoustic scene classification model based on model fusion.

[0022] Figure 7 is the confusion matrix of the M1 model.

[0023] Figure 8 is the confusion matrix of the M2 model.

[0024] Fig. 9 is the confusion matrix of the M3 model.

[0025] Fig.10 It is the confusion matrix of M1+M2+M3 model.

[0026] Fig.11This is the confusion matrix of the multi-level acoustic scene classification model based on model fusion proposed in the present invention.

[0027] Fig.12 A schematic diagram of the flow of the acoustic scene classification method based on model fusion proposed in the present invention.

[0028] Fig.13 Schematic diagram of the system composition of the acoustic scene classification system based on model fusion proposed in the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is clearly and completely described below in conjunction with the accompanying drawings of the present invention. Based on the embodiments in this application, other similar embodiments obtained by ordinary technicians in this field without making creative work should all fall within the scope of protection of this application. In addition, the directional words mentioned in the following embodiments, such as "up", "down", "left", "right", etc., are only reference to the directions of the accompanying drawings. Therefore, the directional words used are used to illustrate rather than limit the invention.

[0030] The present invention will be further described below in conjunction with the accompanying drawings and preferred embodiments.

[0031] See also Figure 1-Figure 13 The present invention provides an acoustic scene classification method based on model fusion, comprising: S100, obtaining a Mel feature spectrum of a noise source speech signal; dividing the obtained Mel feature spectrum into a low frequency band and a high frequency band according to a preset frequency division threshold, and then building a Resnet model based on high and low frequency separation; building a channel attention-based FCNN model based on the obtained Mel feature spectrum; in the process of building the channel attention-based FCNN model, limiting the downsampling of the frequency axis, and then building a FCNN model with limited frequency downsampling.

[0032] Specifically, the acquired Mel feature spectrogram is divided into a low frequency band and a high frequency band according to a preset frequency division threshold, and then a Resnet model based on high and low frequency separation is built, including: The Mel feature spectrograms of the low-frequency band and the high-frequency band are respectively input into the first residual network and the second residual network, and the outputs of each channel of the first residual network and the second residual network are cascaded on the frequency axis. After the cascading, the contribution value of each channel of the first residual network and the second residual network is weighted using the global average pooling layer and the Softmax activation function, thereby obtaining a Resnet model based on high and low frequency separation.

[0033] Specifically, the first residual network and the second residual network both include a first stack layer, a second stack layer, a third stack layer and a fourth stack layer connected in sequence, the first stack layer includes a first residual block, the second stack layer includes a second residual block and a first residual block connected in sequence, the second residual block includes a downsampling path, the downsampling path includes an average pooling layer and a zero padding layer, which are used to perform a downsampling operation. After the downsampling operation, the output result of the downsampling path and the output result of the residual path are spliced ​​to match the output dimension during subsequent convolution operations.

[0034] Specifically, the FCNN model based on the channel attention is built based on the acquired Mel feature spectrogram, including: The Mel feature spectrum is input into the first convolution block, which contains two convolutional layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 2×2, followed by zero padding. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each layer is ReLU, and batch normalization is performed after each layer. The last step of the first convolution block is to perform a maximum pooling operation, and the output result of the first convolution block is input into the second convolution block. The second convolution block contains two convolutional layers. The difference between the second convolution block and the first convolution block is that the second convolution block Both convolutional layers use a convolution kernel of size 3×3 and a stride of 1×1. The output of the second convolutional block is input into the third convolutional block. The third convolutional block has four convolutional layers. Each convolutional layer uses a convolution kernel of size 3×3 and a stride of 1×1. Each convolutional layer uses zero padding, batch normalization, and ReLU function operations. A dropout layer is used after the first three convolutional layers to prevent overfitting. The last step of the third convolutional block performs a 2×2 maximum pooling operation, and finally undergoes a single-layer convolution operation. Its output feature map will be used as the input of the channel attention module.

[0035] Specifically, the FCNN model for limiting frequency downsampling includes: The Mel feature spectrum is input into the first convolution block, which includes two convolution layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 1×2. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each convolution layer is ReLU, and batch normalization is performed after each convolution layer. The output result of the first convolution block is input into the second convolution block, which includes two convolution layers. The size of the convolution kernel of the two convolution layers in the second convolution block is 3×3 and the step size is 1×1. The activation function used after each convolution layer is also ReLU, and batch normalization is also performed after each layer. The output result of the second convolution block is input into the third convolution block. The third convolution block has four convolution layers, and each convolution layer uses a convolution kernel of size 3×3 and a 1× 1, and batch normalization and ReLU function are used. Dropout of 0.3 is used after the first and third convolutional layers to prevent overfitting. The output result of the third convolutional block is input to the fourth convolutional block. The fourth convolutional block includes two convolutional layers. Both convolutional layers of the fourth convolutional block use convolution kernels with a size of 3×3 and a step size of 1×1. Batch normalization, ReLU function activation and a dropout layer with a value of 0.5 are also performed after each convolutional layer of the fourth convolutional block. In the ten convolutional layers of the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block arranged in the order of calculation, the second and fourth convolutional layers are followed by maximum pooling of 3×3 and a step size of 2×2, while the sixth and eighth convolutional layers are followed by maximum pooling of 3×3 and a step size of 1×2.

[0036] S200, the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention are fused, and then the three classifiers of the acoustic scene are trained; the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling are fused, and then the ten classifiers of the acoustic scene are trained.

[0037] Specifically, the classification results of the three classifiers of the acoustic scene include: Indoor scenes, outdoor scenes and traffic scenes.

[0038] Specifically, the classification results of the ten classifiers of the acoustic scene include: Airports, shopping malls, subway stations, pedestrian streets, public squares, street traffic, trams, buses, subways, city parks.

[0039] S300, fusing the output layers of the three classifiers and the ten classifiers to obtain a target acoustic scene classification model, and performing acoustic scene classification using the target acoustic scene classification model.

[0040] Specifically, the output layers of the three classifiers and the ten classifiers are fused to obtain a target acoustic scene classification model, and the target acoustic scene classification model is used to perform acoustic scene classification, including: set up and Respectively represent the three major categories and ten subcategories of acoustic scene sets, and are respectively used and Represents the classification result output of the three-classifier and the ten-classifier, then for the input , the final prediction classification result can be expressed as follows: in, express yes is a superset of ; therefore, the entire classification system predicts that the input audio segment belongs to The probability of is equal to the probability given by the three classifiers The probability of The product of the probabilities of .

[0041] The workflow of the present invention is described below by means of specific examples: Step 1: First, a neural network module for high- and low-frequency separation is proposed, namely, the Resnet model based on high- and low-frequency separation. At the front end of the model, the input features are divided into two branches, low-frequency band and high-frequency band, according to the frequency, and subsequent operations are processed separately, and different strategies are adopted for the time axis and frequency axis. The structure of the Resnet model based on high- and low-frequency separation is as follows: Figure 1As shown in , where ⊕ represents element-by-element addition. The input of the entire model is a concatenated feature spectrogram of size 431×128×3, where 431 is the number of points on the horizontal axis of the image, 128 is the number of points on the vertical axis of the image, and 3 is the RGB value of the Mel spectrogram. There are 128 frequency bands in total, which are divided into two parts at the beginning of the network, a low frequency band of 0 to 63 and a high frequency band of 64 to 127. These two branches are subsequently processed by a residual network with a total of 17 convolutional layers. There are four stacks in total, with two residual blocks in each stack, where batch normalization and ReLU function activation are performed before each convolutional layer, and all convolution kernels in these layers are 3×3 in size. Except for the first stack, the first residual block in the remaining stacks will be downsampled by average pooling, with a pooling layer size of 3×3 and a step size of 1×2. After downsampling, each residual path is padded with zeros in depth, and then the zero-padded residual path is concatenated with the calculation results of the original path to match the input dimension of the subsequent convolution operation. As the depth of the four-layer residual stack increases, the number of input channels of each layer also increases, which are 24, 48, 96, and 192 respectively. After the stack calculation of these 17 convolutional layers, the outputs of the two branch paths are cascaded on the frequency axis to restore the 128 frequency dimensions of the initial input. Then, after two convolutional layers with a convolution kernel size of 1×1, each convolutional layer is followed by a batch normalization layer. The feature vector is finally weighted by the global average pooling layer and the Softmax activation function to facilitate the subsequent classification of the acoustic scene. This model combines the ideas of frequency band splitting and residual connection, and extracts and learns features in the input data by processing low-frequency and high-frequency information separately and using residual blocks at different levels. Finally, the model converts the features into a fixed-length vector through global average pooling, and outputs the probability that the audio to be predicted belongs to each scene category through the Softmax activation function. The above frequency band processing method is helpful for model training and generalization when processing audio data with frequency information.

[0042] It should be noted here that the input size of the neural network module for high and low frequency separation is 423×128×3, with 423 time segments and 128 frequency segments. A scene-representative audio signal may appear at any time point in the 10-second audio data with the same probability. Therefore, in order to enable the network to learn global time features in all time samples, a convolutional layer with a step size of 2 is used to periodically downsample on the time axis. However, a completely different strategy is adopted for the frequency axis. In order to obtain complete frequency features, we do not perform downsampling operations. Therefore, during the learning process of the entire network, the size of the frequency dimension in the feature map of the high and low frequency branches is always maintained at 64. The high and low frequency branches are synchronously calculated in parallel in the network, and they are cascaded and combined on the frequency axis in the later fusion stage of the entire network to form the final audio features.

[0043] Step 2: A fully connected neural network model based on channel attention, namely the FCNN model based on channel attention, is proposed. Its model structure is as follows: Figure 2 As shown in the figure, the input is still a concatenated spectrogram feature of size 431×128×3, which will then be operated on by three convolution blocks in sequence. The first convolution block contains two convolution layers. The first convolution uses a 5×5 convolution kernel and a 2×2 step size, followed by zero padding. The second convolution uses a 3×3 convolution kernel and a 1×1 step size. The activation function used in each layer is ReLU, and batch normalization is performed after each layer. The maximum pooling operation is performed at the end of the convolution block. The structure of the second convolution block is similar to that of the first convolution block, except that both of its two convolution layers use a 3×3 convolution kernel and a 1×1 step size. The third convolution block has four convolution layers, each of which uses a 3×3 convolution kernel and a 1×1 step size, and uses zero padding, batch normalization, and ReLU function operations. A dropout layer is used after the first three convolution layers to prevent overfitting. A 2×2 maximum pooling operation is performed at the end of the convolution block. As the depth of these three convolution blocks increases, the number of channels in their convolutional layers also increases, which are 24, 48, and 96 respectively. Finally, a single-layer convolution operation with a kernel size of 1 and a stride of 1 will be performed, and its output feature map will be used as the input of the channel attention module. The channel attention module enhances the model's perception of different channels, and then reduces the features to one dimension through global average pooling. Finally, the fully connected layer passes through the Softmax activation function, and the result is used for multi-category classification.

[0044] Step 3: Through experimental analysis, it can be found that reducing the maximum pooling on the frequency axis may have a certain improvement effect on the model performance. Therefore, the present invention deploys a network architecture that is very similar to the above-mentioned fully connected network but has 11 convolutional layers, and its structure is as follows Figure 3As shown. The network structure undergoes a total of four convolution blocks. The first convolution block contains two convolution layers. The convolution kernel size of the first convolution layer is 5×5 and the step size is 1×2. The convolution kernel size of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each layer is ReLU, and batch normalization is performed after each layer. The convolution kernel size of the two convolution layers in the second convolution block is 3×3 and the step size is 1×1. The activation function used after each convolution layer is also ReLU, and batch normalization is also performed after each layer. The third convolution block has four convolution layers. Each convolution layer uses a convolution kernel size of 3×3 and a step size of 1×1. Batch normalization and ReLU functions are used. Dropout of 0.3 is used after the first and third convolution layers to prevent overfitting. In the last convolutional block, both convolutional layers use a 3×3 kernel and a 1×1 stride. After each convolutional layer, batch normalization, ReLU activation, and a dropout layer with a value of 0.5 are also performed. As the depth of these four convolutional blocks increases, the number of channels in the convolutional layers also increases, which are 14, 24, 48, and 96 respectively. The difference from the network structure in the first section is that after the second and fourth convolutional layers, a maximum pooling of 3×3 and a stride of 2×2 is applied, while after the sixth and eighth convolutional layers, a maximum pooling of 3×3 and a stride of 1×2 is applied, so that the downsampling of the frequency axis is properly controlled.

[0045] Step 4: The ten types of acoustic scenes included in the TAU Urban Acoustic Scene 2020 Mobile Development Dataset can be divided into two layers according to their relevance, such as Figure 4 As shown in Figure 1, airports, shopping malls and subway stations can be classified as indoor scenes, pedestrian streets, public squares, street traffic and urban parks can be classified as outdoor scenes, and trams, buses and subways can be classified as traffic scenes. Based on this, a multi-level acoustic scene classification system is constructed, as shown in Figure 1. Figure 5 As shown, here is set and Respectively represent the three major categories and ten subcategories of acoustic scene sets, and are respectively used and represents the classification result output of the three-classifier and the ten-classifier. Then for the input , the final prediction classification result can be expressed as follows: in mean Can be For example, the outdoor scene is a superset of the pedestrian street, public square, street traffic, and city park classes. Therefore, the probability that the entire classification system predicts that the input audio clip belongs to the pedestrian street scene is equal to the probability given by The probability of an outdoor scene given by The product of the probability of a given pedestrian street scene. In this way, multi-level features are integrated to improve the classification accuracy of the model. It includes a three-classifier and a ten-classifier. Finally, the outputs of the two classifiers are fused through the above method, and the classification accuracy is further improved by correcting the ten-classification results with the three-classification results. The first classifier is a three-classifier, which divides the input audio scene into three categories: indoor, outdoor, and traffic. The second classifier is a ten-class classifier, which classifies the input audio according to the original ten categories of acoustic scene labels. The network model used in the three-classifier is basically the same as the ten-classifier, the only difference is that the final classification by the Softmax function is divided into three categories instead of ten categories. Different models have different expressive abilities on different data. When multiple models are fused, the fused model can complement each other, avoid the shortcomings of a single model, and can better adapt to different data. At the same time, fusing multiple models can reduce the impact caused by sample noise, greatly reduce the variance of the model, prevent overfitting, and improve the accuracy and generalization ability of the model. Therefore, the network model constructed by the present invention is fused, and this strategy is applied to the above-mentioned multi-level acoustic scene classification model, and finally a multi-level acoustic scene classification model based on model fusion is constructed, and its schematic diagram is shown in FIG. Figure 6 As shown in the figure, the model fusion strategy is applied to both the three-classifier and the ten-classifier. The ten-classifier uses all three models mentioned above for fusion, and the three-classifier uses the Resnet model based on high- and low-frequency separation and the fully connected neural network based on channel attention for fusion.

[0046] Step 5: Result analysis. The performance comparison of each model is shown in Tables 1 and 2, which list the accuracy and cross entropy loss of the best model trained in each case. Table 1 shows the classification accuracy of various models for each acoustic scene. The classification accuracy of the models trained using the three networks built in this chapter alone is greatly improved compared with the baseline model. Among them, the average accuracy of the M2 model is the highest, which is 76.9%, which is 9.5% higher than the baseline model. The classification accuracy of the three models M1, M2, and M3 in the airport, subway station, public square, street traffic, bus, and city park scenes is not much different. In all ten scenes, the three models generally have poor classification effects on the airport, pedestrian street, and public square scenes. All three scenes have the characteristics of large traffic, which causes certain interference to the feature learning of the model. The performance of the M1 model in shopping malls and trams is significantly worse than that of M2 and M3. The recognition accuracy of M2 in subway scenes is significantly lower than that of M1 and M3, while the recognition accuracy of M3 in pedestrian street scenes is significantly lower than that of M1 and M2. This shows that networks with different structures have different focuses when learning features of different scenes, which also leads to significant differences in the classification effects on individual classes. At this time, the model fusion strategy is needed to make up for the shortcomings of the single model. As can be seen from the table, after fusing M1, M2 and M3, the classification accuracy has been improved in most scenes. The classification accuracy of the shopping mall scene has been improved the most compared with the best performance of the single model, increasing by 3%. In the pedestrian street and subway scenes, although the classification accuracy has decreased compared with the best single model, it has been significantly improved for the models with poor classification effects in this scene. Through model fusion, the classification accuracy of the model for 10 categories of scenes is more balanced, avoiding the situation where the classification effect of a single model is extremely poor in individual classes. The average classification accuracy under the model fusion strategy is 79.2%, which is 2.3% higher than the best single model. Finally, the accuracy of the multi-level acoustic scene classification model based on model fusion in each scene is improved compared with the model fusion strategy, among which the classification accuracy of the airport and subway scenes has been greatly improved. The final classification accuracy is 81.8%, which is 2.6% higher than the model fusion strategy. Table 2 shows the classification accuracy of various models for each device. The models trained using the three networks built in this chapter have better classification accuracy for the data collected by each device than the baseline model, and have a significant improvement for unknown devices S4, S5, and S6. Since the data of device A accounts for a large proportion in the training set and its features dominate, the classification accuracy of the three models for device A data is the highest among all devices. Devices S4, S5, and S6 are unknown devices not included in the training set, and their classification accuracy is relatively low. However, there are also individual cases where the classification accuracy of these unknown devices is better than that of some known devices. This shows that the spectrum correction method can improve the generalization performance of the model across devices to a certain extent.After fusing M1, M2 and M3, the classification accuracy was improved on all devices. The classification accuracy of the data collected by device S1 was the largest improvement compared to the best performance of a single model, increasing by 3.6%. This also shows that the model fusion strategy has played a role in complementing each other's strengths and weaknesses and making up for the defects of a single model. Finally, the multi-level acoustic scene classification model based on model fusion has improved the classification accuracy of the audio of each device compared to the model fusion strategy, among which the improvement for unknown devices S4, S5 and S6 is greater. As can be seen from Table 4-5, the three-classification model has a high classification accuracy for each device, and the accuracy gap between different devices is small. This is because indoor, outdoor and traffic are a collection of data for multiple sub-scenes, and their data volume is greatly improved compared to a single sub-scene, and the classification model can learn richer features from them. This result also shows that the multi-level structure has also improved the generalization ability of the model in the cross-device situation to a certain extent. Figure 7-Figure 11 The confusion matrix of various classification models is shown. Figure 7 , Figure 8 , Fig. 9 The confusion matrices of the M1, M2, and M3 models are shown in Figure 1. From these, we can see that there are common commonalities among the easily confused points. It is easy to confuse airports, shopping malls, and pedestrian streets; it is easy to confuse public squares, city parks, pedestrian streets, and street traffic; and it is easy to confuse various modes of transportation. After using the model fusion strategy, Fig.10 It can be seen that the above-mentioned confusing points have been improved, but they still exist. Fig.11From the analysis, it can be seen that the multi-level acoustic scene classification model based on model fusion has a particularly significant improvement on the confusion between airports and pedestrian streets, and between subways and subway stations. Combined with the results in Tables 3 and 4, the three-classification model using the model fusion strategy has a high classification accuracy for the three major categories of scenes, among which the highest accuracy of the traffic category is 98.3%. Airports and subway stations belong to indoor scenes, pedestrian streets belong to outdoor scenes, and subways belong to traffic scenes. By correcting the three-classification results for the ten-classification results, the misjudgment between different major categories of scenes has been improved, thereby improving the overall classification performance of the model. Table 5 shows the accuracy comparison between the model in this paper and the other three models that performed well in DCASE. All four models used the same data set and training division method. The audio features and data enhancement strategies used by each model are listed in the table. Among them, Suh et al. used a single model composed of three parallel ResNets, did not use the model fusion strategy, and did not consider the multi-level characteristics of the acoustic scene. The final accuracy was 74.2%, which was 7.6% lower than the method in this paper. Kim et al. used the BC-ResNet-Mod they built and combined it with the proposed residual normalization method to achieve an accuracy of 75.9%, which is 5.9% lower than the method in this paper. Yen et al. used the Inception model, combined with multiple data enhancement strategies, and also took the multi-level structure of the model into account. The final accuracy was 79.4%, which was 2.4% lower than the model in this paper. From the comparison results, it can be seen that the model in this paper takes into account the data enhancement strategy, model fusion strategy and the multi-level structure of the acoustic scene, which effectively improves the classification accuracy.

[0047] Table 1 Classification accuracy of the ten-class model for each scene Table 2 Classification accuracy of the ten-class model for each device Table 3 Classification accuracy of the three-classification model for each scene Table 4 Classification accuracy of the three-classification model for each device Table 5 Comparison of the accuracy of the classification model in this paper with other models It should be noted that device A in Table 2 and Table 4 is a Soundman OKM II Klassik / studio A3 condenser binaural microphone and a Zoom F8 recorder, with a sampling rate of 48kHz and a resolution of 24 bits. Device B is a Samsung Galaxy S7, and device C is an iPhone SE. The datasets collected by devices S1-S6 are simulated datasets. The datasets collected by devices A, B, C, and S1-S6 were collected by Tampere University of Technology from May 2018 to November 2018. The data collection was supported by the European Research Council, 637422 EVERYSOUND funding agreement. The audio data was recorded in Amsterdam, Barcelona, ​​Helsinki, Lisbon, London, Lyon, Madrid, Milan, Prague, Paris, Stockholm, and Vienna. The datasets collected by devices A, B, C, and S1-S6 are publicly available on the following websites: https: / / dcase.community / challenge2020 / task-acoustic-scene-classification#baseline-system.

[0048] Specifically, the source of the model established by Suh et al. in Table 5 is: Suh S, Park S, Jeong Y, et al. Designing acoustic sceneclassification models with CNN variants[J]. Tech. Rep., DCASE2020 Challenge, 2020.

[0049] Specifically, the source of the model established by Kim et al. in Table 5 is: Kim B, Yang S, Kim J, et al. QTI submission to DCASE 2021: Residualnormalization for device-imbalanced acoustic scene classification with efficient design[J]. arXiv preprint arXiv: 2206.13909, 2022.

[0050] Specifically, the source of the model established by Yen et al. in Table 5 is: Yen H, Yang CHH, Hu H, et al. A lottery ticket hypothesis framework for low-complexity device-robust neural acoustic scene classification[J]. arXiv preprint arXiv: 2107.01461, 2021.

[0051] In summary, the present invention proposes a multi-level acoustic scene classification model based on model fusion. A multi-level structure between acoustic scenes is proposed, and the ten types of acoustic scenes are further divided into three categories, and a three-classifier is constructed to learn the upper-level shared features between similar scenes. Data enhancement and model fusion strategies are introduced in both the three classifiers and the ten classifiers, and a multi-level acoustic scene classification model based on model fusion is built. The system fuses the outputs of the ten classifiers and the three classifiers in the later stage to improve the overall performance of the model. The final classification accuracy is 81.8%. Compared with using the ten classifiers alone, the accuracy is increased by 2.6%, and the logarithmic loss is reduced by 0.04.

[0052] It can be understood that the present invention has built three classification models, and the three models built by the present invention are integrated for use, and the ten types of acoustic scenes are further divided into three categories, and a three-classifier is constructed for learning the upper-level shared features between similar scenes. Data enhancement and model fusion strategies are introduced in both the three classifiers and the ten-classifier, and a multi-level acoustic scene classification model based on model fusion is built. The present invention fuses the outputs of the ten-classifier and the three classifiers in the later stage to improve the overall performance of the model, which greatly improves the accuracy and reliability of acoustic scene classification, and realizes the improvement of the generalization performance and classification accuracy of the neural network model for acoustic scene classification problems in cross-device situations.

[0053] See also Fig.13 The present invention provides another embodiment, which provides an acoustic scene classification system based on model fusion, and the acoustic scene classification system based on model fusion includes: An acquisition module 100 is used to acquire a Mel-feature spectrogram of a noise source speech signal; The control module 200 is used to divide the acquired Mel feature spectrogram into a low frequency band and a high frequency band according to a preset frequency division threshold, and then build a Resnet model based on high and low frequency separation; build an FCNN model based on channel attention based on the acquired Mel feature spectrogram; in the process of building the FCNN model based on channel attention, limit the downsampling of the frequency axis, and then build an FCNN model with limited frequency downsampling; fuse the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention, and then train to obtain a three-classifier of the acoustic scene; fuse the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling, and then train to obtain a ten-classifier of the acoustic scene; fuse the output layers of the three-classifier and the ten-classifier to obtain a target acoustic scene classification model, and use the target acoustic scene classification model to classify the acoustic scene.

[0054] It should be noted here that the present invention has built three classification models, and the three models built by the present invention are integrated for use, the ten types of acoustic scenes are further divided into three categories, and a three-classifier is constructed for learning the upper-level shared features between similar scenes. Data enhancement and model fusion strategies are introduced in both the three classifiers and the ten-classifier, and a multi-level acoustic scene classification model based on model fusion is built. The present invention fuses the outputs of the ten-classifier and the three classifiers in the later stage to improve the overall performance of the model, which greatly improves the accuracy and reliability of acoustic scene classification, and realizes the improvement of the generalization performance and classification accuracy of the neural network model for acoustic scene classification problems in cross-device situations.

[0055] In a preferred embodiment, the present application further provides an electronic device, the electronic device comprising: A memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the acoustic scene classification method based on model fusion is implemented. The computer device can be broadly a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory can provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present invention are performed.

[0056] The present invention may be implemented as a computer-readable storage medium having a computer program stored thereon, which causes the steps of the method of an embodiment of the present invention to be executed when executed by a processor. In one embodiment, the computer program is distributed on a plurality of computer devices or processors coupled to a network so that the computer program is stored, accessed and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be performed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be performed by one or more computer devices or processors, and one or more other method steps / operations may be performed by one or more other computer devices or processors. One or more computer devices or processors may perform a single method step / operation, or perform two or more method steps / operations.

[0057] It will be appreciated by a person skilled in the art that the method steps of the present invention may be performed by instructing related hardware such as a computer device or a processor through a computer program, and the computer program may be stored in a non-temporary computer-readable storage medium, which causes the steps of the present invention to be performed when the computer program is executed. Depending on the circumstances, any reference to memory, storage, database, or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0058] It can be understood that the present invention has built three classification models, and the three models built by the present invention are integrated for use, and the ten types of acoustic scenes are further divided into three categories, and a three-classifier is constructed for learning the upper-level shared features between similar scenes. Data enhancement and model fusion strategies are introduced in both the three classifiers and the ten-classifier, and a multi-level acoustic scene classification model based on model fusion is built. The present invention fuses the outputs of the ten-classifier and the three classifiers in the later stage to improve the overall performance of the model, which greatly improves the accuracy and reliability of acoustic scene classification, and realizes the improvement of the generalization performance and classification accuracy of the neural network model for acoustic scene classification problems in cross-device situations.

[0059] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0060] The specific implementation of the present invention described above does not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An acoustic scene classification method based on model fusion, It is characterized in that The method comprises: S100, obtaining a Mel feature spectrogram of a noise source speech signal; dividing the obtained Mel feature spectrogram into a low frequency band and a high frequency band according to a preset frequency division threshold, and then building a Resnet model based on high and low frequency separation; building a FCNN model based on channel attention based on the obtained Mel feature spectrogram; in the process of building the FCNN model based on channel attention, limiting the downsampling of the frequency axis, and then building a FCNN model with limited frequency downsampling; S200, the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention are fused, and then the three classifiers of the acoustic scene are trained; the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling are fused, and then the ten classifiers of the acoustic scene are trained; S300, fusing the output layers of the three classifiers and the ten classifiers to obtain a target acoustic scene classification model, and using the target acoustic scene classification model to perform acoustic scene classification.

2. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The classification results of the three classifiers of the acoustic scene include: Indoor scenes, outdoor scenes and traffic scenes.

3. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The classification results of the ten classifiers for the acoustic scene include: Airports, shopping malls, subway stations, pedestrian streets, public squares, street traffic, trams, buses, subways, city parks.

4. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The output layers of the three classifiers and the ten classifiers are fused to obtain a target acoustic scene classification model, and the target acoustic scene classification model is used to perform acoustic scene classification, including: set up and Respectively represent the three major categories and ten subcategories of acoustic scene sets, and are respectively used and Represents the classification result output of the three-classifier and the ten-classifier, then for the input , the final prediction classification result can be expressed as follows: in, express yes is a superset of ; therefore, the entire classification system predicts that the input audio segment belongs to The probability of is equal to the probability given by the three classifiers The probability of The product of the probabilities of .

5. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The obtained Mel feature spectrogram is divided into a low frequency band and a high frequency band according to a preset frequency division threshold, and then a Resnet model based on high and low frequency separation is built, including: The Mel feature spectrograms of the low-frequency band and the high-frequency band are respectively input into the first residual network and the second residual network, and the outputs of each channel of the first residual network and the second residual network are cascaded on the frequency axis. After the cascading, the contribution value of each channel of the first residual network and the second residual network is weighted using the global average pooling layer and the Softmax activation function, thereby obtaining a Resnet model based on high and low frequency separation.

6. The acoustic scene classification method based on model fusion according to claim 5, It is characterized in that The first residual network and the second residual network both include a first stack layer, a second stack layer, a third stack layer and a fourth stack layer connected in sequence, the first stack layer includes a first residual block, the second stack layer includes a second residual block and a first residual block connected in sequence, the second residual block includes a downsampling path, the downsampling path includes an average pooling layer and a zero padding layer, which are used to perform a downsampling operation. After the downsampling operation, the output result of the downsampling path and the output result of the residual path are spliced ​​to match the output dimension during subsequent convolution operations.

7. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The FCNN model based on channel attention is built based on the obtained Mel feature spectrum, including: The Mel feature spectrum is input into the first convolution block, which contains two convolutional layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 2×2, followed by zero padding. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each layer is ReLU, and batch normalization is performed after each layer. The last step of the first convolution block is to perform a maximum pooling operation, and the output result of the first convolution block is input into the second convolution block. The second convolution block contains two convolutional layers. The difference between the second convolution block and the first convolution block is that the second convolution block Both convolutional layers use a convolution kernel of size 3×3 and a stride of 1×1. The output of the second convolutional block is input into the third convolutional block. The third convolutional block has four convolutional layers. Each convolutional layer uses a convolution kernel of size 3×3 and a stride of 1×1. Each convolutional layer uses zero padding, batch normalization, and ReLU function operations. A dropout layer is used after the first three convolutional layers to prevent overfitting. The last step of the third convolutional block performs a 2×2 maximum pooling operation, and finally undergoes a single-layer convolution operation. Its output feature map will be used as the input of the channel attention module.

8. The acoustic scene classification method based on model fusion according to claim 1, It is characterized in that The FCNN model for limiting frequency downsampling includes: The Mel feature spectrum is input into the first convolution block, which includes two convolution layers. The size of the convolution kernel of the first convolution layer is 5×5 and the step size is 1×2. The size of the convolution kernel of the second convolution layer is 3×3 and the step size is 1×1. The activation function used in each convolution layer is ReLU, and batch normalization is performed after each convolution layer. The output result of the first convolution block is input into the second convolution block, which includes two convolution layers. The size of the convolution kernel of the two convolution layers in the second convolution block is 3×3 and the step size is 1×1. The activation function used after each convolution layer is also ReLU, and batch normalization is also performed after each layer. The output result of the second convolution block is input into the third convolution block. The third convolution block has four convolution layers, and each convolution layer uses a convolution kernel of size 3×3 and a 1× 1, and batch normalization and ReLU function are used. Dropout of 0.3 is used after the first and third convolutional layers to prevent overfitting. The output result of the third convolutional block is input to the fourth convolutional block. The fourth convolutional block includes two convolutional layers. Both convolutional layers of the fourth convolutional block use convolution kernels with a size of 3×3 and a step size of 1×1. Batch normalization, ReLU function activation and a dropout layer with a value of 0.5 are also performed after each convolutional layer of the fourth convolutional block. In the ten convolutional layers of the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block arranged in the order of calculation, the second and fourth convolutional layers are followed by maximum pooling of 3×3 and a step size of 2×2, while the sixth and eighth convolutional layers are followed by maximum pooling of 3×3 and a step size of 1×2.

9. An acoustic scene classification system based on model fusion, It is characterized in that include: An acquisition module, used for acquiring a Mel feature spectrum of a noise source speech signal; A control module is used to divide the acquired Mel feature spectrogram into a low frequency band and a high frequency band according to a preset frequency division threshold, and then build a Resnet model based on high and low frequency separation; build an FCNN model based on channel attention based on the acquired Mel feature spectrogram; in the process of building the FCNN model based on channel attention, limit the downsampling of the frequency axis, and then build an FCNN model with limited frequency downsampling; fuse the output layers of the Resnet model based on high and low frequency separation and the FCNN model based on channel attention, and then train to obtain a three-classifier of the acoustic scene; fuse the output layers of the Resnet model based on high and low frequency separation, the FCNN model based on channel attention, and the FCNN model with limited frequency downsampling, and then train to obtain a ten-classifier of the acoustic scene; It is used to fuse the output layers of the three classifiers and the ten classifiers to obtain the target acoustic scene classification model, and use the target acoustic scene classification model to perform acoustic scene classification.

10. An electronic device, It is characterized in that include: Memory; and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the acoustic scene classification method based on model fusion according to any one of claims 1 to 8 is implemented.

Images