Time-frequency dual-domain audio classification method and system based on convolutional neural network

By constructing the time-frequency dual-domain feature matrix of the Mel spectrogram and combining channel weight processing, the unreliability problem of randomized networks in audio classification is solved, the accuracy of audio classification and model reliability are improved, and the more Effective feature extraction and classification effects.

CN114913872BActive Publication Date: 2024-11-22YANGZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210560557.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-23
Publication Date
2024-11-22
Estimated Expiration
2042-05-23

AI Technical Summary

Technical Problem

In the existing audio classification methods, the randomized network lacks a supervision mechanism, which leads to unreliable initialization weights and excessive dependence on the model structure, and fails to effectively utilize the time-frequency dual-domain characteristics of audio data, affecting the classification accuracy.

Method used

By constructing the time-frequency dual-domain feature matrix of the Mel spectrogram, supervised tone and time features are extracted using convolution kernels of different sizes, combined with channel weights for weighting, and overcoming randomness through cross-entropy loss function and backpropagation optimization model to improve classification accuracy.

Benefits of technology

It improves the accuracy of audio classification, enhances the reliability of the model and the extraction ability of time-frequency dual-domain features, and significantly improves the classification accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114913872B_ABST
    Figure CN114913872B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for time-frequency dual-domain audio classification based on a convolutional neural network. First, the audio is processed to obtain a mel spectrum graph of the audio; then, based on the time axis and frequency axis calculation of the mel spectrum graph, a timbre feature matrix and a time feature matrix are constructed respectively, and connected to form a time-frequency dual-domain feature matrix; then, it is input into a time-frequency dual-domain attention module for channel weighting and maximum pooling, and finally a time-frequency dual-domain feature vector of the audio is obtained and input into a classifier to obtain a predicted label of the audio, and then the loss between the predicted label and the true label is calculated, and back propagation is performed to minimize the loss. By adding supervisory information, the present invention solves the defects of the original randomized network that the initialization weight method is unreliable and overly dependent on the model structure itself due to the lack of a supervisory mechanism; the time-frequency dual-domain attention module also solves the problem that average pooling will cause partial channel information loss, and further improves the accuracy of audio classification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of audio recognition and classification, and relates to a time-frequency dual-domain audio classification method and system based on a convolutional neural network. Background Art

[0002] In the field of audio pattern recognition, the waveform is the primary feature of audio data. It has the most primitive features of audio and includes the time axis and amplitude value. The Mel-spectrogram is the intermediate feature of audio data. It is obtained by weighting, framing, windowing, fast Fourier transform and logarithmic Mel filter on the waveform. It is a two-dimensional representation of audio features with a time axis and a frequency axis. Compared with the waveform, the Mel-spectrogram has a moderate dimension and is easier to process. In addition, the Mel-spectrogram is closer to the perception range of the human ear and pays more attention to specific frequency components.

[0003] In recent years, people have begun to use deep learning models to automatically extract audio features from waveforms or mel-spectrograms. The convolutional neural network architecture in deep learning is used as an important feature extractor. The benefits of using convolutional neural networks as audio feature extractors are: 1) Mel-spectrogram features are very similar to image features, and advanced technologies in the image field can be introduced into the field of audio classification; 2) Convolutional neural networks focus on local features, and audio is generally segmented rhythm, which is in line with the focus of convolutional neural networks; 3) As a deep learning model, convolutional neural networks can automatically extract from raw features. However, the difference between audio data and image data is that the spectrogram of audio data has time domain and frequency domain, not just a simple grid-like neighborhood structure. How to extract features for these two domains is the key to building a model. In addition, some researchers this year [Pons, J. and Serra, X. Randomly Weighted CNNs for (Music) Audio Classification. IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, 12-17 May 2019, 336-340] used convolutional networks with randomized weights for feature extraction. However, due to the lack of supervision mechanism, the randomized network has the defects of over-reliance on the initialization weight method and the model structure itself. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a time-frequency dual-domain audio classification method and system based on convolutional neural network, which fully utilizes the time-frequency dual-domain characteristics of Mel-spectrogram and overcomes the defects of randomized network that the initialization weight method is unreliable and overly dependent on the model structure itself due to the lack of supervision mechanism, and reduces the partial channel information loss caused by average pooling by dividing channel weights to improve the accuracy of classification.

[0005] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0006] A time-frequency dual-domain audio classification method based on convolutional neural network includes the following steps:

[0007] (1) Convert audio data into Mel-spectrogram;

[0008] (2) Based on the Mel-spectrogram of the audio, the supervised timbre feature matrix and the supervised time feature matrix are calculated respectively to construct the final time-frequency dual-domain feature matrix; including:

[0009] Based on the frequency axis of the Mel-spectrogram, convolution kernels of different sizes are used to extract its features, and the output of each convolution kernel is stacked to form a supervised timbre feature matrix;

[0010] Based on the time axis of the Mel-spectrogram, we use convolution kernels of different sizes to extract its features, and stack the output of each convolution kernel to form a supervised time feature matrix;

[0011] Connect the supervised timbre features and supervised time features to obtain a dual-domain feature matrix of time and frequency;

[0012] (3) The time-frequency dual-domain feature matrix based on the Mel-spectrogram is average-pooled to obtain a feature vector, the feature vector is stimulated by a fully connected layer, and different channel weight vectors are obtained by focusing on the channel information. The weight vector is multiplied by the time-frequency dual-domain feature matrix by channel, and then a maximum pooling operation is performed to obtain the time-frequency dual-domain feature vector;

[0013] (4) Finally, the time-frequency dual-domain feature vector is input into the softmax classifier to output the classification result of the audio data; the cross entropy is used as the loss function, and the convolution kernel parameters in step (2) and the fully connected layer network parameters in step (3) are updated by back propagation to minimize the loss to obtain the trained audio classification model.

[0014] Preferably, in step (2), the method for extracting supervised timbre features is: extracting features of the Mel-spectrogram through different convolutional layers respectively, the convolution kernels of different convolutional layers have different sizes, and each convolutional layer is connected to a maximum pooling layer; the features extracted by each convolutional layer are compressed by the maximum pooling layer, and then stacked vertically to form a supervised timbre feature matrix; the method for extracting supervised time features is: first compressing the Mel-spectrogram into a one-dimensional time vector through a maximum pooling layer, and then extracting features through different convolutional layers respectively, and then stacking them vertically to form a supervised time feature matrix.

[0015] As a preference, represents the Mel spectrum of the i-th audio data, where F is the number of frames of the Mel spectrum of the audio data, and M is the number of Mel filters; in extracting supervised timbre features, the operation of a single convolution kernel is expressed as:

[0016] Y i,a =Conv(X i ,W a )

[0017]

[0018]

[0019]

[0020] Where W a represents the ath timbre convolution kernel, A represents the total number of convolution kernels used to extract supervised timbre features, They are randomly initialized and automatically optimized during the training process according to the back propagation of the loss function. Conv(X i ,W a ) indicates that X i Use the convolution kernel W a Perform convolution operation, RELU is the activation function, which represents the output Y of the convolution layer i,a Perform a nonlinear mapping, MaxPooling represents the maximum pooling operation, is the supervised timbre feature matrix;

[0021] In extracting supervised temporal features, the operation of a single convolution kernel is expressed as:

[0022]

[0023]

[0024]

[0025]

[0026] MaxPooling(X i ) indicates X i Perform the maximum pooling operation on each frame; Express Using a one-dimensional temporal convolution kernel Perform convolution operation; represents the e-th time convolution kernel, E represents the total number of convolution kernels for extracting supervised time features, They are randomly initialized and automatically optimized during training according to the back propagation of the loss function. is the supervision time feature matrix.

[0027] Preferably, the time-frequency dual-domain feature matrix Where S=A+E.

[0028] Preferably, the step (3) specifically comprises:

[0029] First, the time-frequency dual-domain feature matrix FF i Input the average pooling layer to average pool the row vector to obtain a compressed vector Consider S as the number of channels;

[0030] ff i =AvePooling(FF i )

[0031] Then, ff i As input to two consecutive fully connected layers, the weight of each channel is calculated:

[0032] q i =W1ff i +b1

[0033] ff i ′=W2q i +b2

[0034] q i ,ff i ′ represents the output of the first and second fully connected layers respectively; W1, W2 represent the weight matrices of the first and second fully connected layers respectively; b1, b2 represent the bias of the first and second fully connected layers respectively, they are all randomly initialized and will be automatically optimized according to the back propagation of the loss function during the training process;

[0035] Finally, the weight vector ff i ′ and input Multiply and get the output feature vector

[0036] U i =diag(ff i′)×FF i

[0037] u i =MaxPooling(U i )

[0038] diag(ffi i ′) represents the vector ffi i ′ becomes a diagonal matrix; MaxPooling(U i ) indicates that U i Perform the maximum pooling operation.

[0039] A time-frequency dual-domain audio classification system based on convolutional neural network, comprising:

[0040] A preprocessing module, used to convert audio data into Mel-spectrogram;

[0041] The supervised timbre module is used to extract features based on the frequency axis of the Mel-spectrogram using convolution kernels of different sizes, and stack the output of each convolution kernel to form a supervised timbre feature matrix;

[0042] The supervised time module is used to extract features based on the time axis of the Mel-spectrogram using convolution kernels of different sizes, and stack the output of each convolution kernel to form a supervised time feature matrix;

[0043] The time-frequency dual-domain attention module is used to obtain a feature vector by average pooling the time-frequency dual-domain feature matrix obtained by connecting the supervised timbre features and the supervised time features, and to perform an excitation operation on the feature vector through a fully connected layer, focusing on the channel information to obtain different channel weight vectors, and to obtain a time-frequency dual-domain feature vector after performing channel multiplication on the weight vector and the time-frequency dual-domain feature matrix and then performing a maximum pooling operation;

[0044] The classification module is used to input the time-frequency dual-domain feature vector into the softmax classifier and output the classification result of the audio data;

[0045] And a training module, which is used to use cross entropy as the loss function, and to minimize the loss through back propagation to update the convolution kernel parameters in the supervision time module and the supervised time module and the fully connected layer network parameters in the time-frequency dual-domain attention module to obtain a trained audio classification model.

[0046] A computer system comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network are implemented.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network.

[0048] Beneficial effects: The present invention proposes a time-frequency dual-domain audio classification method based on a convolutional neural network for use in the field of audio recognition and classification. This method overcomes two major shortcomings of previous audio classification methods: First, in recent years, when extracting audio features, the audio classification methods based on random weight networks use random network weights, which are completely dependent on the network structure and are not very reliable, and do not fully utilize the label information of the audio data. Our method adds feedback of supervisory information at the output of the network, and adds a step in which the neural network continuously optimizes the model weights by calculating the cross entropy loss and back-propagating, so it is more reliable; second, the introduction of the time-frequency dual-domain attention module can better distinguish the importance of each channel, thereby more effectively extracting audio features that are conducive to classification.

[0049] The advantage of the present invention is that it utilizes the characteristics of the Mel spectrum graph of audio with a time axis and a frequency axis, divides the frequency feature and the time feature into two modules for extraction, and then connects the two features vertically to form a time-frequency dual-domain feature matrix and inputs it into the time-frequency dual-domain attention module for channel weighting to obtain the input neural network, and then trains the neural network model through back propagation and cross entropy loss function. Compared with the traditional audio recognition classification method, our method uses a Mel filter group similar to the principle of the human ear in the process of processing the waveform graph into a Mel spectrum graph. This filter group filters the sound that the human ear cannot perceive, focuses on specific frequency components, and is closer to the perception range of the human ear. In addition, the addition of supervisory information also solves the defects of the traditional randomized network that the initialization weight method is unreliable and overly dependent on the model structure itself due to the lack of a supervisory mechanism. Afterwards, considering the influence of channel information on the classification results, the time-frequency dual-domain attention module is introduced to further improve the accuracy of the model for audio classification. According to experimental data, our method has a relatively good effect on classification accuracy, and the ablation experiment also verifies the importance of channel information to the feature matrix. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is a schematic diagram of the method structure of an embodiment of the present invention. The figure introduces in detail the execution process of the embodiment of the present invention, which is composed of a supervised timbre module, a supervised time module, a supervised timbre feature matrix, a supervised time feature matrix, a time-frequency dual-domain feature matrix, a time-frequency dual-domain attention module, a time-frequency dual-domain feature vector, and back propagation. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] like Figure 1 As shown, the present invention utilizes the characteristics of the Mel spectrum graph of audio with frequency axis and time axis, and constructs a supervised timbre module and a supervised time module to output the supervised timbre feature matrix and supervised time feature matrix of the audio file respectively, thereby improving the defects of the traditional randomized network due to the lack of supervision mechanism, resulting in the unreliable initialization weight method and excessive reliance on the model structure itself. Then the time-frequency dual-domain feature matrix formed by its longitudinal connection is input into the time-frequency dual-domain attention module for channel weighting, and finally the time-frequency dual-domain feature matrix after channel weighting division is output through maximum pooling. The time-frequency dual-domain feature vector is then input into the neural network model to obtain the final classification result.

[0053] Specifically, the embodiment of the present invention discloses a time-frequency dual-domain audio classification method based on a convolutional neural network, which mainly includes the following steps:

[0054] 1. Construct a Mel-spectrogram by preprocessing the audio data.

[0055] The waveform file of each audio is weighted, framed, windowed, fast Fourier transformed and logarithmic Mel filter to obtain a Mel spectrogram with a fixed number of frames. For example, for each audio with a length of 30s, a frame is divided into 20-40ms, and a Mel spectrogram with a time dimension of 1404 frames is constructed.

[0056] Second, based on the Mel-spectrogram, we calculate the supervised timbre feature matrix and the supervised time feature matrix respectively, and construct the final time-frequency dual-domain feature matrix. Including:

[0057] (1) Based on the frequency axis of the Mel-spectrogram, convolution kernels of different sizes are used to extract its features, and the output of each convolution kernel is stacked to form a supervised timbre feature matrix.

[0058] (2) Based on the time axis of the Mel-spectrogram, convolution kernels of different sizes are used to extract its features, and the output of each convolution kernel is stacked to form a supervised time feature matrix.

[0059] (3) Connect the supervised timbre features and the supervised time features to obtain the time-frequency dual-domain feature matrix of the mel-spectrogram.

[0060] Specifically, the Mel spectrogram of the audio is input into the supervised timbre module and the supervised time module respectively, and the supervised timbre feature matrix and the supervised time feature matrix are obtained respectively to construct the final time-frequency dual-domain feature matrix FF iIn the supervised timbre module, 448 convolution kernels of 6 categories are used for feature extraction. Each convolution layer is followed by a maximum pooling layer, and then the feature vectors output by each convolution kernel are stacked vertically to form a supervised timbre feature matrix. Similarly, the supervised time module uses 448 convolution kernels of 4 categories for feature extraction. The difference is that in the supervised time module, the Mel spectrum of the audio will first pass through the maximum pooling layer, then through the convolution operation, and finally the output of each convolution kernel will be stacked vertically to form a supervised time feature matrix. Finally, the supervised time feature matrix and the supervised timbre feature matrix are vertically connected to obtain the time-frequency dual-domain feature matrix. In the model of this embodiment, we set the sizes of the six types of convolution kernels in the supervised timbre module to 7*86, 1*86, 3*86, 7*38, 1*38, and 3*38, and the corresponding number of convolution kernels of each type is set to 32, 64, 128, 32, 64, and 128, respectively, and the step size of the convolution kernel moving on the Mel-spectrogram is 1; the sizes of the four types of convolution kernels in the supervised time module are set to 165*1, 128*1, 64*1, and 32*1, respectively, and the corresponding number of convolution kernels of each type is set to 24, 56, 120, and 248, respectively.

[0061] set up represents the mel-spectrogram used for training, where represents the Mel spectrogram of the i-th training sample, i.e. the i-th audio data, i = 1, 2, ..., n, F is the number of frames of the Mel spectrogram of the audio data, M is the number of Mel filters, and n is the number of training samples; in the module for extracting supervised timbre, the operation of a single convolution kernel is as follows:

[0062] Y i,a =Conv(X i ,W a )#(1)

[0063]

[0064]

[0065]

[0066] Formula (1) represents the i Use the convolution kernel W a Perform convolution operation to get Y i,a , where W a represents the ath convolution kernel in the supervised timbre module, A represents the total number of convolution kernels used to extract supervised timbre features, They are all randomly initialized and automatically optimized according to the back propagation of the loss function during the training process; Formula (2) is the activation function, which represents the output Y of the convolutional layer i,a Perform a nonlinear mapping; Formula (3) represents Perform the maximum pooling operation on each frame and get Formula (4) represents the supervised timbre feature matrix obtained by vertically stacking the outputs of A timbre convolution kernels and the maximum pooling operation:

[0067] In the extraction supervision time module, the operation of a single convolution kernel is as follows:

[0068]

[0069]

[0070]

[0071]

[0072] Formula (5) represents the i Perform the maximum pooling operation on each frame and get Formula (6) represents Using a one-dimensional temporal convolution kernel Perform convolution operation to obtain in represents the e-th time convolution kernel, E represents the total number of convolution kernels for extracting supervised time features, They are all randomly initialized and automatically optimized according to the back propagation of the loss function during the training process; Formula (7) is the activation function, which represents the output Y of the convolutional layer i,a Perform a nonlinear mapping; Formula (8) represents stacking the outputs of E time convolution kernels vertically to obtain the supervised time feature matrix

[0073] By vertically connecting the supervised timbre features and the supervised time features, we can obtain the bidirectional time-frequency feature matrix of the audio. S=A+E.

[0074]

[0075] 3. Considering the influence of channel information on classification results, the time-frequency dual-domain feature matrix is ​​input into the time-frequency dual-domain attention module for channel weighting operation. The time-frequency dual-domain attention module converts the time-frequency dual-domain feature matrix FF of the Mel-spectrogram into i After average pooling, the feature vector ff is obtained i , perform excitation operations on the feature vector, pay attention to the channel information to obtain different channel weights ff i ′, the vector ff i ′ and feature matrix FF i After channel multiplication and maximum pooling operation, the time-frequency dual-domain feature vector u is obtained. i . Specifically expressed as:

[0076] First, the time-frequency dual-domain feature FF i Input the average pooling layer to average pool the row vector to obtain a compressed vector We call S the number of channels.

[0077] ff i =AvePooling(FF i )#(10)

[0078] Formula (10) represents the squeezing operation, which uses average pooling to compress the feature matrix;

[0079] Get the output ff of the squeeze operation i After that, ff i As input to two consecutive fully connected layers, the purpose is to calculate the weight of each channel:

[0080] q i =W1ff i +b1#(11)

[0081] ffi i ′=W2q i +b2#(12)

[0082] Formula (11-12) represents the excitation operation, that is, two consecutive fully connected layers are used to calculate the weight of each channel q i ,ffi i ′ represents the output of the first and second fully connected layers of the i-th sample; W1, W2 represent the weight matrices of the first and second fully connected layers of the t-th sample; b1, b2 represent the bias of the first and second fully connected layers of the i-th sample.

[0083] Finally, the channel weight vector ff i ′ and the input of the time-frequency dual-domain attention module Perform channel multiplication to get the output of the attention module:

[0084] U i =diag(ffi i ′)×FF i #(13)

[0085] u i =MaxPooling(U i )#(14)

[0086] (13) represents the channel weighting operation, that is, according to ffi i The weight of ′ on FF i Each row of is weighted, diag(ffi i′) represents the vector ffi i ′ becomes a diagonal matrix; Equation (14) represents the i Perform the maximum pooling operation and finally get the feature vector

[0087] 4. Input the time-frequency dual-domain feature vector into the softmax classifier and output the classification result.

[0088] Use the softmax function as a classifier and the binary cross entropy as the loss function to train and get the predicted classification results and the true label y i Calculate the loss, minimize the loss through back propagation and train the model. Define the optimization objective loss function of the model as:

[0089]

[0090] Where n represents the number of samples, Represents the output after the i-th sample is input into the neural network, y i represents the true label of the i-th sample, θ represents the parameter set in the neural network, including W1, W2, b1, b2, which uses the gradient descent method with η as the learning rate for back propagation update formula:

[0091]

[0092] In order to ensure the consistency of each input data dimension, we fill in the tail of the mel-spectrogram that is not long enough.

[0093] Tables 1 and 2 show the classification performance of the present invention (time-frequency dual-domain audio classification method based on convolutional neural network, abbreviated as FFANN) under three audio data sets. GTZAN is a music data set, and the music samples of this data set include the following genres: classical, country, disco, hip-hop, jazz, blues, reggae, pop, metal, etc. Among them, the "classical" category includes: choir, piano, string quartet and other categories; the "jazz" category includes: bigband, cool, fusion, piano, quaret, swing and other categories. It has 10 categories, each category has a different 100 samples, and the data set is divided into a training set / validation set / test set with a ratio of 600 / 200 / 200. The ExtendedBallroom data set is audio data in the style of social dance. The ExtendBallroom data set contains 13 categories and a total of 4180 samples. The data set is divided into a training set / validation set / test set with a ratio of 2508 / 836 / 836. UrbanSound8K is an urban sound dataset that contains 10 urban sounds from different categories, including: air conditioners, car horns, children playing, dogs barking, drilling, engine idling, gun shooting, handheld jackhammers, police sirens, street music, a total of 8732 samples.

[0094] Table 1 shows the results of comparative tests comparing the performance of four different methods on three training datasets. Table 2 shows the performance of FFANN with different learning rate parameters on three datasets. From Table 1, we can see that our method has the best accuracy on the three datasets. From Table 2, we can see that in the GTZAN and ExtendBallroom datasets, the performance is best when the learning rate is 0.01, and in the Urban Sound8K dataset, the performance is best when the learning rate is 0.001.

[0095] Table 1: Performance (accuracy) of FFANN and other methods on three datasets

[0096]

[0097] Table 2: Performance of FFANN at different learning rates on three datasets

[0098]

[0099] Based on the same inventive concept, an embodiment of the present invention discloses a time-frequency dual-domain audio classification system based on a convolutional neural network, including: a preprocessing module for converting audio data into a Mel-spectrogram; a supervised timbre module for extracting features based on the frequency axis of the Mel-spectrogram using convolution kernels of different sizes, and stacking the output of each convolution kernel to form a supervised timbre feature matrix; a supervised time module for extracting features based on the time axis of the Mel-spectrogram using convolution kernels of different sizes, and stacking the output of each convolution kernel to form a supervised time feature matrix; a time-frequency dual-domain attention module for concatenating the supervised timbre feature and the supervised time feature to obtain a supervised timbre feature matrix. The time-frequency dual-domain feature matrix is ​​average-pooled to obtain a feature vector, the feature vector is excited through a fully connected layer, and the channel information is paid attention to to obtain different channel weight vectors, and the weight vector is multiplied by the time-frequency dual-domain feature matrix through the channel and then a maximum pooling operation is performed to obtain the time-frequency dual-domain feature vector; a classification module is used to input the time-frequency dual-domain feature vector into a softmax classifier and output the classification result of the audio data; and a training module is used to use cross entropy as the loss function, and to minimize the loss through back propagation to update the supervision time module and the convolution kernel parameters in the supervision time module and the fully connected layer network parameters in the time-frequency dual-domain attention module to obtain a trained audio classification model.

[0100] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of each module described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. The division of the modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules can be combined or integrated into another system.

[0101] Based on the same inventive concept, an embodiment of the present invention discloses a computer system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network are implemented.

[0102] Based on the same inventive concept, an embodiment of the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network are implemented.

[0103] Those skilled in the art can understand that the technical solution of the present invention, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer system (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: various media that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

Claims

1. Time-frequency dual-domain audio classification method based on convolutional neural network, It is characterized in that The steps include: (1) Convert audio data into Mel-spectrogram; (2) Based on the Mel-spectrogram of the audio, the supervised timbre feature matrix and the supervised time feature matrix are calculated respectively to construct the final time-frequency dual-domain feature matrix; including: Based on the frequency axis of the Mel-spectrogram, convolution kernels of different sizes are used to extract its features, and the output of each convolution kernel is stacked to form a supervised timbre feature matrix; Based on the time axis of the Mel-spectrogram, we use convolution kernels of different sizes to extract its features, and stack the output of each convolution kernel to form a supervised time feature matrix; Connect the supervised timbre features and supervised time features to obtain a dual-domain feature matrix of time and frequency; (3) The time-frequency dual-domain feature matrix based on the Mel-spectrogram is average-pooled to obtain a feature vector, the feature vector is stimulated by a fully connected layer, and different channel weight vectors are obtained by focusing on the channel information. The weight vector is multiplied by the time-frequency dual-domain feature matrix by channel, and then a maximum pooling operation is performed to obtain the time-frequency dual-domain feature vector; (4) Finally, the time-frequency dual-domain feature vector is input into the softmax classifier, and the classification result of the audio data is output; the cross entropy is used as the loss function, and the convolution kernel parameters in step (2) and the fully connected layer network parameters in step (3) are updated by back propagation to minimize the loss to obtain a trained audio classification model; In step (2), the method for extracting supervised timbre features is: extracting features of the Mel spectrum graph through different convolutional layers, the convolution kernels of different convolutional layers have different sizes, and each convolutional layer is connected to a maximum pooling layer; the features extracted by each convolutional layer are compressed by the maximum pooling layer, and then stacked vertically to form a supervised timbre feature matrix; the method for extracting supervised time features is: first compressing the Mel spectrum graph into a one-dimensional time vector through the maximum pooling layer, and then extracting features through different convolutional layers, and then stacking them vertically to form a supervised time feature matrix; The time-frequency dual-domain feature matrix Where S=A+E, is the supervised timbre feature matrix, is the supervisory time feature matrix, F is the number of frames of the Mel-spectrogram after audio data preprocessing, A and E are the number of convolution kernels used to extract supervisory timbre features and supervisory time features, respectively; The step (3) specifically comprises: First, the time-frequency dual-domain feature matrix FF i Input the average pooling layer to average pool the row vector to obtain a compressed vector Consider S as the number of channels; ff i =AvePooling(FF i ) AvePooling(FF i ) indicates the FF i Perform average pooling operation; Then, ff i As input to two consecutive fully connected layers, the weight of each channel is calculated: q i =W 1 ff i +b 1 ff i ′=W 2 q i +b 2 q i ,ff i ′ represents the output of the first and second fully connected layers respectively; W 1 ,W 2 Represent the weight matrices of the first and second fully connected layers respectively; b 1 ,b 2 They represent the biases of the first and second fully connected layers, respectively. They are all randomly initialized and will be automatically optimized during training according to the back propagation of the loss function. Finally, the weight vector ff i ′ and input Multiply and get the output feature vector U i =diag(ff i ′)×FF i you i =MaxPooling(U i ) diag(ff i ′) represents the vector ff i ′ becomes a diagonal matrix; MaxPooling(U i ) indicates that U i Perform the maximum pooling operation.

2. The time-frequency dual-domain audio classification method based on convolutional neural network according to claim 1, It is characterized in that set up represents the Mel spectrum of the i-th audio data, where F is the number of frames of the Mel spectrum of the audio data, and M is the number of Mel filters; in extracting supervised timbre features, the operation of a single convolution kernel is expressed as: Y i,a =Conv(X i ,W a ) Where W a represents the ath timbre convolution kernel, A represents the total number of convolution kernels used to extract supervised timbre features, They are randomly initialized and automatically optimized during the training process according to the back propagation of the loss function. Conv(X i ,W a ) indicates that X i Use the convolution kernel W a Perform convolution operation, RELU is the activation function, which represents the output Y of the convolution layer i,a Perform a nonlinear mapping, MaxPooling represents the maximum pooling operation, is the supervised timbre feature matrix; In extracting supervised temporal features, the operation of a single convolution kernel is expressed as: MaxPooling(X i ) indicates X i Perform the maximum pooling operation on each frame; Express Using a one-dimensional temporal convolution kernel Perform convolution operation; represents the e-th time convolution kernel, E represents the total number of convolution kernels for extracting supervised time features, They are randomly initialized and automatically optimized during training according to the back propagation of the loss function. is the supervision time feature matrix.

3. Time-frequency dual-domain audio classification system based on convolutional neural network, It is characterized in that include: A preprocessing module, used to convert audio data into Mel-spectrogram; The supervised timbre module is used to extract features based on the frequency axis of the Mel-spectrogram using convolution kernels of different sizes, and stack the output of each convolution kernel to form a supervised timbre feature matrix; The supervised time module is used to extract features based on the time axis of the Mel-spectrogram using convolution kernels of different sizes, and stack the output of each convolution kernel to form a supervised time feature matrix; The time-frequency dual-domain attention module is used to obtain a feature vector by average pooling the time-frequency dual-domain feature matrix obtained by connecting the supervised timbre features and the supervised time features, and to perform an excitation operation on the feature vector through a fully connected layer, focusing on the channel information to obtain different channel weight vectors, and to obtain a time-frequency dual-domain feature vector after performing channel multiplication on the weight vector and the time-frequency dual-domain feature matrix and then performing a maximum pooling operation; The classification module is used to input the time-frequency dual-domain feature vector into the softmax classifier and output the classification result of the audio data; and a training module, which uses cross entropy as a loss function, and updates the convolution kernel parameters in the supervised time module and the fully connected layer network parameters in the time-frequency dual-domain attention module by back-propagation to obtain a trained audio classification model; In the supervised timbre module, the features of the Mel-spectrogram are extracted through different convolutional layers, the convolution kernels of different convolutional layers are of different sizes, and each convolutional layer is connected to a maximum pooling layer; the features extracted by each convolutional layer are compressed by the maximum pooling layer, and then stacked vertically to form a supervised timbre feature matrix; In the supervised time module, the Mel-spectrogram is first compressed into a one-dimensional time vector through a maximum pooling layer, and then features are extracted through different convolutional layers, and then stacked vertically to form a supervised time feature matrix; The time-frequency dual-domain feature matrix Where S=A+E, is the supervised timbre feature matrix, is the supervised time feature matrix, F is the number of frames of the Mel-spectrogram after audio data preprocessing, A and E are the total number of convolution kernels used to extract supervised timbre features and supervised time features respectively; in the time-frequency dual-domain attention module: First, the time-frequency dual-domain feature matrix FF i Input the average pooling layer to average pool the row vector to obtain a compressed vector Let S be the number of channels; ff i =AvePooling(FF i ) AvePooling(FF i ) indicates the FF i Perform average pooling operation; Then, ff i As input to two consecutive fully connected layers, the weight of each channel is calculated: q i =W 1 ff i +b 1 ff i ′=W 2 q i +b 2 q i ,ff i ′ represents the output of the first and second fully connected layers respectively; W 1 ,W 2 Represent the weight matrices of the first and second fully connected layers respectively; b 1 ,b 2 They represent the biases of the first and second fully connected layers, respectively. They are all randomly initialized and will be automatically optimized during training according to the back propagation of the loss function. Finally, the weight vector ff i ′ and input Multiply and get the output feature vector U i =diag(ff i ′)×FF i you i =MaxPooling(U i ) diag(ff i ′) represents the vector ff i ′ becomes a diagonal matrix; MaxPooling(U i ) indicates that U i Perform the maximum pooling operation.

4. A computer system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the computer program is loaded into a processor, the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network according to claim 1 or 2 are implemented.

5. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the time-frequency dual-domain audio classification method based on a convolutional neural network according to claim 1 or 2 are implemented.