A music classification method based on primary and secondary two-way space-time feature fusion

Abstract

The application discloses a music classification method based on main and auxiliary two-way space-time feature fusion, comprising the following steps: acquiring audio data and labels thereof for training, and carrying out denoising processing, cutting into segments, and extracting a mel spectrum graph; constructing a deep learning network, which is composed of a main path and an auxiliary path, the main path is a space-time attention network, the auxiliary path is a pre-training network, and training results of the two paths are fused; inputting the mel spectrum graph of the training data into the deep learning network in sequence, calculating differences between training results output by the deep learning network and labels thereof, and updating parameters of the network through back propagation; acquiring audio data to be processed, carrying out denoising processing, cutting into segments, and extracting a mel spectrum graph; inputting the mel spectrum graph into the deep learning network, obtaining classification results from the space-time attention network and the pre-training network respectively, and obtaining a final result through fusion. The application enhances the fault tolerance of the deep learning network and also increases the model accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio classification, and particularly relates to a music classification method based on main and auxiliary two-way spatiotemporal feature fusion. BACKGROUND

[0002] With the continuous development of technology, the application of deep learning in audio has developed rapidly. The emergence of models such as convolutional neural network (CNN) makes feature extraction and classification of audio signals more accurate and efficient. At the same time, the recommendation algorithm of various music software is gradually introduced, which adopts content-based filtering (CBF) and collaborative filtering (CF) and other technologies, which can better recommend the same type of music according to the user's listening preferences. In the process of recommending music, the label classification of music files is an important factor, so it is very important to accurately classify music files.

[0003] The most important step in audio classification is feature extraction. According to the difference of feature extraction methods, we can roughly divide it into the following categories:

[0004] (1) Time-domain feature extraction: Time-domain features refer to the characteristics of sound signals on the time axis, including short-time energy, short-time zero-crossing rate, etc.

[0005] (2) Frequency-domain feature extraction: Frequency-domain features refer to the characteristics of sound signals in the frequency domain, including Mel-frequency cepstral coefficients (MFCC), power spectral density, etc.

[0006] (3) Time-frequency domain feature extraction: Time-frequency domain features refer to the characteristics of sound signals in both time and frequency dimensions, including short-time Fourier transform (STFT), wavelet transform, etc.

[0007] (4) Non-linear feature extraction: Non-linear features refer to the features obtained by nonlinear transformation of sound signals, including entropy, fractal dimension, etc.

[0008] (5) Deep learning-based feature extraction: Deep learning features refer to features learned by deep learning models such as deep neural networks, including convolutional neural networks (CNN), recurrent neural networks (RNN), etc.

[0009] But based on the current feature extraction method, all methods are based on the overall extraction of audio features, ignoring the useful frame, in fact, the importance of frame to frame is not the same in audio classification, some frames have no sound characteristics or have sound characteristics that are not too helpful for classification, these are useless frames, and some frames have sound characteristics that are more beneficial to classification, these are useful frames, so we need a network that can distinguish different frames and extract useful features. Moreover, the current deep classification model is based on one angle, only one angle feature is extracted for classification, while most audio contains very rich features, if the features can be extracted from different angles, the classification accuracy of the deep learning network can be greatly improved, so we need a network that can extract features from several different angles. SUMMARY

[0010] In order to solve the technical problems as mentioned above, the application provides a music classification method based on main and auxiliary two-way spatio-temporal feature fusion, which adopts a spatio-temporal attention mechanism to avoid the loss of previous features in long-time audio and greatly improves the accuracy of audio classification.

[0011] In order to achieve the above purpose, the technical scheme adopted by the application is:

[0012] The music classification method based on main and auxiliary two-way spatio-temporal feature fusion provided by the application comprises the following steps:

[0013] S1, obtaining audio data and its label for training, performing denoising processing on the audio data for training, and cutting into segments to extract a mel spectrum graph;

[0014] S2, constructing a deep learning network, the deep learning network comprising a main and auxiliary two-way and a fusion operation, the main way being a spatio-temporal attention network, and the auxiliary way being a pre-training network, and fusing the classification results of the main and auxiliary two ways to obtain a training result;

[0015] S3, inputting the mel spectrum graph of the audio data for training into the deep learning network in sequence, calculating the difference between the training result output by the deep learning network and its label, and updating the parameters of the deep learning network through back propagation;

[0016] S4, obtaining audio data to be processed, performing denoising processing on the audio data to be processed, and cutting into segments to extract a mel spectrum graph;

[0017] S5, inputting the mel spectrum graph of the audio data to be processed into the deep learning network, obtaining classification results from the spatio-temporal attention network and the pre-training network respectively, and obtaining a final result through fusion.

[0018] Preferably, the spatio-temporal attention network comprises a spatial network Densenet, a temporal network LSTM and two fully connected layers, and performs the following operations:

[0019] inputting the mel-spectrogram into the spatial network Densenet to extract a spatial feature vector;

[0020] inputting the mel-spectrogram into the temporal network LSTM to extract a temporal feature vector;

[0021] performing dot product between the spatial feature vector and the temporal feature vector to obtain a spatio-temporal attention weight W;

[0022] performing dot product between the spatio-temporal attention weight W and the temporal feature vector, and then passing through two fully connected layers to obtain a classification result of the spatio-temporal attention network.

[0023] Preferably, the spatial network Densenet comprises a convolutional layer, a pooling layer, a first DenseBlock layer, a first Transition Layer layer, a second DenseBlock layer, a second Transition Layer layer, a third DenseBlock layer, a third Transition Layer layer, a fourth DenseBlock layer and a global pooling layer connected in sequence.

[0024] Preferably, the mel-spectrogram input into the main path and the auxiliary path has a size of 224*224, and the output feature map sizes of the convolutional layer, the pooling layer, the first DenseBlock layer, the first Transition Layer layer, the second DenseBlock layer, the second Transition Layer layer, the third DenseBlock layer, the third Transition Layer layer, the fourth DenseBlock layer and the global pooling layer correspond to 112*112, 56*56, 56*56, 28*28, 28*28, 14*14, 14*14, 7*7, 7*7 and 1*1 in sequence.

[0025] Preferably, the filter size of the convolutional layer is 7*7 with a step of 2, the pooling kernel size of the pooling layer is 3*3 with a step of 2, and the pooling kernel size of the global pooling layer is 7*7.

[0026] Preferably, the pre-training network comprises a Resnet network and two fully connected layers.

[0027] Preferably, the difference between the training result of the deep learning network output and its label is calculated in the form of a cross-entropy loss function.

[0028] Preferably, the fusion is decision-level fusion, and the fusion ratio of the classification result of the spatio-temporal attention network and the classification result of the pre-training network is 6.5:3.5.

[0029] Compared with the prior art, the application has the following beneficial effects:

[0030] The method proposes a deep learning network for main and auxiliary spatio-temporal feature fusion. The main and auxiliary networks capture main features through the main network, which serves as a benchmark for classification. The auxiliary network captures other non-obvious features and fine-tunes the judgment of the main network. Through result fusion, the fault tolerance of the deep learning network is enhanced, and the model accuracy is also increased. Unlike the prior art, the deep learning network for spatio-temporal feature fusion adopts a new idea of spatio-temporal attention mechanism. Spatial and temporal features are extracted respectively, and then the temporal features are embedded into the spatial features to obtain spatio-temporal attention weights. The spatio-temporal attention weights are used to emphasize the frames useful for classification in the audio data, from which more useful acoustic features can be extracted. Embedding spatial features into temporal features can avoid the loss of previous features in long-time audio, greatly improving the accuracy of audio classification. BRIEF DESCRIPTION OF DRAWINGS

[0031] Fig. 1 is a flowchart of the music classification method based on main and auxiliary spatio-temporal feature fusion of the application;

[0032] Fig. 2 is a data transfer diagram of the method of the application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0034] As shown in Figs. 1-2 , a music classification method based on main and auxiliary spatio-temporal feature fusion includes the following steps:

[0035] S1, obtain audio data for training and its label, and perform denoising processing, cut into segments, and extract a mel spectrum graph. In this embodiment, the audio data set for training is two categories of soothing music and exciting music. The soothing music is placed in the same folder, and the exciting music is placed in another folder.

[0036] In the embodiment, the label is equal to 0 or 1, 0 represents that the classified music is of a relaxing type, and 1 represents that the classified music is of an exciting type. The folder names in which the audio is stored are 0 and 1, respectively.

[0037] S2, a deep learning network is constructed, the network is composed of a main path and an auxiliary path, the main path is a space-time attention network, the auxiliary path is a pre-training network, and training results of the two paths are fused. As shown in FIG. 2, the upper layer network represents the space-time attention network, and the space-time network is composed of two layers of networks, and the lower layer network represents the pre-training network. Fig. 2

[0038] The space-time attention network structure includes a space network Densenet, a time network LSTM, and two fully connected layers (Fully Connected Layers), the two fully connected layers are FC1 and FC2 in the main path. Fig. 2

[0039] The space-time attention network performs the following operations:

[0040] The mel spectrum graph is input into the space network Densenet, and a space feature vector is extracted;

[0041] The mel spectrum graph is input into the time network LSTM, and a time feature vector is extracted;

[0042] The space feature vector and the time feature vector are dot multiplied to obtain a space-time attention weight W;

[0043] The space-time attention weight W is dot multiplied with the time feature vector, and then passes through the two fully connected layers to obtain a final classification result.

[0044] The space network Densenet includes convolution layers, pooling layers, a first Dense Block layer, a first Transition Layer layer, a second Dense Block layer, a second Transition Layer layer, a third Dense Block layer, a third Transition Layer layer, a fourth Dense Block layer, and a global pooling layer connected in sequence.

[0045] The pre-training network includes a Resnet network and two fully connected layers, the two fully connected layers are FC1 and FC2 in the auxiliary path. Fig. 2

[0046] The parameters of the two fully connected layers in the main path and the two fully connected layers in the auxiliary path are not necessarily the same.

[0047] ​​​The size of the mel spectrogram of the input main path and auxiliary path is 224*224, and the output feature map sizes of the convolution layer, the pooling layer, the first DenseBlock layer, the first Transition Layer layer, the second Dense Block layer, the second Transition Layer layer, the third Dense Block layer, the third Transition Layer layer, the fourth Dense Block layer and the global pooling layer are 112*112, 56*56, 56*56, 28*28, 28*28, 14*14, 14*14, 7*7, 7*7 and 1*1 respectively.

[0048] The filter size of the convolution layer is 7*7, and the step is 2; the pooling kernel size of the pooling layer is 3*3, and the step is 2; and the global pooling kernel size is 7*7.

[0049] In the embodiment, the fusion is decision-level fusion, and the fusion ratio of the classification result of the spatio-temporal attention network and the pre-trained classification result is 6.5:3.5. The specific fusion ratio is shown in Table 1, and the network accuracy under different ratios is tested, wherein the performance index of 6.5:3.5 is the best.

[0050] Table 1: Fusion ratio test results

[0051]

[0052] S3, the mel spectrogram of the training data is input into the deep learning network in sequence, the difference between the training result output by the deep learning network and the label thereof is calculated, and the parameters of the network are updated through back propagation.

[0053] The difference between the output result and the label thereof is calculated in the form of a cross-entropy loss function. The cross-entropy function first calculates the probability that the result is equal to the label, and then updates the parameters of the deep learning network step by step in reverse. After all the training audios pass through the deep learning network, the parameters of the deep learning network are trained.

[0054] S4, obtaining audio data to be processed, and performing denoising processing, cutting into segments, and extracting a mel spectrogram.

[0055] S5, the mel spectrogram of the test data is input into the deep learning network, and the classification results are obtained from the spatio-temporal attention network and the pre-trained network respectively, and then the final result is obtained through fusion.

[0056] The method disclosed by the application can also be applied to preliminary evaluation of mental health status or emotion classification of audio by changing the type of audio data and the label thereof.

[0057] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0058] The above-described embodiments are merely representative and detailed embodiments described in the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A music classification method based on primary-secondary two-path spatio-temporal feature fusion, characterized in that: The music classification method based on the main and auxiliary two-way spatio-temporal feature fusion comprises the following steps: S1, obtaining audio data for training and labels thereof, performing denoising processing on the audio data for training, and cutting the audio data into segments to extract a mel spectrum graph; S2, constructing a deep learning network, wherein the deep learning network comprises a main and auxiliary two-way and fusion operation, the main way is a spatio-temporal attention network, the auxiliary way is a pre-training network, and the classification results of the main and auxiliary two-way are fused to obtain a training result; S3, inputting the mel spectrum graph of the audio data for training into the deep learning network in sequence, calculating the difference between the training result output by the deep learning network and the label thereof, and thereby updating the parameters of the deep learning network through back propagation; S4, obtaining audio data to be processed, performing denoising processing on the audio data to be processed, and cutting the audio data into segments to extract a mel spectrum graph; S5, inputting the mel spectrum graph of the audio data to be processed into the deep learning network, obtaining classification results from the spatio-temporal attention network and the pre-training network respectively, and obtaining a final result through fusion; The spatio-temporal attention network comprises a spatial network Densenet, a time network LSTM, and two fully connected layers, and performs the following operations: inputting the mel spectrum graph into the spatial network Densenet to extract a spatial feature vector; inputting the mel spectrum graph into the time network LSTM to extract a time feature vector; performing dot product on the spatial feature vector and the time feature vector to obtain a spatio-temporal attention weight W; performing dot product on the spatio-temporal attention weight W and the time feature vector, and then passing through the two fully connected layers to obtain the classification result of the spatio-temporal attention network.

2. The music classification method based on the primary-secondary two-path spatio-temporal feature fusion according to claim 1, characterized in that: The spatial network Densenet comprises convolution layers, pooling layers, a first Dense Block layer, a first Transition Layer layer, a second Dense Block layer, a second Transition Layer layer, a third Dense Block layer, a third Transition Layer layer, a fourth Dense Block layer, and a global pooling layer connected in sequence.

3. The music classification method based on the primary-secondary two-path spatio-temporal feature fusion according to claim 2, characterized in that: The size of the input mel spectrum graph of the main and auxiliary ways is 224*224, and the output feature map sizes of the convolution layers, the pooling layers, the first Dense Block layer, the first Transition Layer layer, the second Dense Block layer, the second Transition Layer layer, the third Dense Block layer, the third Transition Layer layer, the fourth Dense Block layer, and the global pooling layer correspond to 112*112, 56*56, 56*56, 28*28, 28*28, 14*14, 14*14, 7*7, 7*7, and 1*1 in sequence.

4. The music classification method based on the primary-secondary two-path spatio-temporal feature fusion according to claim 3, characterized in that: The filter size of the convolution layer is 7*7, and the step is 2; the pooling kernel size of the pooling layer is 3*3, and the step is 2; and the pooling kernel size of the global pooling layer is 7*7.

5. The music classification method based on the primary-secondary two-path spatio-temporal feature fusion according to claim 1, characterized in that: The pre-training network comprises a Resnet network and two fully connected layers.

6. The music classification method based on the primary-secondary two-path spatio-temporal feature fusion according to claim 1, characterized in that: The manner of calculating the difference between the training result of the deep learning network output and its label is a cross-entropy loss function.

7. The music classification method based on the primary-secondary two-path space-time feature fusion according to claim 1, characterized in that: The fusion is decision-level fusion, and the fusion ratio of the classification result of the spatio-temporal attention network and the classification result of the pre-training network is 6.5:3.5.

Images