Music genre classification method and device based on self-supervision and mixed attention mechanism

Abstract

The invention discloses a music genre classification method and device based on self-supervision and a mixed attention mechanism, and relates to the field of music classification, and the method comprises the steps: constructing and training a music genre classification model, and obtaining a trained music genre classification model; obtaining audio data to be classified, inputting the audio data to be classified into the trained music genre classification model, and extracting the audio data to be classified through a feature extraction layer to obtain a corresponding feature sequence; inputting the feature sequence into a mixed attention mechanism layer to obtain mixed features; the mixed features are input into a depth modeling layer, context-dependent modeling is carried out through a plurality of Transform coding layers, then the output features of the last Transform coding layer are input into a global average pooling layer, and a modeling vector is obtained; and inputting the modeling vector into a classification layer to obtain an audio genre prediction tag corresponding to the to-be-classified audio data. According to the method, the problems of low precision, weak applicability and the like of the existing music genre classification are solved.

Description

Technical Field

[0001] The present invention relates to the field of music classification, and in particular to a music genre classification method and device based on self-supervision and hybrid attention mechanism. Background Art

[0002] There are many music genres, including but not limited to pop music, classical music, jazz, folk music, electronic music, etc. Each genre has its own unique instrument configuration, timbre characteristics, rhythm patterns and playing techniques. The classification of music genres plays an important role in the field of Music Information Retrieval (MIR), and its applications are wide-ranging, covering rhythm detection, music composition, personalized recommendation systems, track separation, and instrument identification.

[0003] With the advancement of deep learning theory and computing hardware, researchers have widely tried to use machine learning algorithms (such as random forests, support vector machines, etc.) and deep learning models (such as convolutional neural networks (CNN), recurrent neural networks (RNN), Transformer, etc.) to extract features and model music in the field of music genre classification. Some research works convert audio signals into mel spectrograms or spectrograms, and then apply deep convolutional networks for learning; there are also literatures that use self-attention mechanisms to capture the contextual information of music sequences. However, for highly complex and multi-layered signals such as music, traditional or single deep network architectures still face certain limitations in practical applications: on the one hand, music is extremely diverse in terms of melody, harmony, arrangement, etc., and simple feature extraction is difficult to fully explore the hidden high-order semantic information; on the other hand, if only a single path is relied on for modeling, it is often difficult to take into account both time series information and frequency domain structure, resulting in the problem of insufficient accuracy in the recognition of certain styles. Especially in new music that blends multiple genres, this defect is becoming more and more obvious.

[0004] In order to further improve the accuracy and generalization ability of music genre classification, new ideas such as multi-branch network fusion, attention mechanism and data enhancement have emerged in recent years. However, existing methods still have the following pain points: First, most attention modules are single self-attention or channel attention, which makes it difficult to flexibly interact between different dimensions and scales; second, data enhancement strategies are relatively limited, mostly relying only on random segmentation or simple sampling, and cannot fully utilize time domain, frequency domain and even multimodal information; third, the modeling of audio data still relies heavily on supervised data. When the annotation is insufficient or the distribution deviation is large, the model is prone to overfitting or lack of robustness. Summary of the invention

[0005] The purpose of this application is to propose a music genre classification method and device based on self-supervision and hybrid attention mechanism in response to the above-mentioned technical problems, aiming to overcome the shortcomings of the existing technology in processing music data with diverse styles, large data scale and insufficient annotation, and to provide a higher accuracy and more applicable solution for the task of music genre classification.

[0006] In a first aspect, the present invention provides a music genre classification method based on self-supervision and hybrid attention mechanism, comprising the following steps:

[0007] Constructing and training a music genre classification model to obtain a trained music genre classification model, the trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer and a classification layer connected in sequence, the feature extraction layer adopts a self-supervised pre-trained wav2vec2.0 model, and the deep modeling layer includes several Transformer encoding layers stacked in sequence and a global average pooling layer;

[0008] The audio data to be classified is obtained, and the audio data to be classified is input into the trained music genre classification model. The audio data to be classified passes through the feature extraction layer to extract the corresponding feature sequence; the feature sequence is input into the hybrid attention mechanism layer to obtain the hybrid feature; the hybrid feature is input into the deep modeling layer, and first passes through several Transformer encoding layers for context dependency modeling, and then the output feature of the last Transformer encoding layer is input into the global average pooling layer to obtain the modeling vector; the modeling vector is input into the classification layer to obtain the audio genre prediction label corresponding to the audio data to be classified.

[0009] Preferably, the hybrid attention mechanism layer includes a multi-head self-attention layer, a first normalization layer, a channel attention layer and a second normalization layer. The feature sequence is input into the hybrid attention mechanism layer, first passes through the multi-head self-attention layer to obtain the output features of the multi-head self-attention layer, the feature sequence is added to the output features of the multi-head self-attention layer to obtain a first addition result, and then the first addition result is input into the first normalization layer to obtain an intermediate feature. The intermediate feature passes through the channel attention layer to obtain the output features of the channel attention layer, the intermediate feature and the output features of the channel attention layer are weightedly fused to obtain a first fused feature, the feature sequence and the first fused feature are added to obtain a second addition result, and then the second addition result is input into the second normalization layer to obtain a hybrid feature.

[0010] Preferably, the Transformer coding layer includes a local-global attention unit, a feedforward network unit and a third normalization layer, the local-global attention unit includes a local attention module, a global attention module and a fourth normalization layer, the feedforward network unit includes a first fully connected layer and a second fully connected layer, and the ReLU activation function is used in the first fully connected layer; when the first Transformer coding layer is used as the current Transformer coding layer, the mixed feature is used as the output feature of the previous Transformer coding layer; the output feature of the previous Transformer coding layer is input into the current Transformer coding layer, firstly passes through the local attention module and the global attention module of the local-global attention unit of the current Transformer coding layer respectively, and obtains the output feature of the local attention module and the output feature of the global attention module, and performs weighted fusion on the output feature of the local attention module and the output feature of the global attention module to obtain a second fused feature, and adds the output feature of the previous Transformer coding layer to the second fused feature to obtain a third addition result, and then inputs the third addition result into the fourth normalization layer to obtain the output feature of the local-global attention unit of the current Transformer coding layer, specifically including:

[0011] First, perform the query, key, and value operations in the local attention module as shown below:

[0012] , , ;

[0013] in, , and are the query weight, key weight, and value weight in the local attention module, respectively. , and are the query matrix, key matrix and value matrix in the local attention module respectively. Represents the output features of the l-1th Transformer encoding layer;

[0014] The local attention weight is calculated using the following formula :

[0015] ;

[0016] in, represents the dimension of the key matrix in the local attention module, is the local window centered at the tth time step, s is the sth time step in the local window, represents the key matrix in the local attention module at the s-th time step within the local window, represents the query matrix in the local attention module at the t-th time step;

[0017] The output features of the local attention module at the tth time step are calculated using the following formula: :

[0018] ;

[0019] in, Represents the value matrix in the local attention module at the s-th time step within the local window;

[0020] Secondly, perform the query, key, and value operations in the global attention module as shown below:

[0021] , , ;

[0022] in, , and are the query weight, key weight, and value weight in the global attention module, respectively. , and They are the query matrix, key matrix, and value matrix in the global attention module respectively;

[0023] The global attention weight is calculated using the following formula :

[0024] ;

[0025] in, represents the dimension of the key matrix in the global attention module, u represents the u-th time step, represents the key matrix in the global attention module at the u-th time step, represents the query matrix in the global attention module at the t-th time step;

[0026] The output features of the local attention module at the tth time step are calculated using the following formula: :

[0027] ;

[0028] in, represents the value matrix in the local attention module at the uth time step, and T represents the total number of time steps;

[0029] The output features of the local attention module at the t-th time step and the output features of the global attention module at the t-th time step are fused to obtain the second fused features at the t-th time step, as shown in the following formula:

[0030] ;

[0031] in, represents the second fusion feature at the t-th time step, represents weighted weight;

[0032] The second fusion feature and Add the residuals and pass through the fourth normalization layer to get the output features of the local-global attention unit of the lth Transformer encoding layer , as shown below:

[0033] ;

[0034] in, Representation layer normalization operation;

[0035] The output features of the local-global attention units of the current Transformer coding layer are input into the feedforward network units of the current Transformer coding layer, and pass through the first fully connected layer and the second fully connected layer in sequence to obtain the output features of the feedforward network units of the current Transformer coding layer. The output features of the feedforward network units of the current Transformer coding layer are input into the third normalization layer of the current Transformer coding layer to obtain the output features of the current Transformer coding layer. The output features of the current Transformer coding layer are used as the output features of the previous Transformer coding layer and the calculation process of the current Transformer coding layer is repeated until the output features of the last Transformer coding layer are obtained.

[0036] Preferably, the classification layer includes a Dropout layer, a third fully connected layer, a fourth fully connected layer, a Softmax function layer and an argmax function layer connected in sequence, and the ReLU activation function is adopted in the third fully connected layer; the modeling vector passes through the Dropout layer to obtain an adjusted modeling vector, and the adjusted modeling vector passes through the third fully connected layer, the fourth fully connected layer and the Softmax function layer in sequence to obtain the predicted probability distribution of each music genre, and the predicted probability distribution of each music genre is input into the argmax function layer, and the music genre corresponding to the maximum predicted probability is determined as the audio genre prediction label corresponding to the audio data to be classified.

[0037] Preferably, the wav2vec2.0 model that has undergone self-supervised pre-training includes an encoding network and a context network. During the training of the music genre classification model, the parameters of the encoding network are fixed, and the parameters of the context network are adjusted. The loss function used during the training of the music genre classification model is:

[0038] ;

[0039] in, represents the loss function, N represents the total number of samples in the training data, q represents the qth sample, c represents the cth audio genre, C represents the total number of audio genres, represents the weight of the c-th audio genre, represents the predicted probability that the qth sample belongs to the cth audio genre, The true label indicating whether the qth sample belongs to the cth audio genre.

[0040] Preferably, the process of acquiring the training data used in the training process of the music genre classification model is as follows:

[0041] Collect raw audio data with true audio genre labels;

[0042] Perform random time-domain cropping on the original audio data to obtain several audio clips;

[0043] Randomly mix and add noise to each audio clip to obtain a noisy audio clip;

[0044] Perform time-frequency masking on the noisy audio clip to obtain enhanced audio data;

[0045] The enhanced audio data and its corresponding audio genre true labels constitute the training data.

[0046] In a second aspect, the present invention provides a music genre classification device based on self-supervision and hybrid attention mechanism, comprising:

[0047] A model building module is configured to build and train a music genre classification model to obtain a trained music genre classification model, wherein the trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer, and a classification layer connected in sequence, wherein the feature extraction layer adopts a wav2vec2.0 model that has undergone self-supervised pre-training, and the deep modeling layer includes a plurality of Transformer encoding layers stacked in sequence and a global average pooling layer;

[0048] The classification module is configured to obtain audio data to be classified, input the audio data to be classified into a trained music genre classification model, and the audio data to be classified passes through a feature extraction layer to extract a corresponding feature sequence; the feature sequence is input into a hybrid attention mechanism layer to obtain a hybrid feature; the hybrid feature is input into a deep modeling layer, first passes through several Transformer encoding layers for context dependency modeling, and then the output feature of the last Transformer encoding layer is input into a global average pooling layer to obtain a modeling vector; the modeling vector is input into a classification layer to obtain an audio genre prediction label corresponding to the audio data to be classified.

[0049] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0051] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) The music genre classification method based on self-supervision and hybrid attention mechanism proposed in the present invention constructs a music genre classification model, and uses the hybrid attention mechanism layer that integrates multi-head self-attention mechanism and channel attention mechanism in the music genre classification model to perform feature weighting and fusion in different dimensions, which can capture richer information to improve the discrimination of complex music fragments. The deep modeling layer is further used to further model and deeply refine the mixed features output by the hybrid attention mechanism layer.

[0054] (2) The music genre classification method based on self-supervision and hybrid attention mechanism proposed in the present invention adopts the wav2vec2.0 model that has been pre-trained through self-supervision, which can learn universal representations on large-scale unlabeled audio data, thereby providing richer and more robust features for downstream music genre classification and reducing the difficulty of training.

[0055] (3) The music genre classification method based on self-supervision and hybrid attention mechanism proposed in the present invention further adopts diversified data enhancement technology to construct training data, which helps the music genre classification model capture the feature distribution under various time-frequency transformations and noise conditions during the training process, and can effectively improve the applicability and classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Figure 1 A flowchart of a music genre classification method based on self-supervision and hybrid attention mechanism according to an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a music genre classification model of a music genre classification method based on self-supervision and hybrid attention mechanism according to an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a hybrid attention mechanism layer of a music genre classification method based on self-supervision and hybrid attention mechanism according to an embodiment of the present application;

[0060] Figure 4 A schematic diagram of a Transformer encoding layer of a music genre classification method based on self-supervision and hybrid attention mechanism according to an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a music genre classification device based on self-supervision and hybrid attention mechanism according to an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Figure 1 A music genre classification method based on self-supervision and hybrid attention mechanism provided in an embodiment of the present application is shown, comprising the following steps:

[0065] S1, constructs a music genre classification model and trains it to obtain a trained music genre classification model. The trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer and a classification layer connected in sequence. The feature extraction layer adopts the wav2vec2.0 model that has been pre-trained by self-supervision, and the deep modeling layer includes several Transformer encoding layers stacked in sequence and a global average pooling layer.

[0066] In a specific embodiment, the process of acquiring the training data used in the training process of the music genre classification model is as follows:

[0067] Collect raw audio data with true audio genre labels;

[0068] Perform random time-domain cropping on the original audio data to obtain several audio clips;

[0069] Randomly mix and add noise to each audio clip to obtain a noisy audio clip;

[0070] Perform time-frequency masking on the noisy audio clip to obtain enhanced audio data;

[0071] The enhanced audio data and its corresponding audio genre true labels constitute the training data.

[0072] Specifically, when constructing the training data used in the training process of the music genre classification model, the original audio data is first enhanced, including random time domain cropping, random mixed noise addition, and time-frequency masking (SpecAugment). By greatly increasing the number of training samples while ensuring that the true labels of the audio genres remain unchanged, the model's adaptability to different timbres, noise environments, and music clips is improved. The specific steps are as follows:

[0073] (1) Random time domain cropping: The original audio data is sampled into segments of equal length, and the sampling rate (sr) is set to 22050 Hz. The specific method is to select a window of length of 3 seconds in the original audio data and select five non-overlapping random positions in the window. , and intercept a length at each position , get some audio clips . The original audio data is recorded as , T' represents the maximum time of the original audio data, represents the set of real numbers, then:

[0074] ;

[0075] in, represents the i-th audio segment, Represents a random position in the original audio data The length intercepted from the starting position is The audio data, , , This process generates more training samples while keeping the true labels of the audio genres unchanged.

[0076] (2) Random mixing and noise addition: After the audio segment is cut On the , different types or intensities of noise n are superimposed to form a noisy audio segment, as shown in the following formula:

[0077] ;

[0078] in, represents the i-th audio clip after adding noise, It is a noise intensity control parameter, and its value can be adjusted according to the noise type and target signal-to-noise ratio (SNR). This step can effectively improve the model's adaptability to various environmental noises.

[0079] (3) Time-frequency masking (SpecAugment): In order to simulate the similar effect of SpecAugment at the time domain waveform level, the time domain interval can be randomly selected , set the values ​​in this interval to 0. Mathematically, it can be expressed as:

[0080] ;

[0081] in, Indicates that the i-th enhanced audio data is in The value at the time step, Indicates that the i-th noisy audio segment is in The enhanced audio data contains multiple enhancement effects such as random cropping, noise addition, and time domain occlusion. Finally, all enhanced audio data and their true audio genre labels are used as training data, and the music genre classification model is trained using the training data. Since the wav2vec2.0 model is directly used to model the time domain waveform in the subsequent process, the enhanced audio data is used as the input data of the music genre classification model during the training process.

[0082] refer to Figure 2 The music genre classification model includes feature extraction layer, hybrid attention mechanism layer, deep modeling layer and classification layer. First, the feature extraction layer is constructed. The feature extraction layer uses the wav2vec2.0 model that has been pre-trained by self-supervision. The model has been pre-trained on large-scale unlabeled audio data and has the ability to deeply represent the contextual information of audio sequences.

[0083] The enhanced audio data is input into the wav2vec2.0 model, and the context representation learned on large-scale unlabeled audio is used to perform deep feature extraction on the input audio data to obtain a feature sequence with more robustness and generalization ability. The specific steps include: Input the encoder network of the wav2vec2.0 model, without manually extracting mel-spectrograms or other traditional audio features; use the encoder network of the wav2vec2.0 model composed of multiple convolutional layers to capture short-term features in the time domain and output a potential representation sequence. Since we still want to retain the "self-supervision" feature in the fine-tuning stage, mask a part of the potential representation sequence to obtain a masked representation sequence, so that the model can infer the masked frame based on the unmasked information. Input the masked representation sequence into the context network of the wav2vec2.0 model, which adopts a transformer structure and outputs a context representation sequence, which is used as the feature sequence output by the feature extraction layer.

[0084] Secondly, a hybrid attention mechanism layer that combines multi-head self-attention and channel attention is constructed to jointly model the feature sequence output from the feature extraction layer in terms of time series dimension and feature channel dimension. On the one hand, long-range dependencies are captured in the feature sequence; on the other hand, key frequency bands or instrument parts are focused to improve the discrimination of complex music clips.

[0085] Secondly, a deep modeling layer is constructed to further perform sequence modeling and deep semantic extraction on the mixed features output by the hybrid attention mechanism layer. The network design takes into account both structural simplicity and efficiency to ensure that the computing power requirements are relatively controllable during deployment.

[0086] Finally, a classification layer is constructed, and the modeling vector output by the deep modeling layer is passed into the classification layer to obtain the corresponding audio genre prediction label.

[0087] In a specific embodiment, the wav2vec2.0 model that has undergone self-supervised pre-training includes an encoding network and a context network. During the training of the music genre classification model, the parameters of the encoding network are fixed, and the parameters of the context network are adjusted. The loss function used during the training of the music genre classification model is:

[0088] ;

[0089] in, represents the loss function, N represents the total number of samples in the training data, q represents the qth sample, c represents the cth audio genre, C represents the total number of audio genres, represents the weight of the c-th audio genre, represents the predicted probability that the qth sample belongs to the cth audio genre, The true label indicating whether the qth sample belongs to the cth audio genre.

[0090] Specifically, a loss function is constructed according to the real audio genre label and the predicted audio genre label, the music genre classification model is trained based on the above loss function, and the parameters are updated using an optimizer such as stochastic gradient descent or Adam, so as to train the model and obtain a trained music genre classification model. During the training process, since the feature extraction layer uses the wav2vec2.0 model that has been pre-trained by self-supervision, during the training process of the music genre classification model, the parameters of the encoding network in the wav2vec2.0 model are frozen, and the parameters of the context network in the wav2vec2.0 model are adjusted.

[0091] S2, obtain the audio data to be classified, input the audio data to be classified into the trained music genre classification model, the audio data to be classified passes through the feature extraction layer, and the corresponding feature sequence is extracted; the feature sequence is input into the hybrid attention mechanism layer to obtain the hybrid feature; the hybrid feature is input into the deep modeling layer, first passes through several Transformer encoding layers for context dependency modeling, and then the output feature of the last Transformer encoding layer is input into the global average pooling layer to obtain the modeling vector; the modeling vector is input into the classification layer to obtain the audio genre prediction label corresponding to the audio data to be classified.

[0092] Specifically, the trained music genre classification model is deployed. In the inference stage, the acquired audio data to be classified is input into the trained music genre classification model, and the audio data to be classified is input into the feature extraction layer of the wav2vec2.0 model to extract the corresponding feature sequence. The feature sequence is then passed through the hybrid attention mechanism layer and the deep modeling layer to obtain a modeling vector. The modeling vector is input into the classification layer to obtain the corresponding audio genre prediction label.

[0093] In a specific embodiment, the hybrid attention mechanism layer includes a multi-head self-attention layer, a first normalization layer, a channel attention layer and a second normalization layer. The feature sequence is input into the hybrid attention mechanism layer, first passes through the multi-head self-attention layer to obtain the output features of the multi-head self-attention layer, the feature sequence is added to the output features of the multi-head self-attention layer to obtain a first addition result, and then the first addition result is input into the first normalization layer to obtain an intermediate feature. The intermediate feature passes through the channel attention layer to obtain the output features of the channel attention layer, the intermediate feature and the output features of the channel attention layer are weightedly fused to obtain a first fused feature, the feature sequence and the first fused feature are added to obtain a second addition result, and then the second addition result is input into the second normalization layer to obtain a hybrid feature.

[0094] Specifically, refer to Figure 3 The multi-head self-attention layer adopts the multi-head self-attention mechanism (MHA). The calculation process of the multi-head self-attention mechanism is the existing calculation process, and its details are not repeated here. The output features of the multi-head self-attention layer are residually connected with the feature sequence, and then pass through the first normalization layer to obtain the intermediate features, which are input into the channel attention layer for the calculation of the channel attention mechanism (SE). The calculation process of the channel attention mechanism is also the existing calculation process, and its details are not repeated here. Then a weighted fusion strategy is adopted to simultaneously introduce the multi-head attention mechanism and the channel attention mechanism to obtain the first fusion feature, and finally the first fusion feature is residually connected with the feature sequence, and then passes through the second normalization layer to further stabilize the training and accelerate convergence.

[0095] In a specific embodiment, the Transformer coding layer includes a local-global attention unit, a feedforward network unit and a third normalization layer, the local-global attention unit includes a local attention module, a global attention module and a fourth normalization layer, the feedforward network unit includes a first fully connected layer and a second fully connected layer, and the ReLU activation function is used in the first fully connected layer; when the first Transformer coding layer is used as the current Transformer coding layer, the mixed features are used as the output features of the previous Transformer coding layer; the output features of the previous Transformer coding layer are input into the current Transformer coding layer, and first pass through the local attention module and the global attention module of the local-global attention unit of the current Transformer coding layer respectively to obtain the output features of the local attention module and the output features of the global attention module, and weightedly fuse the output features of the local attention module and the output features of the global attention module to obtain the second fused features, and the output features of the previous Transformer coding layer are weightedly fused. The output feature is added to the second fusion feature to obtain a third addition result, and the third addition result is input into the fourth normalization layer to obtain the output feature of the local-global attention unit of the current Transformer coding layer; the output feature of the local-global attention unit of the current Transformer coding layer is input into the feedforward network unit of the current Transformer coding layer, and passes through the first fully connected layer and the second fully connected layer in sequence to obtain the output feature of the feedforward network unit of the current Transformer coding layer; the output feature of the feedforward network unit of the current Transformer coding layer is input into the third normalization layer of the current Transformer coding layer to obtain the output feature of the current Transformer coding layer; the output feature of the current Transformer coding layer is used as the output feature of the previous Transformer coding layer and the calculation process of the current Transformer coding layer is repeated until the output feature of the last Transformer coding layer is obtained.

[0096] Specifically, refer to Figure 4 , the mixed features obtained by the hybrid attention mechanism layer after integrating multi-head self-attention and channel / spatial attention As the input data of the deep modeling layer, denoted as , where d represents the number of channels, and T represents the total number of time steps. The mixed feature has been enhanced in both the time dimension and the channel dimension, and can better characterize the timbre, rhythm, melody and other elements of various musical instruments in music. In an embodiment of the present application, a Transformer encoding layer with L stacked layers is set, and each Transformer encoding layer includes two unit blocks, namely a local-global attention unit and a feedforward network unit, and cooperates with a third normalization layer; wherein the local-global attention unit includes a local attention module, a global attention module and a fourth normalization layer. After completing the mixed attention mechanism, the Transformer encoding layer focuses on deeper context-dependent modeling of the mixed features, and fully integrates the instrument sounds and melody structures at different time steps.

[0097] The calculation process of the deep modeling layer is as follows:

[0098] The output features of the l-1th Transformer encoding layer The data is input into the lth Transformer encoding layer and first passes through the local attention module and the global attention module in the local-global attention unit of the lth Transformer encoding layer;

[0099] First, perform the query, key, and value operations in the local attention module as shown below:

[0100] , , ;

[0101] in, , and are the query weight, key weight, and value weight in the local attention module, respectively. , and are the query matrix, key matrix and value matrix in the local attention module respectively. Represents the output features of the l-1th Transformer encoding layer;

[0102] The local attention weight is calculated using the following formula :

[0103] ;

[0104] in, represents the dimension of the key matrix in the local attention module, is the local window centered at the tth time step, s is the sth time step in the local window, represents the key matrix in the local attention module at the s-th time step within the local window, represents the query matrix in the local attention module at the t-th time step;

[0105] The output features of the local attention module at the tth time step are calculated using the following formula: :

[0106] ;

[0107] in, Represents the value matrix in the local attention module at the s-th time step within the local window;

[0108] Secondly, perform the query, key, and value operations in the global attention module as shown below:

[0109] , , ;

[0110] in, , and are the query weight, key weight, and value weight in the global attention module, respectively. , and They are the query matrix, key matrix, and value matrix in the global attention module respectively;

[0111] The global attention weight is calculated using the following formula :

[0112] ;

[0113] in, represents the dimension of the key matrix in the global attention module, u represents the u-th time step, represents the key matrix in the global attention module at the u-th time step, represents the query matrix in the global attention module at the t-th time step;

[0114] The output features of the local attention module at the tth time step are calculated using the following formula: :

[0115] ;

[0116] in, represents the value matrix in the local attention module at the uth time step, and T represents the total number of time steps;

[0117] The output features of the local attention module at the t-th time step and the output features of the global attention module at the t-th time step are fused to obtain the second fused features at the t-th time step, as shown in the following formula:

[0118] ;

[0119] in, represents the second fusion feature at the t-th time step, represents weighted weight;

[0120] The second fusion feature and Add the residuals and pass through the fourth normalization layer to get the output features of the local-global attention unit of the lth Transformer encoding layer , as shown below:

[0121] ;

[0122] in, Representation layer normalization operation.

[0123] The output features of the local-global attention unit of the lth Transformer encoding layer Input to the feedforward network unit of the lth Transformer encoding layer, through the first fully connected layer and the second fully connected layer Perform full connection mapping to obtain the output features of the feedforward network unit of the lth Transformer encoding layer , as shown below:

[0124] ;

[0125] in, represents the ReLU activation function, and are the weights of the first fully connected layer and the second fully connected layer, and They are the bias items corresponding to the first fully connected layer and the second fully connected layer respectively.

[0126] right and Add the residuals to get the output features of the lth Transformer encoding layer , as shown below:

[0127] ;

[0128] The output features of the lth Transformer encoding layer Pass it to the l+1th Transformer encoding layer until the stacking of the Lth Transformer encoding layer is completed, and finally the output feature of the Lth Transformer encoding layer is obtained .Will Input to the global average pooling layer to map the variable-length sequence into a fixed-length modeling vector , as shown below:

[0129] ;

[0130] in, Represents a global average pooling operation.

[0131] In a specific embodiment, the classification layer includes a Dropout layer, a third fully connected layer, a fourth fully connected layer, a Softmax function layer and an argmax function layer connected in sequence, and the ReLU activation function is used in the third fully connected layer; the modeling vector passes through the Dropout layer to obtain an adjusted modeling vector, and the adjusted modeling vector passes through the third fully connected layer, the fourth fully connected layer and the Softmax function layer in sequence to obtain the predicted probability distribution of each music genre, and the predicted probability distribution of each music genre is input into the argmax function layer, and the music genre corresponding to the maximum predicted probability is determined as the audio genre prediction label corresponding to the audio data to be classified.

[0132] Specifically, the modeling vector As the input vector of the classification layer, in order to improve training stability or prevent overfitting, the input vector of the classification layer is first subjected to Dropout by the Dropout layer. The probability of Dropout is set to , then the jth element in the input vector of the classification layer is The probability of zero is set to obtain the adjusted modeling vector This operation helps to alleviate overfitting caused by insufficient data or highly correlated features, and can be controlled by hyperparameters in actual training. Adjust its intensity.

[0133] In order to allow the network to perform certain nonlinear combinations before output, the third and fourth fully connected layers are set to make the adjusted modeling vector It passes through the third fully connected layer and the fourth fully connected layer in sequence, as shown in the following formula:

[0134] ;

[0135] ;

[0136] in, and Represent the weight and bias of the third fully connected layer, respectively. represents the output features of the third connection layer, and Represent the weight and bias of the fourth fully connected layer, respectively. Represents the output features of the fourth connection layer.

[0137] Then apply the Softmax function for normalization to obtain the predicted probability distribution of each music genre, and finally use the argmax function to select the music genre with the largest predicted probability as the final music genre prediction label, as shown in the following formula:

[0138] ;

[0139] ;

[0140] in, represents the value of the c-th audio genre in the output feature of the fourth connection layer corresponding to the audio data to be classified, represents the value of the a-th audio genre in the output feature of the fourth connection layer corresponding to the audio data to be classified, represents the predicted probability that the audio data f to be classified belongs to the cth audio genre, represents the music genre prediction label corresponding to the audio data f to be classified, Indicates taking the audio genre index with the highest prediction probability.

[0141] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a music genre classification device based on self-supervision and hybrid attention mechanism. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0142] The embodiment of the present application provides a music genre classification device based on self-supervision and hybrid attention mechanism, comprising:

[0143] A model building module 1 is configured to build and train a music genre classification model to obtain a trained music genre classification model, wherein the trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer, and a classification layer connected in sequence, wherein the feature extraction layer adopts a wav2vec2.0 model that has undergone self-supervised pre-training, and the deep modeling layer includes a plurality of Transformer encoding layers stacked in sequence and a global average pooling layer;

[0144] The classification module 2 is configured to obtain audio data to be classified, input the audio data to be classified into a trained music genre classification model, and the audio data to be classified passes through a feature extraction layer to extract a corresponding feature sequence; the feature sequence is input into a hybrid attention mechanism layer to obtain a hybrid feature; the hybrid feature is input into a deep modeling layer, first passes through several Transformer encoding layers for context dependency modeling, and then the output feature of the last Transformer encoding layer is input into a global average pooling layer to obtain a modeling vector; the modeling vector is input into a classification layer to obtain an audio genre prediction label corresponding to the audio data to be classified.

[0145] Figure 6 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 6 As shown, the electronic device of this embodiment includes: a processor 601 and a memory 602; wherein the memory 602 is used to store computer-executable instructions; the processor 601 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0146] Optionally, the memory 602 may be independent or integrated with the processor 601 .

[0147] When the memory 602 is independently provided, the electronic device further includes a bus 603 for connecting the memory 602 and the processor 601 .

[0148] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 601 executes the computer execution instructions, the above method is implemented.

[0149] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 601, the above method is implemented.

[0150] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0151] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0152] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0153] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 601 to perform some steps of the methods of various embodiments of the present application.

[0154] It should be understood that the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 601 may be any conventional processor 601, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 601, or may be executed by a combination of hardware and software modules in the processor 601.

[0155] The memory 602 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0156] The bus 603 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 603 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 603 in the drawings of the present application is not limited to only one bus 603 or one type of bus 603.

[0157] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0158] An exemplary storage medium is coupled to the processor 601, so that the processor 601 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 601. The processor 601 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 601 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0159] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A music genre classification method based on self-supervision and hybrid attention mechanism, characterized in that: The following steps are involved: Constructing and training a music genre classification model to obtain a trained music genre classification model, wherein the trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer and a classification layer connected in sequence, wherein the feature extraction layer adopts a wav2vec2.0 model that has undergone self-supervised pre-training, and the deep modeling layer includes a plurality of Transformer encoding layers stacked in sequence and a global average pooling layer; Acquire audio data to be classified, input the audio data to be classified into the trained music genre classification model, pass the audio data to be classified through the feature extraction layer, and extract the corresponding feature sequence; input the feature sequence into the hybrid attention mechanism layer to obtain hybrid features; input the hybrid features into the deep modeling layer, first pass through several Transformer encoding layers for context dependency modeling, and then input the output features of the last Transformer encoding layer into the global average pooling layer to obtain a modeling vector; input the modeling vector into the classification layer to obtain an audio genre prediction label corresponding to the audio data to be classified.

2. The music genre classification method based on self-supervision and hybrid attention mechanism according to claim 1, characterized in that: The hybrid attention mechanism layer includes a multi-head self-attention layer, a first normalization layer, a channel attention layer and a second normalization layer. The feature sequence is input into the hybrid attention mechanism layer, first passes through the multi-head self-attention layer to obtain the output feature of the multi-head self-attention layer, the feature sequence is added to the output feature of the multi-head self-attention layer to obtain a first addition result, and then the first addition result is input into the first normalization layer to obtain an intermediate feature, the intermediate feature passes through the channel attention layer to obtain the output feature of the channel attention layer, the intermediate feature and the output feature of the channel attention layer are weightedly fused to obtain a first fused feature, the feature sequence and the first fused feature are added to obtain a second addition result, and then the second addition result is input into the second normalization layer to obtain the hybrid feature.

3. The music genre classification method based on self-supervision and hybrid attention mechanism according to claim 1, characterized in that: The Transformer coding layer includes a local-global attention unit, a feedforward network unit and a third normalization layer, the local-global attention unit includes a local attention module, a global attention module and a fourth normalization layer, the feedforward network unit includes a first fully connected layer and a second fully connected layer, and the first fully connected layer adopts a ReLU activation function; when the first Transformer coding layer is used as the current Transformer coding layer, the mixed features are used as the output features of the previous Transformer coding layer; the output features of the previous Transformer coding layer are input into the current Transformer coding layer, and are first respectively passed through the current The local attention module and the global attention module of the local-global attention unit of the Transformer coding layer are used to obtain the output features of the local attention module and the output features of the global attention module, and the output features of the local attention module and the output features of the global attention module are weightedly fused to obtain a second fused feature, and the output feature of the previous Transformer coding layer is added to the second fused feature to obtain a third addition result, and the third addition result is input into the fourth normalization layer to obtain the output feature of the local-global attention unit of the current Transformer coding layer, specifically including first performing the query, key, and value operations in the local attention module, as shown in the following formula: ， ， ； in, , and are the query weight, key weight, and value weight in the local attention module, respectively. , and are the query matrix, key matrix and value matrix in the local attention module respectively. Represents the output features of the l-1th Transformer encoding layer; The local attention weight is calculated using the following formula : ； in, represents the dimension of the key matrix in the local attention module, is the local window centered at the tth time step, s is the sth time step in the local window, represents the key matrix in the local attention module at the s-th time step within the local window, represents the query matrix in the local attention module at the t-th time step; The output features of the local attention module at the tth time step are calculated using the following formula: : ； in, Represents the value matrix in the local attention module at the s-th time step within the local window; Secondly, perform the query, key, and value operations in the global attention module as shown below: ， ， ； in, , and are the query weight, key weight, and value weight in the global attention module, respectively. , and They are the query matrix, key matrix and value matrix in the global attention module respectively; The global attention weight is calculated using the following formula : ； in, represents the dimension of the key matrix in the global attention module, u represents the u-th time step, represents the key matrix in the global attention module at the u-th time step, represents the query matrix in the global attention module at the t-th time step; The output features of the local attention module at the tth time step are calculated using the following formula: : ； in, represents the value matrix in the local attention module at the uth time step, and T represents the total number of time steps; The output features of the local attention module at the t-th time step and the output features of the global attention module at the t-th time step are fused to obtain the second fused features at the t-th time step, as shown in the following formula: ； in, represents the second fusion feature at the t-th time step, represents weighted weight; The second fusion feature and Add the residuals and pass through the fourth normalization layer to get the output features of the local-global attention unit of the lth Transformer encoding layer , as shown below: ； in, Representation layer normalization operation; The output features of the local-global attention unit of the current Transformer coding layer are input into the feedforward network unit of the current Transformer coding layer, and sequentially pass through the first fully connected layer and the second fully connected layer to obtain the output features of the feedforward network unit of the current Transformer coding layer. The output features of the feedforward network unit of the current Transformer coding layer are input into the third normalization layer of the current Transformer coding layer to obtain the output features of the current Transformer coding layer. The output features of the current Transformer coding layer are used as the output features of the previous Transformer coding layer and the calculation process of the current Transformer coding layer is repeated until the output features of the last Transformer coding layer are obtained.

4. The music genre classification method based on self-supervision and hybrid attention mechanism according to claim 1, characterized in that: The classification layer includes a Dropout layer, a third fully connected layer, a fourth fully connected layer, a Softmax function layer and an argmax function layer connected in sequence, and the ReLU activation function is used in the third fully connected layer; the modeling vector passes through the Dropout layer to obtain an adjusted modeling vector, and the adjusted modeling vector passes through the third fully connected layer, the fourth fully connected layer and the Softmax function layer in sequence to obtain the predicted probability distribution of each music genre, and the predicted probability distribution of each music genre is input into the argmax function layer, and the music genre corresponding to the maximum predicted probability is determined as the audio genre prediction label corresponding to the audio data to be classified.

5. The music genre classification method based on self-supervision and hybrid attention mechanism according to claim 1, characterized in that: The self-supervised pre-trained wav2vec2.0 model includes an encoding network and a context network. During the training of the music genre classification model, the parameters of the encoding network are fixed, and the parameters of the context network are adjusted. The loss function used during the training of the music genre classification model is: ； in, represents the loss function, N represents the total number of samples in the training data, q represents the qth sample, c represents the cth audio genre, C represents the total number of audio genres, represents the weight of the c-th audio genre, represents the predicted probability that the qth sample belongs to the cth audio genre, The true label indicating whether the qth sample belongs to the cth audio genre.

6. The music genre classification method based on self-supervision and hybrid attention mechanism according to claim 1, characterized in that: The process of obtaining the training data used in the training process of the music genre classification model is as follows: Collect raw audio data with true audio genre labels; Performing random time-domain cropping on the original audio data to obtain a plurality of audio clips; Randomly mix and add noise to each audio clip to obtain a noisy audio clip; Performing time-frequency masking on the noisy audio clip to obtain enhanced audio data; The enhanced audio data and its corresponding audio genre true labels constitute the training data.

7. A music genre classification device based on self-supervision and hybrid attention mechanism, characterized in that: include: A model building module is configured to build and train a music genre classification model to obtain a trained music genre classification model, wherein the trained music genre classification model includes a feature extraction layer, a hybrid attention mechanism layer, a deep modeling layer and a classification layer connected in sequence, wherein the feature extraction layer adopts a wav2vec2.0 model that has undergone self-supervised pre-training, and the deep modeling layer includes a plurality of Transformer encoding layers stacked in sequence and a global average pooling layer; The classification module is configured to obtain audio data to be classified, input the audio data to be classified into the trained music genre classification model, and the audio data to be classified passes through the feature extraction layer to extract the corresponding feature sequence; the feature sequence is input into the hybrid attention mechanism layer to obtain a hybrid feature; the hybrid feature is input into the deep modeling layer, first passes through several Transformer encoding layers for context dependency modeling, and then the output feature of the last Transformer encoding layer is input into the global average pooling layer to obtain a modeling vector; the modeling vector is input into the classification layer to obtain an audio genre prediction label corresponding to the audio data to be classified.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images