Piano music emotion recognition method and system based on joint cross attention

Abstract

The present invention belongs to the technical field of music information processing, and relates to a piano music emotion recognition method and system based on joint cross-attention, the method converts piano music data into music score files and resamples the music score files to obtain resampled music score samples; and converts piano music data into music Mel-gram spectrograms; extracts features from the sampled music score samples to obtain music score features; extracts features from the music Mel-gram spectrograms to obtain music Mel-gram spectrogram features; and processes the music score features and the music Mel-gram spectrograms through a joint cross-attention Transformer encoder to obtain music score joint cross features and music score joint cross features for classification prediction. The present invention can effectively model the context of the current generation position, and improve the efficiency and accuracy of emotion recognition.

Description

Technical Field

[0001] The invention belongs to the technical field of music information processing, and relates to a piano music emotion recognition method and system based on joint cross attention. Background Art

[0002] As an indispensable part of the brilliant culture created by humans, music is not only a form of artistic expression, but also has many positive effects, such as promoting emotional expression, enhancing memory, and relieving stress. However, how to identify the emotions expressed by music has always been a challenge. Although humans can feel the emotions expressed by music by listening to it, music itself is subjective, abstract, and individual, making it difficult to accurately locate the emotions expressed by music.

[0003] At present, music emotion recognition based on machine learning mainly includes four parts: step 1: music data collection and analysis, step 2: music feature extraction, step 3: emotion model selection, step 4: emotion recognition model training and testing. In step 1, it is necessary to collect music data sets and analyze the data sets. In the music feature extraction stage, the relevant features useful for emotion recognition are mainly extracted. The emotion recognition model is selected according to the analysis of the data set and the extraction of music features. The emotion recognition model training and testing mainly completes the training and prediction process of music features.

[0004] The two parts that have the greatest impact on the accuracy of music emotion recognition are the extraction of music emotion features and the establishment of emotion recognition models. In the early days, the research on music emotion often used a single music emotion feature or was based on traditional machine learning models. This would make the results non-generalizable because each piece of music expresses emotions in different ways. Therefore, when performing the recognition task of another music dataset, it is necessary to re-extract, which is very inefficient and greatly reduces the recognition accuracy. Summary of the invention

[0005] In order to improve the accuracy of music emotion recognition, the present invention proposes a piano music emotion recognition method and system based on joint cross-attention, which performs joint cross-attention operations on music notation features and music Mel-atlas features, and uses them together for emotion recognition prediction.

[0006] The present invention is implemented by the following technical scheme: a piano music emotion recognition method based on joint cross attention, the steps are as follows:

[0007] Step 1: Piano music data collection;

[0008] Step 2: converting the piano music data into a music notation file and resampling the music notation file to obtain a resampled music notation sample; and converting the piano music data into a music Mel-atlas;

[0009] Step 3: Extract features from the sampled music notation samples to obtain music notation features; extract features from the music Mel-atrogram to obtain music Mel-atrogram features;

[0010] Step 4: Encode the music notation features through the music notation feature encoder, and encode the music mel-spectrum features through the music mel-spectrum feature encoder. The music mel-spectrum feature encoder obtains the music mel-spectrum feature encoder information related to the music notation feature encoding position through the cross-attention mechanism, and outputs the music spectrum joint cross feature. The music notation feature encoder obtains the music notation feature encoder information related to the music Mel-atrogram encoding position through the cross-attention mechanism, and outputs the music notation joint cross-feature ;in, is the music graph cross feature, For the crossover feature of music notation, It is the cross feature of music notation The SA transformer encoder processing result is: It is the cross feature of music notation The result after cross-attention RMS normalization, is the music graph cross feature The SA transformer encoder processing result is: is the music graph cross feature The result after cross-attention RMS normalization;

[0011] Step 5: Use the SA Transformer encoder to capture the pixel-level music graph joint cross-features and music notation joint cross-feature pairwise relationships and channel dependency extraction, then use the linear layer to linearly transform the extraction results to obtain the output of the specified dimension, and finally obtain the prediction result through the softmax layer.

[0012] Further preferably, in step 2, the shortest duration of each note in the music score file is 0.25 seconds, which is defined as a single character block, and every 32 character blocks constitute a new music score sample, then each piece of music Perform the following processing:

[0013] ;

[0014] in, Indicates music The resampled music score sample. Indicates Music score files, Indicates the number of music score files. Function refers to the The character block with a length of 32 is taken as the first A music score file, and the insufficient part at the end is discarded.

[0015] Further preferably, in step 3, a long short-term memory network (LSTM) is used to extract features of the sampled music notation samples.

[0016] Further preferably, in step 3, a ResNet backbone network is used to perform feature extraction on the music mel-atlas.

[0017] Further preferably, the music Mel-spectrum feature encoder and the music notation feature encoder have the same structure. They are both composed of two sub-layers connected in series, and the first sub-layer includes a cross self-attention mechanism, a root mean square normalization layer, and a residual connection in sequence; the first sub-layer of the music Mel-spectrum feature encoder obtains the music notation feature encoder information related to the music Mel-spectrum feature encoding position through the cross attention mechanism, and then obtains the music spectrum cross feature through the root mean square normalization layer and the residual connection operation. The first sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then obtains the music notation cross feature through the root mean square normalization layer and the residual connection operation. .

[0018] Further preferably, the second sublayer includes two branches in sequence, one branch is a cross self-attention mechanism and a root mean square normalization layer, and the other branch is a SA Transformer encoder, which processes the cross features of the music spectrum. Or music notation cross feature Processing to obtain or , the outputs of the two branches are element-wise added. Similarly, the second sublayer of the music Mel-spectrogram feature encoder obtains the music notation feature encoder information related to the music Mel-spectrogram feature encoding position through the cross attention mechanism, and then processes it through the RMS normalization layer to obtain The second sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then processes it through the root mean square normalization layer to obtain .

[0019] Further preferably, the cross self-attention mechanism is: the music mel-atlas feature As the key vector of attention Sum value vector , the musical notation features Query vector as attention ; Using query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music Mel spectrum encoder information.

[0020] Similarly, the musical notation features As the key vector of attention Sum value vector , the music Mel-atrogram feature Query vector as attention , using the query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music score feature encoder information.

[0021] Further preferably, the RMS normalization layer abandons the centering operation, and the normalization process only implements scaling, and the scaling coefficient is the RMS, and when the mean is 0, it is RMS normalization.

[0022] The present invention also provides a system for implementing a piano music emotion recognition method based on joint cross attention, comprising:

[0023] Data acquisition module, used for collecting piano music data;

[0024] A data preprocessing module, used for converting piano music data into music score samples and music Mel-grams;

[0025] A feature extraction module is used to extract features from music notation samples and music mel-atlas to obtain music notation features and music mel-atlas features;

[0026] A feature fusion module is used to perform a joint cross-attention operation on the music notation features and the music Mel-atlas features, and output the music atlas joint cross-features and the music notation joint cross-features;

[0027] The recognition module is used to recognize the joint cross features of music spectra and the joint cross features of music notations, and output prediction results.

[0028] Beneficial effects of the present invention: A joint cross-attention Transformer encoder is proposed for transfer learning of piano music emotion recognition, which can enable the encoder to effectively model the context of the current generation position and improve the efficiency and accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flow chart of the method of the present invention.

[0030] Figure 2 The experimental results are compared for different piano music emotion recognition methods. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below with reference to the accompanying drawings.

[0032] Reference Figure 1 , a piano music emotion recognition method based on joint cross attention, the steps are as follows:

[0033] Step 1: Piano music data collection;

[0034] According to the aesthetic characteristics of the audience, music is also constrained and influenced by factors such as pitch and scale. Through the study of music, musicologists have decomposed music into 8 emotional elements, as shown in Table 1. Each column is an emotional type. Each song emotional group can be represented by many words, and the meanings of the words in each group are also very similar.

[0035] Table 1

[0036]

[0037] The piano music datasets selected by the present invention include a self-built dataset and the MAESTRO Dataset open source dataset. The self-built dataset is based on the 8 different emotions in Table 1. Each emotion collects 10 different styles of music, and the differences between each style can also be felt.

[0038] Step 2: converting the piano music data into a music notation file and resampling the music notation file to obtain a resampled music notation sample; and converting the piano music data into a music Mel-atlas;

[0039] The music score file records the type of instrument played, the pitch, duration and intensity of each note. Through these musical elements, we can create colorful music. For example, higher pitch and stronger intensity are used to express joy, excitement and excitement, and soft pitch and intensity are used to express tranquility, comfort and relaxation. Therefore, each musical element is indispensable. Through the description of various musical elements in the original music through these digital information, we can more accurately identify and record various notes, chords, rhythms and other elements in the music, so as to more accurately identify the emotions of piano music.

[0040] The effect of music emotion recognition is closely related to the representation of music features. The audio in the dataset cannot be used directly for model training. Before data training, the data needs to be processed in a series of ways to meet the input format requirements of the model. Since the length of music score files varies, the music score files need to be processed. The minimum duration of each note in the music score file is 0.25 seconds, which is defined as a single character block. Every 32 character blocks constitute a new music score sample. Then, each piece of music Perform the following processing:

[0041] ;

[0042] in, Indicates music The resampled music score sample. Indicates Music score files, Indicates the number of music score files. Function refers to the The character block with a length of 32 is taken as the first A music score file, and the insufficient part at the end is discarded.

[0043] Using music mel spectrograms is more in line with the auditory characteristics of the human ear. According to human physiological research, the human ear has different degrees of auditory response to sound signals of different frequencies. The human auditory system's sensory effect on sound signals shows a logarithmic relationship with the frequency of the sound signal, that is, the relationship between the mel frequency B(f) and the actual frequency f can be expressed by the following formula: .

[0044] Step 3: Use the long short-term memory network (LSTM) to extract features from the sampled music notation samples to obtain music notation features; use the ResNet backbone network to extract features from the music Mel-atlas to obtain music Mel-atlas features;

[0045] Long short-term memory network (LSTM) is a variant of recurrent neural network. It not only inherits the advantages of traditional recurrent neural network, but also overcomes the gradient explosion or vanishing problem of recurrent neural network. It can effectively process sequence data of any length and capture long-term data dependencies. The total amount of data processed and the training speed are greatly improved compared with traditional machine learning models. Long short-term memory network is composed of many repeating units, which are called memory blocks. Each memory block contains three gates and a memory unit. The three gates are forget gate, input gate and output gate.

[0046] Forget gate: The first thing in the internal structure of LSTM is the forget gate, which determines which output information of the previous position should be forgotten. This is mainly achieved through a sigmoid activation function layer and a point-by-point multiplication operation. Each element output by the sigmoid activation function layer is a real number between 0 and 1, which represents the weight of allowing the corresponding information to pass. Among them, 0 means "do not allow any information to pass", and 1 means "allow all information to pass". The input of this layer is the output of the previous position. and the input of the current position Make a connection, then go through a linear transformation, and finally use the sigmoid activation function to select which information to forget. The calculation formula is:

[0047] ;

[0048] in, represents the characteristics of the forget gate output, represents the forget gate bias term, represents the weight parameter matrix of the forget gate, Represents the sigmoid activation function.

[0049] Input gate: The second part of the LSTM internal structure is the input gate. It mainly consists of two parts. The input is also the output of the previous position. and the input of the current position Make a connection, then do a linear transformation. The first part updates the information, and the second part determines which parts need to be updated. The first is the sigmoid activation function layer, which determines which values ​​need to be updated, and the second is the tanh function layer to update the input information. The calculation formula is:

[0050] ;

[0051] ;

[0052] in, Indicates information that needs to be updated. represents the updated weight parameter matrix of the input gate, represents the bias term of the input gate, Indicates updating the previous position output state, Represents the weight parameter matrix of the tanh function layer, is the bias term of the tanh function layer;

[0053] Output gate: The last structure inside LSTM is the output gate. First, the sigmoid layer is used to determine which information needs to be output. Then the neuron state is multiplied by the tanh function to update the neuron state information. Finally, the desired output part is obtained after multiplication.

[0054] Step 4: Encode the music notation features through the music notation feature encoder, and encode the music mel-spectrum features through the music mel-spectrum feature encoder. The music mel-spectrum feature encoder obtains the music mel-spectrum feature encoder information related to the music notation feature encoding position through the cross-attention mechanism, and outputs the music spectrum joint cross feature. The music notation feature encoder obtains the music notation feature encoder information related to the music Mel-atrogram encoding position through the cross-attention mechanism, and outputs the music notation joint cross-feature ;in, is the music graph cross feature, For the crossover feature of music notation, It is the cross feature of music notation The SA transformer encoder processing result is: It is the cross feature of music notation The result after cross-attention RMS normalization, is the music graph cross feature The SA transformer encoder processing result is: is the music graph cross feature The results after cross-attention RMS normalization.

[0055] The present invention proposes a joint cross-attention Transformer encoder for transfer learning of piano music emotion recognition. In order to cross the information between the music mel-spectrum feature encoder and the music notation feature encoder, a joint cross-attention is constructed. The music mel-spectrum feature encoder will adjust the attention of the output of the music notation feature encoder to obtain the music notation feature encoder information related to the music mel-spectrum feature encoding position, and can also obtain the music mel-spectrum encoder information related to the music notation feature encoding position. The encoder can effectively model the context of the current generation position.

[0056] The music Mel-spectrum feature encoder and the music notation feature encoder have the same structure. They are both composed of two sub-layers connected in series. The first sub-layer includes a cross self-attention mechanism, a root mean square normalization layer, and a residual connection in sequence. The first sub-layer of the music Mel-spectrum feature encoder obtains the music notation feature encoder information related to the music Mel-spectrum feature encoding position through a cross attention mechanism, and then obtains the music spectrum cross feature through a root mean square normalization layer and a residual connection operation. The first sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then obtains the music notation cross feature through the root mean square normalization layer and the residual connection operation. .

[0057] The second sublayer consists of two branches, one of which is a cross self-attention mechanism and a root mean square normalization layer, and the other is a SA Transformer encoder, which processes the cross features of the music graph. Or music notation cross feature Processing to obtain or , the outputs of the two branches are element-wise added. Similarly, the second sublayer of the music Mel-spectrogram feature encoder obtains the music notation feature encoder information related to the music Mel-spectrogram feature encoding position through the cross attention mechanism, and then processes it through the RMS normalization layer to obtain The second sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then processes it through the root mean square normalization layer to obtain .

[0058] In order to capture the characteristics of two different subspaces, the music Mel-spectrum features are As the key vector of attention Sum value vector , the musical notation features Query vector as attention Using query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music Mel spectrum encoder information:

[0059] ;

[0060] in, The query vector The parameter matrix of is the key vector The parameter matrix of is a value vector The parameter matrix of , d is the data dimension;

[0061] Similarly, the musical notation features As the key vector of attention Sum value vector , the music Mel-atrogram feature Query vector as attention , using the query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music notation feature encoder information:

[0062] ;

[0063] in, The query vector The parameter matrix of is the key vector The parameter matrix of is a value vector The parameter matrix of , d is the data dimension.

[0064] As the number of network layers increases, the parameters may start to become too large or too small after multi-layer calculations, which may cause abnormalities in the learning process and the model may converge very slowly. The root mean square normalization introduced in the present invention abandons the centralization operation, and the normalization process only realizes scaling. The scaling coefficient is the root mean square, and when the mean is 0, it is the root mean square normalization. The root mean square only scales and does not change the original distribution of the data, which is conducive to the stability of the output of the activation function (scaling does not change the direction of the vector):

[0065] , , , ;

[0066] in, Indicates Neurons encode feature results, Indicates The weight of the neuron, Indicates the input Encoded features, Express request RMS value, Indicates that the element calculates a nonlinear function, Indicates The output of each neuron is represents the gain parameter used to rescale the normalized total input, express After RMS normalization, the result is express Number.

[0067] Step 5: Use the SA Transformer encoder to capture the pixel-level music graph joint cross-features and music notation joint cross-feature pairwise relationships and channel dependency extractions, and then use the linear layer to obtain the output of the specified dimension by linearly changing the extraction results, which is the role of transforming the dimension; the softmax layer scales the numbers in the last-dimensional vector to the probability value range of 0-1, and satisfies that their sum is 1, and combines the music emotion type to obtain the prediction result.

[0068] The present invention uses a piano music emotion recognition method based on joint cross attention and a piano music emotion recognition method without joint cross attention to conduct a comparative experiment. The results are as follows: Figure 2 As shown, it is shown that the training recognition rate of the piano music emotion recognition method based on joint cross-attention proposed by the present invention is generally higher than that of the piano music emotion recognition method without using joint cross-attention.

[0069] Another embodiment of the present invention provides a system for implementing a piano music emotion recognition method based on joint cross attention, comprising:

[0070] Data acquisition module, used for collecting piano music data;

[0071] A data preprocessing module, used for converting piano music data into music score samples and music Mel-grams;

[0072] A feature extraction module is used to extract features from music notation samples and music mel-atlas to obtain music notation features and music mel-atlas features;

[0073] A feature fusion module is used to perform a joint cross-attention operation on the music notation features and the music Mel-atlas features, and output the music atlas joint cross-features and the music notation joint cross-features;

[0074] The recognition module is used to recognize the joint cross features of music spectra and the joint cross features of music notations, and output prediction results.

[0075] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0076] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A piano music emotion recognition method based on joint cross attention, characterized in that: Here are the steps: Step 1: Piano music data collection; Step 2: converting the piano music data into a music notation file and resampling the music notation file to obtain a resampled music notation sample; and converting the piano music data into a music Mel-atlas; Step 3: Extract features from the sampled music notation samples to obtain music notation features; extract features from the music Mel-atrogram to obtain music Mel-atrogram features; Step 4: Encode the music notation features through the music notation feature encoder, and encode the music mel-spectrum features through the music mel-spectrum feature encoder. The music mel-spectrum feature encoder obtains the music mel-spectrum feature encoder information related to the music notation feature encoding position through the cross-attention mechanism, and outputs the music spectrum joint cross feature. The music notation feature encoder obtains the music notation feature encoder information related to the music Mel-atrogram encoding position through the cross-attention mechanism, and outputs the music notation joint cross-feature ;in, is the music graph cross feature, For the crossover feature of music notation, It is the cross feature of music notation The SAtransformer encoder processes the result. It is the cross feature of music notation The result after cross-attention RMS normalization, is the music graph cross feature The SA transformer encoder processing result is: is the music graph cross feature The result after cross-attention RMS normalization; Step 5: Use the SA Transformer encoder to capture the pixel-level music graph joint cross-features and music notation joint cross-feature pairwise relationships and channel dependency extraction, then use the linear layer to linearly transform the extraction results to obtain the output of the specified dimension, and finally obtain the prediction result through the softmax layer.

2. The piano music emotion recognition method based on joint cross attention according to claim 1 is characterized in that: In step 2, the shortest duration of each note in the music score file is 0.25 seconds, which is defined as a single character block. Every 32 character blocks constitute a new music score sample. Perform the following processing: ； in, Indicates music The resampled music score sample. Indicates Music score files, Indicates the number of music score files. Function refers to the The character block with a length of 32 is taken as the first A music score file, and the insufficient part at the end is discarded.

3. The piano music emotion recognition method based on joint cross attention according to claim 1 is characterized in that: In step 3, a long short-term memory network is used to extract features from the sampled music notation samples.

4. The piano music emotion recognition method based on joint cross attention according to claim 1 is characterized in that: In step 3, the ResNet backbone network is used to extract features from the music mel-level atlas.

5. The piano music emotion recognition method based on joint cross attention according to claim 1 is characterized in that: The music Mel-spectrum feature encoder and the music notation feature encoder have the same structure; both are composed of two sub-layers connected in series, and the first sub-layer includes a cross self-attention mechanism, a root mean square normalization layer, and a residual connection in sequence; the first sub-layer of the music Mel-spectrum feature encoder obtains the music notation feature encoder information related to the music Mel-spectrum feature encoding position through the cross attention mechanism, and then obtains the music spectrum cross feature through the root mean square normalization layer and the residual connection operation. The first sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then obtains the music notation cross feature through the root mean square normalization layer and the residual connection operation. .

6. The piano music emotion recognition method based on joint cross attention according to claim 5 is characterized in that: The second sublayer consists of two branches, one of which is a cross self-attention mechanism and a root mean square normalization layer, and the other is a SA Transformer encoder, which processes the cross features of the music graph. Or music notation cross feature Processing to obtain or , the outputs of the two branches are element-wise added. Similarly, the second sublayer of the music Mel-spectrogram feature encoder obtains the music notation feature encoder information related to the music Mel-spectrogram feature encoding position through the cross attention mechanism, and then processes it through the RMS normalization layer to obtain The second sublayer of the music notation feature encoder obtains the music Mel-spectrum encoder information related to the music notation feature encoding position through the cross attention mechanism, and then processes it through the root mean square normalization layer to obtain .

7. The piano music emotion recognition method based on joint cross attention according to claim 5 is characterized in that: The cross self-attention mechanism is: As the key vector of attention Sum value vector , the musical notation features Query vector as attention ; Using query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music Mel spectrum encoder information; Similarly, the musical notation features As the key vector of attention Sum value vector , the music Mel-atrogram feature Query vector as attention , using the query vector With key vector Calculate the attention weight matrix and compare it with the value vector Multiply them together to get the music score feature encoder information.

8. The piano music emotion recognition method based on joint cross attention according to claim 5 is characterized in that: The RMS normalization layer abandons the centering operation, and the normalization process only realizes scaling. The scaling coefficient is the RMS, and when the mean is 0, it is RMS normalization.

9. A system for implementing the piano music emotion recognition method based on joint cross attention according to any one of claims 1 to 8, characterized in that: include: Data acquisition module, used for collecting piano music data; A data preprocessing module, used for converting piano music data into music score samples and music Mel-grams; A feature extraction module is used to extract features from music notation samples and music mel-atlas to obtain music notation features and music mel-atlas features; A feature fusion module is used to perform a joint cross-attention operation on the music notation features and the music Mel-atlas features, and output the music atlas joint cross-features and the music notation joint cross-features; The recognition module is used to recognize the joint cross features of music spectra and the joint cross features of music notations, and output prediction results.

Images