Music emotion recognition method and system based on parallel local attention and converter

Abstract

The invention belongs to the technical field of audio signal processing, and discloses a music emotion recognition method and system based on parallel local attention and a converter, and the method comprises the steps: inputting an audio feature vector into a parallel local attention network and a converter network, and respectively extracting a local music feature and a global music feature; carrying out weighted fusion to obtain a fused feature vector; a music encoder is used to encode music features; using an emotion encoder to serialize the emotion labels into vectors, and outputting emotion features; using canonical correlation analysis to maximize the correlation between the music features and the emotion features; the similarity between music and emotion distribution is optimized through KL divergence loss; and by calculating the similarity, bidirectional retrieval of music recognition emotion and emotion recognition music is realized. The method can effectively improve the performance of the music emotion recognition task, has a wide application prospect, and particularly has an important application value in the fields of music recommendation, emotion calculation, human-computer interaction and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio signal processing, and in particular relates to a music emotion recognition method and system based on parallel local attention and converter. Background Art

[0002] Music emotion recognition is one of the hot topics in artificial intelligence research in recent years. Most of the existing music emotion recognition methods rely on traditional feature extraction techniques, such as spectral features and time domain features, but these methods cannot fully exploit the fine-grained music features in audio data, resulting in limited accuracy of emotion recognition. With the development of deep learning technology, deep learning-based methods have shown good performance in music emotion recognition. However, most of the existing methods have problems such as insufficient processing of long-term features and difficulty in fusing local music features with global music features. Summary of the invention

[0003] To solve the above problems, the present invention provides a music emotion recognition method and system based on parallel local attention and converter, which can accurately extract music features from audio signals and improve the accuracy of music emotion recognition through efficient feature extraction and emotion analysis technology.

[0004] The present invention is achieved through the following technical solutions: A music emotion recognition method based on parallel local attention and transformer, comprising the following steps: Step S1: extracting an audio feature vector from an audio signal; Step S2: Input the audio feature vector into a parallel local attention network (PLAN) to extract local music features; input the audio feature vector into a transformer network (Transformer) to extract global music features; the parallel local attention network includes two branches, each of which consists of a convolutional layer and a convolutional attention module; Step S3: weighted fusion of local music features and global music features to obtain a fused feature vector; Step S4: using a music encoder to process the fused feature vector and output the music feature; using an emotion encoder to serialize the emotion label into a vector and output the emotion feature; Step S5: using canonical correlation analysis (CCA) to maximize the correlation between music features and emotional features; optimizing the similarity between music and emotional distributions through KL divergence loss; combining canonical correlation analysis (CCA) loss and KL divergence loss to obtain a comprehensive loss function; Step S6: By calculating the similarity, the bidirectional retrieval of music recognition emotion (M2E) and emotion recognition music (E2M) is realized.

[0005] Further optimization is to evenly divide the input audio feature vector according to the time step and input it into two branches of the parallel local attention network (PLAN) at the same time. The input features are convolved in the two branches to obtain the features of the corresponding branches, and then the channel attention and spatial attention are calculated. The features of the first branch After being processed by the convolutional attention module, the convolutional attention module output features of the first branch are: , The spatial attention output feature of the first branch is obtained by weighting. ,in and are two weight parameters; the convolutional attention module output features of the second branch , is the characteristic of the second branch, The second branch outputs the spatial attention feature. ; The two branch output features are weighted and then globally pooled through the pooling layer, and then fully connected through the fully connected layer to finally obtain the local music features. .

[0006] Further preferably, in step S3, the weight parameters of the adaptive learning are used to weight the local music features and the global music features, and the specific operation is as follows: ; Among them, F represents the fused feature vector, and are two weight parameters obtained through network adaptive learning, which are used to control the weights of local music features and global music features respectively. is the local music feature, It is the global music feature.

[0007] Further preferably, a multi-layer perceptron MLP and a bidirectional GRU are selected as the music encoder, and a bidirectional GRU is used as the emotion encoder.

[0008] Further optimization is performed by using canonical correlation analysis to maximize the correlation between music features and emotional features. The process is as follows: The music features Transform into music feature embedding vector using fully connected layer , the emotional features Transform into sentiment feature embedding vector using fully connected layer , and projected to dimension via linear activation The embedding space In; for music and emotion samples, the music feature embedding vector set and sentiment feature embedding vector set is strongly correlated, n is the sample number, and N is the total number of samples; Use the fully connected layer to transform the music features of the nth sample and the sentiment feature of the nth sample Refine into one dimensional music feature vector and a dimensional sentiment feature vector , is the music feature vector dimension, is the dimension of the sentiment feature vector, then we get a dimensional correlation function: ; in, and are respectively from the music feature vector set and sentiment feature vector set A random vector drawn from the estimated probability distribution; and yes and A weight vector of two dimensions, for The transpose of for The transpose of They are respectively , , The calculated cross-covariance matrix and covariance matrix are is a set of music feature-emotion feature vector pairs; Canonical correlation analysis maximizes and The sum of each correlation calculated in one dimension, and batch optimization of weight parameters for all dimensions is done by solving the following constrained optimization problem: ; ; in, Represents the canonical correlation analysis function, tr represents the tracking operation, which is used for summation and correlations in each dimension; is the music feature weight parameter matrix The transpose of is the emotional feature weight parameter matrix, and the optimal music feature weight parameter matrix is ​​obtained and the optimal sentiment feature weight parameter matrix After that, the music feature vector and sentiment feature vector The embeddings based on canonical correlation analysis are calculated as and .

[0009] Further preferably, the process of optimizing the similarity between music and emotion distribution by KL divergence loss is as follows: Using the two branches of the fully connected layer Transformed into mean vector and the covariance matrix , the mean vector and covariance matrix are defined as the embedding space A multivariate Gaussian distribution of additional embedding ), is the mean value of music features, is the standard deviation of the music feature. Similarly, we define The multivariate Gaussian distribution of ), is the mean value of sentiment features, is the standard deviation of sentiment features and calculates the KL divergence of two multivariate Gaussian distributions.

[0010] The present invention also provides a music emotion recognition system based on parallel local attention and converter, comprising: Feature extraction module: used to extract audio feature vectors from audio signals; Music and emotion encoding module: used to input the audio feature vector into the parallel local attention network to extract local music features; input the audio feature vector into the transformer network to extract global music features; weightedly fuse the local music features with the global music features to obtain a fused feature vector; and process the fused feature vector using the music encoder and output the music features; use the emotion encoder to serialize the emotion label into a vector and output the emotion features; Correlation analysis module: Use canonical correlation analysis to maximize the correlation between music features and emotional features; optimize the similarity between music and emotional distribution through KL divergence loss; combine canonical correlation analysis loss and KL divergence loss to obtain a comprehensive loss function; Composite function module: used to calculate similarity and realize bidirectional retrieval of music emotion recognition and emotion recognition music.

[0011] The present invention also provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the above-mentioned music emotion recognition method based on parallel local attention and converter.

[0012] The present invention has the following advantages: 1) The present invention captures both the local details and the global temporal structure of the audio signal through the collaborative design of the parallel local attention network (PLAN) and the converter network, thus overcoming the problem that the traditional single network (CNN or RNN) cannot take into account both local and global features.

[0013] 2) The present invention introduces learnable weight parameters to automatically adjust the contribution ratio of local and global features according to the time-frequency characteristics of the input music, and combines the multi-objective optimization strategy of canonical correlation analysis (CCA) and KL divergence to significantly reduce the distribution differences between music features and emotion labels in the embedding space.

[0014] 3) The present invention adopts a multi-objective loss function to realize the joint optimization of feature extraction, fusion and embedding learning, and supports the bidirectional retrieval of music recognition emotion (M2E) and emotion recognition music (E2M). By freezing the underlying parameters of the pre-trained VGGish16 network and combining it with a lightweight Transformer encoder design, the model parameters are reduced and the accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is an overall flow chart of the method of the present invention.

[0016] Figure 2 This is a system network structure diagram of the present invention.

[0017] Figure 3 This is the mean cosine similarity graph of the DEAM dataset music recognition emotion (M2E) of the present invention.

[0018] Figure 4 This is the mean cosine similarity graph of music recognition emotion (M2E) of the PMEmo dataset of the present invention.

[0019] Figure 5 This is the average cosine similarity graph of the emotion recognition music (E2M) of the DEAM dataset of the present invention.

[0020] Figure 6 This is the average cosine similarity graph of emotion recognition music (E2M) of the PMEmo dataset of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0022] like Figure 1 and Figure 2 As shown, a music emotion recognition method based on parallel local attention and transformer includes the following steps: Step S1: extracting an audio feature vector from an audio signal; In order to obtain meaningful music features from audio signals, we first need to extract the spectral information, time domain features and frequency distribution of the audio signals, and generate an audio feature vector with a time step of T and a feature dimension of D through the VGGish16 network. ; Step S2: Input the audio feature vector into the parallel local attention network (PLAN) to extract local music features; input the audio feature vector into the transformer network (Transformer) to extract global music features; The local music features are extracted by the parallel local attention network (PLAN). The parallel local attention network consists of two branches, each of which is composed of a convolutional layer and a convolutional attention module. It can fully focus on key features and maintain spatial correlation when extracting local information of the audio signal, and finally obtain local music features. The specific steps include: The input audio feature vector is evenly divided according to the time step and input into the two branches of the parallel local attention network (PLAN) at the same time. The input feature of the first branch The features of the first branch are obtained through two one-dimensional 3×1 convolution operations. , similarly, the input features of the second branch The features of the second branch are obtained through the convolution layer .

[0023] In audio signal processing, features at different time steps may have different importances, and music emotions fluctuate. By using channel attention and spatial attention, we can capture the emotional information at different locations in the music information, so that the model can more accurately capture the emotional expression in the music. (i=1,2), first use average pooling and maximum pooling operations to aggregate the spatial information of the features, generate two different spatial context descriptors, and obtain the average pooling feature of the channel dimension and the maximum pooling feature of the channel dimension , t represents time, T represents total time, d represents dimension, and i represents branch number. Then, the two descriptors are propagated to the shared network to generate channel attention Channel ( ). After applying the shared network to each average pooled feature and max pooled feature, the output feature vectors are combined using element-wise summation.

[0024] The shared network consists of a multilayer perceptron (MLP) with one hidden layer. After applying the shared network to each descriptor, the output feature vectors are combined using the element-wise summation method. That is, the channel attention calculation method is: ; in, is the channel attention of the i-th branch, is the average pooling operation, is the maximum pooling operation, represents the sigmoid function, , are the weight parameters of two multi-layer perceptrons, processed by the ReLU activation function, and finally the spatial attention input features are obtained .

[0025] for , through average pooling and maximum pooling operations, the channel information of a feature is aggregated to generate the average pooling feature of the spatial dimension and the maximum pooling feature of the spatial dimension Then they are concatenated and convolved through a standard convolutional layer to generate spatial attention features. The calculation method of spatial attention is: ; in, represents the sigmoid function, represents a convolution operation with a filter size of 4×4; then the spatial attention output feature is obtained .

[0026] Features of the first branch After being processed by the convolutional attention module, the convolutional attention module output features of the first branch are: , The spatial attention output feature of the first branch is obtained by weighting. ,in and are two weight parameters; the convolutional attention module output features of the second branch , is the characteristic of the second branch, The second branch outputs the spatial attention feature. .

[0027] The two branch output features are weighted and then globally pooled through the pooling layer, and then fully connected through the fully connected layer to finally obtain the local music features. .

[0028] Audio signals often have long-term characteristics, including the duration of music, the overall structure of the music, etc. Through the multi-head attention mechanism of the transformer network, we focus on the information at different positions and perform weighted fusion on the outputs of multiple attention heads to obtain a more comprehensive and accurate feature representation. The specific steps include: The audio feature vector first passes through a one-dimensional 3×1 convolutional layer as the input of the transformer network.

[0029] The Transformer network consists of a sequence of 12 attention modules. Each sequence consists of 8 layers of multi-head self-attention (MHSA) and multi-layer perceptron (MLP). The multi-layer perceptron consists of 3 fully connected layers for feature projection. Finally, a classification head implemented with a single-layer MLP is used for classification output. The input feature x is linearly transformed into query Q, key K and value V. The attention head can be expressed as: ; in, is the output of the h-th attention head, , , ; , , are randomly initialized parameter matrices for query Q, key K, and value V, respectively, is the scaling factor, is the activation function. Finally, multi-head self-attention (MHSA) can be described as: ; in, is multi-head self-attention, Concat represents the connection operation, is a randomly initialized parameter matrix; then the connection operation is performed to obtain the global music features .

[0030] Step S3: weighted fusion of local music features and global music features to obtain a fused feature vector.

[0031] The weight parameters of adaptive learning are used to weight the local music features and the global music features. The specific operation is as follows: ; Among them, F represents the fused feature vector, and These are two weight parameters obtained through network adaptive learning, which are used to control the weights of local music features and global music features respectively.

[0032] Step S4: using a music encoder to process the fused feature vector and output the music feature; using an emotion encoder to serialize the emotion label into a vector and output the emotion feature; Multilayer perceptron MLP and bidirectional GRU are selected as music encoders to transform the fused feature vector F into the input feature vector of the multilayer perceptron = , for No. The sub-eigenvector at time, for The first moment sub-eigenvector of The average eigenvector of , multi-layer perceptron Perform multiple nonlinear transformations; calculate separately through bidirectional GRU The time characteristics in the forward and backward directions are express The time characteristics in the forward direction at time t, express The time characteristics in the backward direction at time t. Finally, the recursive aggregation function is obtained: , for The sub-eigenvector at time t; , by connecting exist Temporal characteristics in the direction of moment advancement and exist Time characteristics in the backward direction get Finally, the bidirectional GRU feeds back the high-level features of the bidirectional temporal features to the fully connected layer to generate higher-level music features. .

[0033] The role of the emotion encoder is to serialize the emotion labels corresponding to the music samples into vectors and map them into the embedding space. First, the arousal and effect values ​​corresponding to the music samples are converted into an emotion sequence. = , the length is , for The first sub-emotion sequence of for No. A sub-emotion sequence. Generate more advanced sentiment features through bidirectional GRU .

[0034] Step S5: using canonical correlation analysis (CCA) to maximize the correlation between music features and emotional features; optimizing the similarity between music and emotional distributions through KL divergence loss; combining canonical correlation analysis (CCA) loss and KL divergence loss to obtain a comprehensive loss function; In order to optimize the correlation between music features and emotional features, the present invention adopts the canonical correlation analysis (CCA) method to maximize the correlation between music features and emotional features. Canonical correlation analysis (CCA) optimizes the embedding space by maximizing the correlation between music features and emotional features. The specific steps include: The music features Transformed into music feature embedding vector using fully connected (FC) layer Similarly, the emotional features Transformed into sentiment feature embedding vector using fully connected (FC) layer , and projected to dimension via linear activation The embedding space For music and emotion samples, the music feature embedding vector set and sentiment feature embedding vector set are strongly correlated, n is the sample number, and N is the total number of samples.

[0035] First, the music features of the nth sample are transformed into and the sentiment feature of the nth sample Refine into one dimensional music feature vector and a dimensional sentiment feature vector , R represents the number field, is the music feature vector dimension, is the dimension of the sentiment feature vector, then we can get a dimensional correlation function: ; in, and are respectively from the music feature vector set and sentiment feature vector set A random vector drawn from the estimated probability distribution. and yes and A weight vector of two dimensions, for The transpose of for The transpose of They are respectively , , The calculated cross-covariance matrix and covariance matrix are is a set of music feature-emotion feature vector pairs.

[0036] CCA maximizes and The sum of each correlation calculated in one dimension, The d-th dimension weight vector representing the music feature, is the d-th dimension weight vector of the sentiment feature, and the weight parameters for all dimensions are Batch optimization of can be accomplished by solving the following constrained optimization problem: ; ; in, Represents the canonical correlation analysis function, tr represents the tracking operation, which is used for summation and Correlation in each dimension. is the music feature weight parameter matrix The transpose of is the emotional feature weight parameter matrix, and the optimal music feature weight parameter matrix is ​​obtained and the optimal sentiment feature weight parameter matrix After that, the music feature vector and sentiment feature vector The embeddings based on canonical correlation analysis are calculated as and .

[0037] KL divergence loss minimizes the distribution difference between music and emotion in the embedding space. The specific steps include: Using the two branches of the fully connected layer Transformed into mean vector and the covariance matrix , the mean vector and covariance matrix are defined as the embedding space A multivariate Gaussian distribution of additional embedding ), is the mean value of music features, is the standard deviation of the music feature. Similarly, we define The multivariate Gaussian distribution of ), is the mean value of sentiment features, is the standard deviation of the sentiment feature.

[0038] Projecting music features and their related emotional features into mutually similar multivariate Gaussian distributions )and ), the music features and their unrelated emotional features are projected into different multivariate Gaussian distributions and multivariate Gaussian distribution of uncorrelated sentiment features ,make for , for , for ,So The above conditions can be expressed as triples: ; in: Represents the distance between two distributions; Is a boundary hyperparameter that determines how far the difference between the distances of positive and negative pairs is allowed to be. Similarly, another triple can be obtained: ; is a multivariate Gaussian distribution of non-correlated music features .

[0039] Finally, the ranking loss The combination of two hinge losses is defined as: ; ; KL divergence as dimensionality of The first multivariate Gaussian distribution in and the second multivariate Gaussian distribution The distance between , the calculation method is as follows: ; in, and are the mean and standard deviation of the first multivariate Gaussian distribution, and Expanded to and , for The first element after expansion, for After expansion elements, for The first element after expansion, for After expansion elements, correspondingly, and are the mean and standard deviation of the second multivariate Gaussian distribution, and Finally, KL(G) is defined as the sum of all positive pairs in the multivariate Gaussian distribution G. The sum of . Therefore, the KL divergence between each positive pair is minimized, while the KL divergence between each negative pair is maximized.

[0040] Combining CCA loss and KL divergence loss, we get the comprehensive loss function: ; in is the weight parameter that balances CCA(F) and KL(G).

[0041] Step S6: By calculating the similarity, the bidirectional retrieval of music recognition emotion (M2E) and emotion recognition music (E2M) is realized.

[0042] Evaluate the similarity between emotion and music by calculating cosine similarity: ; in, is the cosine similarity between music and emotion, To query music samples, To test the emotional samples, To analyze the similarity of correlations based on canonical correlation, represents a negative KL divergence, To query the embedding of music samples based on canonical correlation analysis, To test the embedding of sentiment samples based on canonical correlation analysis, To query the embedding of music samples based on multivariate Gaussian distribution, is the embedding of the test sentiment sample based on the multivariate Gaussian distribution. q is the query sample index ), is the number of examples in the test set. j is the test sample index ). By checking and searching the music samples The performance of M2E is evaluated by whether the relevant test sentiment ranks high in the sorted output list. Similarly, E2M can be obtained: ; in, is the cosine similarity between emotion and music, To query sentiment samples, To test the music samples, To query sentiment samples based on canonical correlation analysis, To test the embedding of music samples based on canonical correlation analysis, To query the embedding of sentiment samples based on multivariate Gaussian distribution, Embeddings for test music samples based on multivariate Gaussian distribution.

[0043] Finally, the performance of E2M is evaluated by examining the ranking of music samples related to the query sentiment.

[0044] The effect of the present invention is further illustrated by simulation below: (1) Simulation conditions The proposed model is implemented using python3.8 and tensorflow2.5. The model is trained on a machine equipped with 1 NVIDIA GeForce RTX 3090. Average reciprocal rank, average rank, and cosine similarity are used as performance metrics. The model is trained using the Adam optimizer and is run on DEAM. , and PMEmo The batch size on both datasets is 32, the learning rate is 1e-5, and the ratio of training set to test set is 8:2.

[0045] (2) Simulation content and results The proposed model is compared with the existing advanced methods, and the results are shown in Table 1 and Table 2. It can be seen that on both datasets, whether it is music recognition emotion (M2E) or emotion recognition music (E2M), we have achieved advanced records in average reciprocal ranking and average ranking. In particular, our model outperforms previous methods based on MLP baseline regression, bidirectional GRU baseline regression and compound loss model; as well as single parallel network and single transformer network. It can be seen that the advantage of the present invention is to process the local information and global information of music for fusion, and analyze their correlation and distribution differences, which are ignored by previous methods.

[0046] DEAM Dataset

[0047] Table 1. Comparison of DEAM dataset with previous state-of-the-art methods.

[0048] PMEmo dataset

[0049] Table 2. Comparison of PMEmo dataset with previous state-of-the-art methods.

[0050] MRR and AR are global indicators that only depend on the rank of the music sample or the sentiment associated with the query. It is also necessary to check the relevance of the top-ranked music samples (or sentiments) to the query. To this end, the present invention calculates the average cosine similarity of M2E and E2M, which averages the acoustic cosine similarity between the query music sample and the top 5% emotion-related features retrieved, and the same is true for E2M. A higher average cosine similarity means that the model can identify music samples that express emotions similar to the query emotion, or emotions expressed in music samples that are acoustically similar to the query music sample.

[0051] refer to Figure 3 and Figure 4 ,The median of mean cosine similarity of music recognition emotion (M2E) for DEAM dataset and PMEmo dataset are 0.782 and 0.861 respectively. Figure 3 and Figure 4 The box plots in Figure 1 show the distribution of cosine similarity, showing that at least 75% of the values ​​are above the 25th percentile (0.734 for the DEAM dataset and 0.844 for the PMEmo dataset); Figure 5 and Figure 6 ,The median average cosine similarity of emotion recognition music (E2M) for DEAM dataset and PMEmo dataset are 0.889 and 0.995, respectively. Figure 5 and Figure 6 The box plots in Figure 1 show the distribution of cosine similarity, showing that at least 75% of the values ​​are above the 25th percentile (0.751 for DEAM dataset and 0.981 for PMEmo dataset), which proves to be robust in identifying music samples that are highly correlated with the query sentiment. Even though some music samples are ranked lower, the top 5% of the identified samples still match the query sentiment.

[0052] Another embodiment of the present invention provides a music emotion recognition system based on parallel local attention and transformer, comprising: Feature extraction module: used to extract audio feature vectors from audio signals; Music and emotion encoding module: used to input the audio feature vector into the parallel local attention network to extract local music features; input the audio feature vector into the transformer network to extract global music features; weightedly fuse the local music features with the global music features to obtain a fused feature vector; and process the fused feature vector using the music encoder and output the music features; use the emotion encoder to serialize the emotion label into a vector and output the emotion features; Correlation analysis module: Use canonical correlation analysis to maximize the correlation between music features and emotional features; optimize the similarity between music and emotional distribution through KL divergence loss; combine canonical correlation analysis loss and KL divergence loss to obtain a comprehensive loss function; Composite function module: used to calculate similarity and realize bidirectional retrieval of music emotion recognition and emotion recognition music.

[0053] Yet another embodiment of the present invention provides a non-volatile computer storage medium storing computer executable instructions for executing the music emotion recognition method based on parallel local attention and converter of the above embodiment.

[0054] The present invention improves the model's attention to music features by designing parallel local attention and converters, and designs a composite loss function to analyze the correlation and distribution differences between music and emotion. Finally, the superiority of the model is determined by retrieval and query to check whether music or emotion is higher in the output ranking.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A music emotion recognition method based on parallel local attention and transformer, characterized in that The following steps are involved: Step S1: extracting an audio feature vector from an audio signal; Step S2: inputting the audio feature vector into a parallel local attention network to extract local music features; inputting the audio feature vector into a transformer network to extract global music features; the parallel local attention network includes two branches, each of which is composed of a convolutional layer and a convolutional attention module; Step S3: weighted fusion of local music features and global music features to obtain a fused feature vector; Step S4: using a music encoder to process the fused feature vector and output the music feature; Use the emotion encoder to serialize the emotion label into a vector and output the emotion feature; Step S5: using canonical correlation analysis to maximize the correlation between music features and emotional features; optimizing the similarity between music and emotional distributions through KL divergence loss; combining canonical correlation analysis loss and KL divergence loss to obtain a comprehensive loss function; Step S6: By calculating similarity, a two-way retrieval of music recognition emotion and emotion recognition music is realized.

2. The music emotion recognition method based on parallel local attention and transformer according to claim 1, characterized in that: The input audio feature vector is evenly divided according to the time step and input into the two branches of the parallel local attention network at the same time. The input features are convolved in the two branches to obtain the features of the corresponding branches, and then the channel attention and spatial attention are calculated. The features of the first branch After being processed by the convolutional attention module, the convolutional attention module output features of the first branch are: , The spatial attention output feature of the first branch is obtained by weighting. ,in and are two weight parameters, is the input feature of the first branch; The convolutional attention module output features of the second branch , is the characteristic of the second branch, The second branch outputs the spatial attention feature. , is the input feature of the second branch; The two branch output features are weighted and then globally pooled through the pooling layer, and then fully connected through the fully connected layer to finally obtain the local music features. .

3. The music emotion recognition method based on parallel local attention and transformer according to claim 2, characterized in that: The channel attention calculation method is: ； in, is the channel attention of the i-th branch, is the average pooling operation, is the maximum pooling operation, represents the sigmoid function, , are the weight parameters of two multilayer perceptrons, is the average pooling feature of the channel dimension, is the maximum pooling feature of the channel dimension; Then it is processed by the ReLU activation function, and finally the spatial attention input feature is obtained ; The calculation method of spatial attention is: ； in, represents the sigmoid function, Represents a convolution operation with a filter size of 4×4; is the average pooling feature of the spatial dimension; is the maximum pooling feature of the spatial dimension; Then we get the spatial attention output feature .

4. The music emotion recognition method based on parallel local attention and transformer according to claim 1, characterized in that: In step S3, the weight parameters of the adaptive learning are used to weight the local music features and the global music features. The specific operation is as follows: ； Among them, F represents the fused feature vector, and are two weight parameters obtained through network adaptive learning, which are used to control the weights of local music features and global music features respectively. is the local music feature, It is the global music feature.

5. The music emotion recognition method based on parallel local attention and transformer according to claim 1, characterized in that: Multilayer perceptron and bidirectional GRU are selected as music encoders, and bidirectional GRU is used as emotion encoder.

6. The music emotion recognition method based on parallel local attention and transformer according to claim 1, characterized in that: Canonical correlation analysis is used to maximize the correlation between music features and emotional features. The process is as follows: The music features Transform into music feature embedding vector using fully connected layer , the emotional features Transform into sentiment feature embedding vector using fully connected layer , and projected to dimension via linear activation The embedding space In; for music and emotion samples, the music feature embedding vector set and sentiment feature embedding vector set is strongly correlated, n is the sample number, and N is the total number of samples; Use the fully connected layer to transform the music features of the nth sample and the sentiment feature of the nth sample Refine into one dimensional music feature vector and a dimensional sentiment feature vector , is the music feature vector dimension, is the dimension of the sentiment feature vector, then we get a dimensional correlation function: ； in, and are respectively from the music feature vector set and sentiment feature vector set A random vector drawn from the estimated probability distribution; and yes and A weight vector of two dimensions, for The transpose of for The transpose of They are respectively , , The calculated cross-covariance matrix and covariance matrix are is a set of music feature-emotion feature vector pairs; Canonical correlation analysis maximizes and The sum of each correlation calculated in one dimension, and batch optimization of weight parameters for all dimensions is done by solving the following constrained optimization problem: ； ； in, Represents the canonical correlation analysis function, tr represents the tracking operation, which is used for summation and Correlation in each dimension; is the music feature weight parameter matrix The transpose of is the emotional feature weight parameter matrix, and the optimal music feature weight parameter matrix is ​​obtained and the optimal sentiment feature weight parameter matrix After that, the music feature vector and sentiment feature vector The embeddings based on canonical correlation analysis are calculated as and .

7. The music emotion recognition method based on parallel local attention and transformer according to claim 6, characterized in that: The process of optimizing the similarity between music and emotion distribution through KL divergence loss is as follows: Using the two branches of the fully connected layer Transformed into mean vector and the covariance matrix , the mean vector and covariance matrix are defined as the embedding space A multivariate Gaussian distribution of additional embedding ), is the mean value of music features, is the standard deviation of the music feature. Similarly, we define The multivariate Gaussian distribution of ), is the mean value of sentiment features, is the standard deviation of sentiment features and calculates the KL divergence of two multivariate Gaussian distributions.

8. The music emotion recognition method based on parallel local attention and transformer according to claim 1, characterized in that: In step S6, the similarity between emotion and music is evaluated by calculating the cosine similarity: ； in, is the cosine similarity between music and emotion, To query music samples, To test the emotional samples, To analyze the similarity of correlations based on canonical correlation, represents a negative KL divergence, To query the embedding of music samples based on canonical correlation analysis, To test the embedding of sentiment samples based on canonical correlation analysis, To query the embedding of music samples based on multivariate Gaussian distribution, To test the embedding of emotional samples based on multivariate Gaussian distribution; q is the query sample index, j is the test sample index; by checking and querying music samples The ranking of the relevant test emotions in the sorted output list is used to evaluate the performance of music recognition emotion; ； in, is the cosine similarity between emotion and music, To query sentiment samples, To test the music samples, To query sentiment samples based on canonical correlation analysis, To test the embedding of music samples based on canonical correlation analysis, To query the embedding of sentiment samples based on multivariate Gaussian distribution, Embeddings based on multivariate Gaussian distributions for testing music samples; The performance of emotion recognition music is evaluated by examining the ranking of music samples related to the query emotion.

9. A music emotion recognition system based on parallel local attention and transformer, characterized in that include: Feature extraction module: used to extract audio feature vectors from audio signals; Music and emotion encoding module: used to input the audio feature vector into the parallel local attention network to extract local music features; input the audio feature vector into the converter network to extract global music features; weightedly fuse the local music features with the global music features to obtain a fused feature vector; and process the fused feature vector using the music encoder and output the music features; Use the emotion encoder to serialize the emotion label into a vector and output the emotion feature; Correlation analysis module: Use canonical correlation analysis to maximize the correlation between music features and emotional features; optimize the similarity between music and emotional distribution through KL divergence loss; combine canonical correlation analysis loss and KL divergence loss to obtain a comprehensive loss function; Composite function module: used to calculate similarity and realize bidirectional retrieval of music emotion recognition and emotion recognition music.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions execute the music emotion recognition method based on parallel local attention and converter as described in any one of claims 1-8.

Images