A personalized music dynamic emotion recognition method and device

Abstract

The present invention discloses a personalized dynamic music emotion recognition method and device thereof. The music emotion recognition meta-model obtained by the present invention through meta-task set training can build a personalized emotion recognition model that matches the user's personalized emotion based on the user's emotional needs, and obtain a predicted emotion recognition sequence that meets the user's emotional tendency by performing emotion recognition on the music audio through the personalized emotion recognition model. The present invention cuts and segments the music audio so that the music emotion recognition meta-model can recognize the Valence value and Arousal value of each segment of the music audio, thereby obtaining the Valence curve and Arousal curve corresponding to the music audio, and accurately recognizes the emotion based on the Valence curve and Arousal curve.

Description

Technical Field

[0001] The present invention belongs to the field of emotional computing technology, and in particular relates to a personalized music dynamic emotion recognition method and a device thereof. Background Art

[0002] In the current digital age, music emotion recognition technology has become an important tool for improving human-computer interaction and user experience. Music is not only a form of artistic expression, but also a medium that can profoundly affect human emotions and psychological states. However, current music emotion recognition methods generally lack personalized considerations. This limitation leads to the fact that the identified music emotion labels are often different from the actual feelings of users.

[0003] Traditional music emotion recognition technology mainly relies on the analysis of audio signals, extracting explicit features such as pitch, rhythm, and timbre from music, and then using standard emotion models (such as the Valence-Arousal model) to classify emotions. This method can effectively identify the emotional tone of music to a certain extent, but it is difficult to consider and reflect the user's personal preferences and emotional resonance. In practical applications, users may have different emotional reactions to the same piece of music, depending on their individual experience, cultural background, emotional memory and other factors, and existing methods are difficult to effectively adjust for these personalized factors.

[0004] The invention patent with the publication number CN118013036A discloses a method, device, medium and product for predicting text-image sentiment based on Transformer, which relates to the field of information technology. The method includes: obtaining text data of a target object and image data corresponding to the text data; preprocessing the text data and image data to obtain processed text data and text data of optical character recognition; extracting features of the processed text data, text data of optical character recognition and image data to obtain text features and image features; and determining the sentiment category of the target object using a Transformer-based text-image sentiment analysis model based on text features and image features. The invention patent application comprehensively measures factors such as information between text and images, reasonably utilizes information between modalities, and mines multimodal sentiment information of social media platforms as much as possible, thereby improving the accuracy of sentiment analysis on social media.

[0005] The invention patent with publication number CN115062115A discloses a method for predicting response emotion categories in dialogue strategies. The method includes the following steps: the knowledge-enhanced dialogue context encoding module encodes the dialogue history information V1:M-1 of the first M-1 rounds into a context vector representation X1:M-1 through a hierarchical Transformer network, and integrates the dialogue features in the dialogue; the potential emotion response learning module uses a variational network to learn the mapping from the context vector representation X1:M-1 to the emotional response in the corpus, thereby modeling the potential distribution z; the interactive emotion prediction module uses the context vector representation X1:M-1 and the potential distribution z output by the dialogue context encoding module as input to predict the response emotion E′Y. The method proposed in this invention patent application realizes the prediction of response emotion categories, gives the dialogue agent the ability to adaptively express diverse emotions, and alleviates the problem of rationality of inference caused by the lack of target text.

[0006] There are already some applications on the market that attempt to enhance personalized experience (such as Spotify, Apple Music, etc.), which recommend music by analyzing the user's playback history or interactive behavior. However, these tools and the emotion recognition disclosed in the above patent application are still limited to the analysis of general audio features, and cannot truly realize the recognition and adjustment from the user's personalized emotional dimension. This leads to the lack of fit between the music recommended by the system and the user's actual emotional experience, affecting the user's usage satisfaction and music exploration experience.

[0007] Therefore, it is particularly important to develop a dynamic music emotion recognition technology that can recognize and adapt to users' personalized emotional responses. Summary of the invention

[0008] The present invention provides a personalized music dynamic emotion recognition method, which can realize emotion recognition of music audio based on personal emotional tendencies.

[0009] A specific embodiment of the present invention provides a personalized music dynamic emotion recognition method, comprising:

[0010] Obtain multiple music audios and corresponding emotion label sequences, divide the multiple music audios into a training sample set and a verification sample set, and randomly extract training samples and verification samples to construct a meta-task set;

[0011] Constructing a training model, the training model includes a time domain feature extraction module, a frequency domain feature extraction module, a Transformer and a sentiment prediction module, performing time domain feature extraction on multiple audio clips obtained from training samples in the cutting meta-task through the time domain feature extraction module to obtain a time domain feature sequence, performing frequency domain feature extraction on a Mel-language spectrogram sequence obtained by converting multiple audio clips through the frequency domain feature extraction module to obtain a frequency domain feature sequence, performing feature fusion on the time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence to obtain a fused feature sequence, performing feature extraction on each fused feature of the fused feature sequence through the Transformer to obtain a sentiment feature sequence, and performing full connection on each sentiment feature in the sentiment feature sequence through the sentiment prediction module to obtain a predicted sentiment recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values;

[0012] Selecting meta-tasks from the meta-task set in order to train the training model, wherein the training model is first trained with training samples in the meta-task, emotion recognition is performed on verification samples in the meta-task based on the trained training model to obtain a predicted emotion recognition sequence, and the training model is updated based on a loss function constructed based on the predicted emotion recognition sequence of the verification sample and the corresponding emotion label sequence to obtain a music emotion recognition meta-model;

[0013] When applied, the music emotion recognition meta-model is trained again based on the music audio with emotion label sequence input by the user to obtain a personalized emotion recognition model with emotion tendency, and the personalized emotion recognition model is used to perform emotion recognition on other music audio input by the user to obtain the corresponding predicted emotion recognition sequence.

[0014] Preferably, the time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence are subjected to feature fusion to obtain a fused feature sequence, including:

[0015] The time domain features of the time domain feature sequence and the frequency domain features of the frequency domain feature sequence are added accordingly, and each added feature is activated by the Sigmoid activation function to obtain a fused feature sequence.

[0016] Preferably, the training sample is cut based on a set interval time to obtain a plurality of audio segments.

[0017] Preferably, a time domain feature extraction module is used to extract time domain features from a plurality of audio segments obtained by cutting the training samples to obtain a time domain feature sequence, and the time domain feature extraction module is a pre-trained Imagebind model.

[0018] Preferably, the mel-spectrogram sequence obtained by converting multiple audio clips includes: converting each audio clip into a corresponding mel-spectrogram, and constructing the mel-spectrogram sequence from the multiple mel-spectrograms;

[0019] Among them, each audio clip is converted into a corresponding Mel spectrogram, including:

[0020] Set the sampling rate, sample the audio clip, perform high-pass filtering on the sampled audio clip, split the filtered audio clip into frames of equal length, and truncate each frame signal by applying a window function;

[0021] Perform Fourier transform on each truncated frame signal to obtain the spectrum of each frame, and calculate the power spectrum of each frame spectrum;

[0022] The spectrum is filtered by a Mel filter bank and the linear frequency axis is converted to a Mel scale frequency axis to obtain a Mel spectrogram sequence.

[0023] Preferably, the Mel-speech spectrogram sequence converted from multiple audio clips is subjected to frequency domain feature extraction by a frequency domain feature extraction module to obtain a frequency domain feature sequence, and the frequency domain feature extraction module includes a three-layer convolutional neural network.

[0024] Preferably, the emotion prediction module fully connects each emotion feature in the emotion feature sequence to obtain a predicted emotion recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values, including:

[0025] The emotion prediction module is a fully connected layer, through which a corresponding Valence value and Arousal value pair is obtained based on each emotion feature, and multiple pairs of Valence values ​​and Arousal values ​​are used to construct a predicted emotion recognition sequence.

[0026] A specific embodiment of the present invention also provides a personalized music dynamic emotion recognition device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the personalized music dynamic emotion recognition method.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The music emotion recognition meta-model obtained by the present invention through meta-task set training can build a personalized emotion recognition model that matches the user's personalized emotions based on the user's emotional needs, and obtain a predicted emotion recognition sequence that conforms to the user's emotional tendencies by performing emotion recognition on music audio through the personalized emotion recognition model.

[0029] The present invention cuts the music audio into segments so that the music emotion recognition meta-model can identify the Valence value and Arousal value of each music audio segment, thereby obtaining the Valence curve and Arousal curve corresponding to the music audio, and accurately identifying the emotion based on the Valence curve and Arousal curve. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart of a personalized music dynamic emotion recognition method provided by a specific embodiment of the present invention;

[0031] Figure 2 A flowchart of a personalized music dynamic emotion recognition method provided by a specific embodiment of the present invention;

[0032] Figure 3 A music emotion curve diagram provided for a specific embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to achieve personalized emotion recognition for music audio, a specific embodiment of the present invention constructs a meta-task set to train a training model with different emotional responses to obtain a music emotion recognition meta-model. The music emotion recognition meta-model can obtain a personalized music emotion recognition model that matches the user's emotional response based on an emotion label sequence provided by the user. Based on the personalized music emotion recognition model, personalized emotion recognition for music audio can be achieved.

[0034] The specific steps of the personalized music dynamic emotion recognition method provided by the specific embodiment of the present invention are as follows: Figure 1 and Figure 2 As shown, including:

[0035] S1. Construct a meta-task set based on obtaining multiple music audios and corresponding emotion label sequences: obtain multiple music audios and corresponding emotion label sequences, divide the multiple music audios into a training sample set and a verification sample set, randomly extract a certain number of training samples from the training sample set, randomly extract a certain number of verification samples from the verification sample set, construct a meta-task with the extracted training samples and verification samples, construct a meta-task set based on multiple meta-tasks, and the specific embodiment of the present invention trains the model through the constructed meta-task set so that the model can recognize multiple personality emotions.

[0036] Specifically, the multiple music audios obtained in the specific embodiment of the present invention are from the DEAM data set. Before allocating the training sample set and the verification sample set, the multiple music audios are preprocessed, including trimming the obtained music audios to a set length.

[0037] S2. Constructing a training model: The training model provided by the specific embodiment of the present invention includes a time domain feature extraction module, a frequency domain feature extraction module, a Transformer and a sentiment prediction module, wherein the time domain feature extraction module extracts time domain features from multiple audio segments obtained by cutting the training samples of the meta-task to obtain a time domain feature sequence, the frequency domain feature extraction module extracts frequency domain features from a Mel-language spectrogram sequence obtained by converting multiple audio segments to obtain a frequency domain feature sequence, the time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence are feature fused to obtain a fused feature sequence, and the Transformer extracts features from each fused feature of the fused feature sequence to obtain a sentiment prediction module. Feature sequence, through the emotion prediction module, each emotion feature in the emotion feature sequence is fully connected to obtain a predicted emotion recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values. The specific embodiment of the present invention fuses the frequency domain features and time domain features of each fragment of music audio to retain as many emotion-related features of each fragment as possible, which is beneficial to the subsequent Transformer to extract the emotion-related features, so that the Valence value and Arousal value of each fragment can be more accurately identified through the emotion prediction module. The specific embodiment of the present invention can more accurately identify the emotional changes of music audio based on the Valence value and Arousal value of each fragment.

[0038] In a specific embodiment, the training sample is cut into multiple audio segments based on a set interval time. Specifically, the set time is 0.5s.

[0039] In a specific embodiment, the mel-spectrogram sequence obtained by converting multiple audio clips includes: converting each audio clip into a corresponding mel-spectrogram, and constructing the mel-spectrogram sequence from the multiple mel-spectrograms;

[0040] Among them, each audio clip is converted into a corresponding Mel spectrogram, including:

[0041] (1) Adjusting the sampling rate: Setting the sampling rate to sample the audio clip. In one embodiment, the sampling frequency is 44100 Hz.

[0042] (2) Pre-emphasis: Apply a high-pass filter to the sampled audio clip to increase the energy of the high-frequency part.

[0043] (3) Framing: The filtered audio segment is divided into frames of equal length. In one embodiment, each frame is 25 milliseconds.

[0044] (4) Windowing: A window function is applied to each frame signal. In one embodiment, the window function is a Hamming window to reduce boundary effects.

[0045] (5) Fourier transform: Perform Fourier transform on each truncated frame of signal to obtain the spectrum of each frame.

[0046] (6) Obtain power spectrum: Calculate the power spectrum of each frame spectrum.

[0047] (7) Mel filter bank: The spectrum is filtered by the Mel filter bank, and the linear frequency axis is converted to a Mel-scale frequency axis to obtain a Mel-scale spectrogram sequence.

[0048] In a specific embodiment, a frequency domain feature sequence is obtained by performing frequency domain feature extraction on a Mel-language spectrogram sequence converted from a plurality of audio segments through a frequency domain feature extraction module, and the frequency domain feature extraction module includes a three-layer convolutional neural network.

[0049] In a specific embodiment, a time domain feature extraction module is used to extract time domain features from a plurality of audio segments obtained by cutting training samples to obtain a time domain feature sequence, and the time domain feature extraction module is a pre-trained Imagebind model.

[0050] In a specific embodiment, the specific embodiment of the present invention performs feature fusion on the time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence to obtain a fused feature sequence, including:

[0051] The time domain features of the time domain feature sequence and the frequency domain features of the frequency domain feature sequence are added accordingly, and each added feature is activated by the Sigmoid activation function to obtain a fused feature sequence.

[0052] In a specific embodiment, the specific embodiment of the present invention performs full connection on each emotion feature in the emotion feature sequence through the emotion prediction module to obtain a predicted emotion recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values, including:

[0053] The emotion prediction module is a fully connected layer, through which a corresponding Valence value and Arousal value pair is obtained based on each emotion feature, and multiple pairs of Valence values ​​and Arousal values ​​are used to construct a predicted emotion recognition sequence.

[0054] S3. Train the training model based on the meta-task set to obtain a music emotion recognition meta-model: select meta-tasks from the meta-task set in order to train the training model, wherein the training model is first trained with training samples in the meta-task, and the verification samples in the meta-task are subjected to emotion recognition based on the trained training model to obtain a predicted emotion recognition sequence, and the training model is updated based on a loss function constructed based on the predicted emotion recognition sequence of the verification sample and the corresponding emotion label sequence to obtain a music emotion recognition meta-model. The music emotion recognition meta-model trained in the specific embodiment of the present invention can construct a personalized emotion recognition model that can recognize user emotions based on music audio with emotion label sequences provided by the user.

[0055] The loss function sequence is calculated based on the emotion label sequence and the predicted emotion recognition sequence of the training sample, and then the average is taken to train the training model through the training sample.

[0056] S4. Based on the music audio with emotion label sequence input by the user, the music emotion recognition meta-model is trained again to obtain a personalized emotion recognition model with emotion tendency, and the personalized emotion recognition model is used to perform emotion recognition on other music audio input by the user to obtain the corresponding predicted emotion recognition sequence.

[0057] In a specific embodiment, the personalized music dynamic emotion recognition method provided by this embodiment includes:

[0058] (1) Dataset preprocessing: The DEAM dataset is obtained and preprocessed, including cutting all music audio into 30s in length, randomly shuffling it to select 80% for training (training sample set) and 20% for testing (validation sample set).

[0059] (2) Meta-task division: Randomly extract 1 music audio as a training sample and 5 music audio as a validation sample set. Extract the training sample and validation sample set as a task. Repeat this extraction process to use all the obtained music audio to construct a meta-task set containing multiple meta-tasks.

[0060] (3) Model training: Extract meta-tasks from the meta-task set obtained in step (2) in order, first save the current training model as θ, the structure of the training model is the same as step S2, then use the training samples in the meta-task to train the training model to obtain a new model θ', then use the new model θ' to predict the verification sample set and calculate the loss, and then use the loss to optimize the model θ. Repeat the above steps until the predetermined number of training times is reached to obtain the music emotion recognition meta-model. The specific embodiment of the present invention is different from the conventional method (using the loss value to θ' to optimize θ'). The specific embodiment of the present invention uses the loss value to optimize θ. In short, the optimization means that the loss of the model θ after training with the training sample is as small as possible when predicting the verification sample, that is, for any new task, after training with a training sample, the accuracy of predicting the verification sample of the new task is as high as possible. In essence, it makes the model more adaptable to new tasks.

[0061] (4) Model saving: After completing the model training, the parameters of the music emotion recognition meta-model will be saved in a file for subsequent use.

[0062] (5) Model personalization: When a user needs to use the music emotion recognition meta-model to predict personalized emotions, the user will first input a sequence of emotion values ​​(based on the Valence-Arousal model) annotated by the user for a piece of music audio. The sample with the annotated emotion value sequence will be used to train the music emotion recognition meta-model obtained in step 4 to obtain a personalized music emotion recognition model.

[0063] (6) Personalized prediction: Based on the personalized music emotion recognition model in step 5, the music audio to be predicted is input, and the Valence value and Arousal value are output as the emotion prediction result of the music. The prediction result is as follows: Figure 3 shown.

[0064] In order to realize the recognition of personalized music emotions of different users, the specific embodiment of the present invention uses the MAML (Model-Agnostic Meta-Learning) meta-learning training strategy for training. When predicting personalized music emotions, it is only necessary to input the emotional value of a piece of music marked by the user (based on the Valence-Arousal model) to predict any music emotion for the user's personality. Specifically, the method adopts a model structure based on Transformer: after inputting the music audio, the Mel spectrogram of the audio is calculated, and then the Mel spectrogram is input into the frequency domain feature extractor, the audio is input into the time domain feature extractor, and the time and frequency domain features are fused and input into the Transformer to further extract the sequence emotion features, and finally the sequence emotion features are input into the emotion predictor to obtain the Valence (alertness) value and Arousal (arousal) value of the Valence-Arousal model, that is, the emotion label.

[0065] A specific embodiment of the present invention also provides a personalized music dynamic emotion recognition device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the personalized music dynamic emotion recognition method.

Claims

1. A personalized music dynamic emotion recognition method, characterized in that: include: Obtain multiple music audios and corresponding emotion label sequences, divide the multiple music audios into a training sample set and a verification sample set, and randomly extract training samples and verification samples to construct a meta-task set; Constructing a training model, the training model includes a time domain feature extraction module, a frequency domain feature extraction module, a Transformer and a sentiment prediction module, performing time domain feature extraction on multiple audio clips obtained from training samples in the cutting meta-task through the time domain feature extraction module to obtain a time domain feature sequence, performing frequency domain feature extraction on a Mel-language spectrogram sequence obtained by converting multiple audio clips through the frequency domain feature extraction module to obtain a frequency domain feature sequence, performing feature fusion on the time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence to obtain a fused feature sequence, performing feature extraction on each fused feature of the fused feature sequence through the Transformer to obtain a sentiment feature sequence, and performing full connection on each sentiment feature in the sentiment feature sequence through the sentiment prediction module to obtain a predicted sentiment recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values; Selecting meta-tasks from the meta-task set in order to train the training model, wherein the training model is first trained with training samples in the meta-task, emotion recognition is performed on verification samples in the meta-task based on the trained training model to obtain a predicted emotion recognition sequence, and the training model is updated based on a loss function constructed based on the predicted emotion recognition sequence of the verification sample and the corresponding emotion label sequence to obtain a music emotion recognition meta-model; When applied, the music emotion recognition meta-model is trained again based on the music audio with emotion label sequence input by the user to obtain a personalized emotion recognition model with emotion tendency, and the personalized emotion recognition model is used to perform emotion recognition on other music audio input by the user to obtain the corresponding predicted emotion recognition sequence.

2. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: The time domain features of the time domain feature sequence and the frequency domain features corresponding to the frequency domain feature sequence are fused to obtain a fused feature sequence, including: The time domain features of the time domain feature sequence and the frequency domain features of the frequency domain feature sequence are added accordingly, and each added feature is activated by the Sigmoid activation function to obtain a fused feature sequence.

3. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: The training samples are cut into multiple audio segments based on the set interval time.

4. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: The time domain feature extraction module is used to extract time domain features from multiple audio clips obtained by cutting the training samples to obtain a time domain feature sequence. The time domain feature extraction module is a pre-trained Imagebind model.

5. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: The mel-spectrogram sequence obtained by converting multiple audio clips includes: converting each audio clip into a corresponding mel-spectrogram, and constructing the mel-spectrogram sequence from the multiple mel-spectrograms; Among them, each audio clip is converted into a corresponding Mel spectrogram, including: Set the sampling rate, sample the audio clip, perform high-pass filtering on the sampled audio clip, split the filtered audio clip into frames of equal length, and truncate each frame signal by applying a window function; Perform Fourier transform on each truncated frame signal to obtain the spectrum of each frame, and calculate the power spectrum of each frame spectrum; The spectrum is filtered by a Mel filter bank and the linear frequency axis is converted to a Mel scale frequency axis to obtain a Mel spectrogram sequence.

6. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: The Mel-language spectrogram sequence converted from multiple audio clips is subjected to frequency domain feature extraction by a frequency domain feature extraction module to obtain a frequency domain feature sequence, wherein the frequency domain feature extraction module includes a three-layer convolutional neural network.

7. The personalized music dynamic emotion recognition method according to claim 1 is characterized in that: Through the emotion prediction module, each emotion feature in the emotion feature sequence is fully connected to obtain a predicted emotion recognition sequence constructed by multiple pairs of Valence values ​​and Arousal values, including: The emotion prediction module is a fully connected layer, through which a corresponding Valence value and Arousal value pair is obtained based on each emotion feature, and multiple pairs of Valence values ​​and Arousal values ​​are used to construct a predicted emotion recognition sequence.

8. A personalized music dynamic emotion recognition device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the personalized music dynamic emotion recognition method described in any one of claims 1-7.

Images