A smart home device music recommendation method based on continuous emotions of a user

Abstract

The application discloses a kind of music recommendation method of intelligent household equipment based on user continuous emotion.First, the personalized music recommendation list is generated according to user basic information and historical interaction information, then the three-dimensional PAD emotional feature matrix of music data and user voice data is extracted using deep full convolution network FCN.The similarity score between different music in the recommendation list is calculated, and the emotional matching degree between user current voice data and music is calculated.The music with the highest emotional matching degree is selected for playing, and the music in the music recommendation list is sorted and played in order from high to low according to the similarity with the currently played music.In the process of playing, the emotional state of the user is detected in real time, and the play list is adjusted according to the current emotional state of the user.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and relates to emotion mining and connection, specifically to a method for recommending music to smart home devices based on users' continuous emotions. Background Technology

[0002] Metaverse technology is a fusion of the virtual and real worlds, providing users with an immersive experience and interaction. Smart home is one of the main application scenarios of metaverse, often referred to as a bridge connecting the real and virtual worlds. With the gradual popularization of smart home devices, intelligent human-computer interaction is becoming increasingly important. Smart home devices can receive various information from users, analyze it, and then provide users with suitable lighting, temperature, background music, and other home environment settings.

[0003] In smart home environments, music is not only a simple form of entertainment but also a way to express emotions. Appropriate background music can improve users' quality of life and increase their satisfaction with smart home devices. Providing personalized background music recommendations and playback based on users' emotional states can create a more pleasant and adaptive smart home environment, thereby enhancing the user's home experience. This is highly valuable for the development of virtual assistants, smart speakers, and home service robots.

[0004] Currently, music recommendation mainly falls into two categories: content-based recommendation and collaborative filtering. Content-based recommendation analyzes song attributes and user history to recommend similar music; while collaborative filtering recommends music with similar tastes based on the similarity of user behavior to other users. However, these methods perform poorly in scenarios that match users' real-time needs because they lack real-time perception of users' current emotional state. Therefore, emotion recognition has become an indispensable part of music recommendation in smart home environments. By modeling the emotions in music and understanding the user's current emotional state, it is possible to select music more accurately, thereby providing users with more personalized and emotional services. For example, when a user is in a good mood, the system can recommend upbeat and cheerful music, while when the user is in a bad mood, it can choose more soothing and slower-paced tracks.

[0005] Current mainstream emotion recognition methods primarily focus on exploring discrete emotion classification, categorizing speech or music into specific emotion types such as happiness, sadness, and anger. However, discrete classification has limited ability to express emotions effectively. In real life, emotions are often continuously changing and difficult to accurately reflect with simple discrete labels. In recent years, with increased awareness of the dynamic nature of emotions, researchers have gradually shifted towards continuous emotion models. Among them, the Valence-Arousal (VA) model has become a research hotspot. It views emotion as a continuous change across two dimensions: Valence and Arousal. This continuous representation allows the model to capture and differentiate emotions more precisely, making speech and music emotion recognition closer to a realistic user experience. However, representing emotion only from two dimensions lacks comprehensiveness, especially in the field of music recommendation. Relying solely on the similarity of Valence and Arousal for music recommendations is biased. With the widespread adoption of various smart home devices, music recommendation requires a more refined approach to emotion perception. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a smart home device music recommendation method based on continuous user emotions. By enabling smart home devices to interact with users online, a corresponding feature matrix is ​​generated based on historical interaction logs and real-time voice data to provide accurate personalized music recommendations.

[0007] A method for recommending music to smart home devices based on users' continuous emotions includes the following steps:

[0008] Step 1: Generate a user feature matrix based on the user's basic information and historical interaction logs, including the user's voiceprint features, music preference types, user basic information, and historical playback records.

[0009] Step 2: Collect user speech, extract Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC) from the speech, concatenate them in order, and input them into a pre-trained voiceprint recognition model to identify the user ID. Select the feature column corresponding to the user ID from the user feature matrix established in Step 1, and use a collaborative filtering algorithm to process the selected feature column to generate a personalized music recommendation list.

[0010] Step 3: Use a deep fully convolutional network (FCN) to perform sentiment recognition on the music data in the music recommendation list obtained in Step 2, and output a three-dimensional PAD sentiment feature matrix (P m A m D m ), where P m A m D m These represent the emotional pleasure, activation, and dominance of the input music data, respectively.

[0011] Step 4: Calculate the similarity between different pieces of music based on emotional features, and construct a music similarity matrix. In the similarity matrix, each element represents the similarity score between two input music data points. The higher the similarity score, the higher the similarity between the corresponding music data.

[0012] Step 5: Use a deep fully convolutional network (FCN) to perform sentiment analysis on the user's real-time speech, and output a three-dimensional PAD sentiment feature matrix (P u A u D u ), where P u A u D u These represent the user's current level of enjoyment, engagement, and advantage in speaking, respectively.

[0013] Step 6: Calculate the user-music emotional matching degree matrix based on the emotional characteristics of users and music, and select the music with the highest matching value.

[0014] s6.1 Calculate the emotional similarity between user voice and music data in terms of pleasure and activation:

[0015]

[0016] Where, U=(P u A u M = (P) M A M ).

[0017] s6.2 Calculate the emotional complementarity between user voice and music data in terms of dominance:

[0018] Complementarity (D) u D M )=|D u -D M |

[0019] s6.3 Set weighting factors ω1 and ω2 to calculate the matching degree between the user's current voice and the music data:

[0020] Matching degree (U,M) = ω1 Similarity (U,M) + ω2 Complementarity (D) u D M )

[0021] s6.4 Select the music with the highest matching degree with the user's current voice from the music recommendation list and play it. Based on the music similarity matrix obtained in step four, sort and play the music in the music recommendation list in descending order of similarity with the currently playing music.

[0022] Preferably, the weighting factor ω1 > ω2, that is, the weight of emotional similarity is greater than that of emotional complementarity.

[0023] As a preferred approach, during music playback, the user's voice data is collected simultaneously, and sentiment analysis is performed according to step five. When the user's emotional state changes, the matching degree of the music data is recalculated according to step six, and the music recommendation list and playback order are adjusted in a timely manner to ensure that the user can obtain music that matches their current emotional state.

[0024] The present invention has the following beneficial effects:

[0025] 1. By generating a user feature matrix based on basic user information and historical interaction logs, and combining it with the user's voiceprint features and music preferences, a collaborative filtering algorithm is used to generate personalized music recommendation lists, thereby more accurately meeting users' individual music needs. Furthermore, online interaction through smart home devices makes it easier for users to personalize music recommendations.

[0026] 2. By using efficient acoustic feature acquisition methods such as horizontal splicing of Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC), combined with a pre-trained voiceprint recognition model, more accurate and faster recognition of user speech can be achieved.

[0027] 3. A deep fully convolutional network (FCN) is introduced to perform emotion recognition on music data and real-time user speech in the music recommendation list, generating a PAD emotion feature matrix. Mel spectrum data, more suitable for speech analysis, is extracted from user speech as input, while CQT spectrum data, more suitable for music analysis, is extracted from song music as input, achieving optimal analysis results for both audio signals. Simultaneously, the network has fewer layers and parameters, resulting in faster computation. By calculating the emotional similarity between music data and user speech in terms of pleasure and activation, and the emotional complementarity in terms of dominance, the interactive influence of music and user emotions is considered. Furthermore, a weighting factor is set to adjust the weights of similarity and complementarity, establishing a user-music emotion matching matrix to achieve more intelligent and higher-matching music recommendations.

[0028] 4. During music playback, user voice data is collected synchronously. Through real-time sentiment analysis, the music recommendation list and playback order are adjusted in a timely manner to ensure that users receive music that matches their current emotional state when their emotional state changes, thereby improving the real-time nature of music recommendations and user experience. Attached Figure Description

[0029] Figure 1 A flowchart illustrating a method for recommending music to smart home devices based on users' continuous emotions.

[0030] Figure 2This is the architecture of a deep fully convolutional network used for music emotion recognition in the embodiment.

[0031] Figure 3 This is a schematic diagram of the PAD continuous emotion model. Detailed Implementation

[0032] The present invention will be further explained below with reference to the accompanying drawings;

[0033] like Figure 1 The method for recommending music to smart home devices based on continuous user emotions, as shown, mainly includes the following steps:

[0034] Step 1: Generate a user feature matrix based on basic user information and historical interaction logs.

[0035] s1.1 User interaction logs can reveal user preferences and their direct interaction characteristics with music. They provide a direct expression of the user's connection with music, such as the music genres the user listened to most frequently in the past 7 days and the music genres with the lowest completion rates in the past 7 days. Historical music playback records, including song titles and playback times, are obtained from the user's historical interaction logs. Music preference types are analyzed based on user tags and click records. Basic user information, including gender and age, is also collected.

[0036] s1.2 Integrate the information of different users collected in s1.1 into a user feature matrix. Each row of the feature matrix corresponds to a user's feature, and each column describes a feature of different users, including voiceprint features, music preference type, user basic information, and historical playback records.

[0037] Step 2: Collect user voice data, determine the current user ID based on voiceprint recognition, select the feature column corresponding to the ID from the user feature matrix established in Step 1, and generate a personalized music recommendation list.

[0038] s2.1. Based on the voice signal acquisition device, the user's voice is acquired, the Mel-frequency cepstral coefficients (MFCC) and linear predictive coding (LPC) are extracted and concatenated in sequence, and then input into the pre-trained voiceprint recognition model to identify the user's ID.

[0039] s2.2 Select the feature columns corresponding to the user IDs identified in s2.1 from the user feature matrix to obtain various user feature attributes. Use a collaborative filtering algorithm to process the selected feature columns and generate a personalized music recommendation list. In addition to music with high historical play rates from the user's interaction logs, the personalized music recommendation list also includes new songs that have never been played but have a high similarity to previously played music. Recommending new songs can expand the user's music experience.

[0040] Step 3: Perform sentiment recognition on the music data in the music recommendation list obtained in Step 2. First, segment the audio file of each song and extract an 84x128 CQT spectrogram using the librosa library. Then, input the spectrogram into a pre-trained deep fully convolutional network (FCN) for feature extraction. Use the softmax activation function to output a three-dimensional PAD sentiment feature matrix (P... m A m D m ), where P m A m D m These represent the emotional pleasure, activation, and dominance of the input music data, respectively.

[0041] The model structure of a deep fully convolutional network (FCN) is as follows: Figure 2 As shown, the 3D PAD emotion model includes four cascaded 3x3 convolutional layers and a max-pooling layer. Figure 3 As shown.

[0042] Step 4: Based on sentiment features, calculate the cosine similarity (A, B) between different songs in the music recommendation list:

[0043]

[0044] Where A = (P1, A1, D1) and B = (P2, A2, D2), ||A|| and ||B|| are the moduli of the two music tracks in the three-dimensional emotional space, respectively. The cosine similarity value ranges from -1 to 1, with values ​​closer to 1 indicating greater similarity between the two music tracks. An emotional similarity matrix of the music tracks in the music recommendation list is constructed.

[0045] Step 5: Perform sentiment analysis on the user's real-time voice and output a three-dimensional PAD sentiment feature matrix (P u A u D u )

[0046] s5.1 When a user launches the application or interacts with the system, the guided interaction module is activated. This module can be triggered by the user launching the application, using voice commands, gestures, or other inputs. It captures the user's real-time voice signal, including the user's voiceprint, speech, dialogue, and audio commands, through a microphone or other audio input device. The captured voice signal is then transmitted to the voice sentiment analysis module for subsequent sentiment analysis.

[0047] s5.2. Perform sentiment analysis on the captured speech signal using the speech sentiment analysis module. First, preprocessing is performed, including noise removal, audio normalization, segmentation, and feature extraction. The preprocessed speech signal is then converted into a Mel spectrogram. Feature extraction is performed using the same pre-trained deep fully convolutional network (FCN) as in step three, outputting a three-dimensional PAD sentiment feature matrix (P...). u A u D u ).

[0048] Step Six: Calculate the emotional similarity between the user's voice and music data in terms of pleasure and activation, and the emotional complementarity in terms of dominance. Then, set the weighting factors ω1 = 0.7 and ω2 = 0.3 to calculate the matching degree between the user's current voice and the music data.

[0049] Matching degree (U,M) = ω1 Similarity (U,M) + ω2 Complementarity (D) u D M )

[0050] Based on the emotional characteristics of users and music, a user-music emotional matching matrix is ​​established to select the music with the highest matching value.

[0051] Step 7: Select the music with the highest matching degree with the user's current voice from the music recommendation list and play it. Based on the music similarity matrix obtained in Step 4, sort and play the music in the music recommendation list in descending order of similarity with the currently playing music to ensure that the user can obtain music that matches their current emotional state.

[0052] During music playback, the user's voice data is collected simultaneously, and sentiment analysis is performed according to step five. When the user's emotional state changes, the matching degree of the music data is recalculated according to step six, and the music recommendation list and playback order are adjusted in a timely manner to ensure that the user can obtain music that matches their current emotional state.

[0053] The above description of the embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for recommending music for smart home devices based on continuous user emotions, characterized in that: Specifically, the following steps are included: Step 1: Generate a user feature matrix based on user basic information and historical interaction logs, including user voiceprint features, music preference types, user basic information, and historical playback records; Step 2: Collect user voice data and use a voiceprint recognition model to identify the user ID; select the feature column corresponding to the user ID from the user feature matrix established in Step 1, and use a collaborative filtering algorithm to process the selected feature column to generate a personalized music recommendation list. Step 3: Use a deep fully convolutional network (FCN) to perform sentiment recognition on the music data in the music recommendation list obtained in Step 2, and output a three-dimensional PAD sentiment feature matrix. , where P m A m D m These represent the emotional pleasure, activation, and dominance of the input music data, respectively. Step 4: Calculate the similarity between different songs based on emotional features and construct a music similarity matrix; Step 5: Use a deep fully convolutional network (FCN) to perform sentiment analysis on the user's real-time speech, and output a three-dimensional PAD sentiment feature matrix. , where P u A u D u These represent the user's current level of enjoyment, engagement, and advantage in speaking, respectively. Step 6: Calculate the emotional similarity between user voice and music data in terms of pleasure and activation, as well as the emotional complementarity in terms of dominance, to obtain the user-music emotional matching matrix. Select the music with the highest matching degree with the user's current voice from the music recommendation list and play it. Based on the music similarity matrix obtained in Step 4, sort and play the music in the music recommendation list in descending order of similarity with the currently playing music. The emotional similarity between the user's voice and music data in terms of pleasure and activation. ,in, , ; Emotional complementarity in terms of strengths The degree of matching between the user's current voice and music data ,in This is a weighting factor.

2. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: Mel frequency cepstral coefficients and linear predictive codes of the user's speech are extracted, concatenated in sequence, and input into a pre-trained voiceprint recognition model to identify the user's ID.

3. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: In the music similarity matrix, each element represents the similarity score between two input music data. The higher the similarity score, the higher the similarity between the corresponding music data.

4. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: The trade-off factors > In other words, in the calculation of the matching degree between the user's current voice and music data, the weight of emotional similarity is greater than that of emotional complementarity.

5. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: The personalized music recommendation list includes not only music with high historical play rates from user interaction logs, but also new songs that have never been played.

6. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: The deep fully convolutional network FCN consists of four cascaded 3x3 convolutional layers and a max pooling layer.

7. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: Calculate the cosine similarity between different music data in the playlist To construct a music similarity matrix: ； in, , , These are the model lengths of the two pieces of music in a three-dimensional emotional space.

8. The method for recommending music for smart home devices based on continuous user emotions as described in claim 1, characterized in that: During music playback, the user's voice data is collected synchronously, and sentiment analysis is performed according to step five. When the user's emotional state changes, the matching degree of the music data is recalculated according to step six, and the music recommendation list and playback order are adjusted.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of any one of claims 1 to 8.

Images