Vehicle-mounted music interaction system based on context awareness

Abstract

The invention belongs to the technical field of vehicle-mounted human-computer interaction, and particularly relates to a vehicle-mounted music recommendation system based on context awareness, which covers the technologies of computer vision, recommendation systems, human-computer interaction and the like. The whole system is divided into two main parts: the first part is a context information collection module, and the module realizes real-time recognition of the emotion of a driver by collecting facial images and voice information of the driver; and the second part is a feedback module used for processing interaction between the driver and different emotion analysis results, including mapping rule reasoning and music recommendation. The system generates a personalized music recommendation list by analyzing the facial expression of the driver and the emotional characteristics extracted from the voice information, so as to reduce the music selection burden of the driver and prevent the occurrence of extreme emotions, thereby having important significance for guaranteeing the personal safety of the driver and the road traffic safety.

Description

Technical Field

[0001] The present invention relates to a vehicle-mounted music interaction system based on context perception, belonging to the technical fields of computer vision, recommendation system, human-computer interaction and the like. Background Art

[0002] Music is not only a means of entertainment during driving, but also an important factor in regulating the driver's emotions and affecting driving performance. Studies have shown that appropriate music can effectively relieve fatigue and anxiety caused by long-term driving, improve driving concentration and reaction speed, and reduce the risks caused by emotional fluctuations. Traditional in-car music systems mostly rely on the driver's manual operation or limited preset options. Users need to actively select the music to be played, adjust the volume or switch tracks. This operation is easy to distract attention during driving and increase potential safety hazards. With the development of intelligent connected vehicle technology, smart cockpits have gradually become the core of the in-car experience. Today's smart cockpits not only focus on the user's operational needs, but also pay more attention to the user's emotions and status management. Under this trend, the role of in-car personalized applications in car cockpits is becoming more and more important. As an important part of the human-computer interaction in the cabin, the intelligent in-car music interaction system can not only enrich the in-car experience, but also reduce the driver's distraction and improve driving safety.

[0003] To meet this demand, it is necessary to develop a context-aware in-vehicle music interaction system. This system can obtain the driver's facial expressions and voice information, judge the driver's status and needs in real time, and provide the driver with a natural and continuous interactive experience.

[0004] There are many existing music recommendation systems. However, most of them focus on simple music classification, such as recommending music by completing the genre classification of music works and the historical information of users. Some music recommendation systems also obtain users' preferences for different music features, establish a project matrix between users, and combine user information with users' music preference information to recommend music. However, these do not take into account the complex and changing environment and rich emotions around the driver, and it is difficult to regulate the extreme driving emotions that the driver may have.

[0005] With the development of emotion recognition technology and recommendation systems, we can consider integrating driver emotion analysis with the recommendation system to capture the driver's possible emotional state in real time and recommend music that meets their current emotional needs. This approach is of great significance to ensuring the driver's personal safety and road traffic safety.

[0006] At present, emotion recognition technology based on computer vision-related algorithms is already very mature. Based on the driver's facial image, the present invention uses a recurrent neural network (RNN) to extract the driver's facial information, and on this basis, the driver's voice information is simultaneously considered. By combining the two modal information, the user's emotion category is analyzed and the relevant emotion vector is generated. The emotion vector generated in the first stage can be directly used as the input of the second stage. In the second stage, the system integrates the user's emotion analysis results and historical interaction records through the interactive feedback module, and evaluates the potential impact of the recommended music on the driver's cognitive state. Through the introduction of adaptive rules, the system can realize personalized music recommendations for the driver while ensuring driving safety.

[0007] The present invention combines the driver's facial expression with voice emotion recognition to intelligently analyze the driver's emotions and generate a music recommendation list. The system not only helps to regulate the driver's emotions and improve driving safety, but also helps to enhance the driving experience and reduce safety hazards caused by emotional fluctuations. Summary of the invention

[0008] To solve the above problems, the present invention proposes an in-vehicle music interaction system based on context perception, which aims to provide an in-vehicle music interaction method that can not only meet the personalized needs of the driver, but also regulate the driver when he or she has extreme emotions. The system is mainly divided into two parts: a context information extraction module and a user interaction feedback module with different context information, which mainly includes the following steps:

[0009] S1: Context information collection part, during the driver's driving, the driver's facial expressions and voice information are collected in real time through the on-board camera and microphone. This information is used to capture the driver's emotional changes and driving status.

[0010] S2: The driver's real-time facial expression and voice information are input into the emotion recognition module, which extracts the driver's emotional state based on a dual-modal emotion recognition method that fuses facial expression and voice information.

[0011] S3: Based on the driver's emotional state, the system generates a list of in-car music recommendations. The recommendation list consists of two parts: the first part is the mapping rule reasoning module, which uses preset rules to infer the appropriate music type based on the emotion recognition results. The second part is based on the driver's personalized needs, and on the premise of ensuring safe driving, it makes personalized music recommendations to reduce the driver's burden of choosing music.

[0012] S4: The driver's emotion classification result is transmitted to the backend server, and the server generates and feeds back a music recommendation list based on the emotion-tagged music works stored in the database.

[0013] S5: When the driver emotion recognition module detects that the driver is in an extreme emotional state, the system will recommend music based on the preset mapping rules to help relieve emotions and ensure driving safety.

[0014] S6: When the driver’s mood is within the normal range, the interactive module considers the driver’s historical preferences more and provides personalized recommendations based on previous music preferences to enhance the driving experience.

[0015] Beneficial effects of the present invention:

[0016] The present invention first proposes to perform emotion recognition by combining the driver's facial expressions and voice to achieve intelligent analysis of the driver's emotions, thereby generating a driving emotion score, and then generating a music recommendation list based on the driver's real-time emotion score and historical interaction records during driving. When the emotion score is within the normal range, the system pays more attention to the evolution of the driver's interest preferences. When the driver's emotion score is judged to be an extreme emotion, the system recommends music with relevant attribute tags according to the inference rules. The system not only reduces the driver's selection burden, but also helps to regulate the driver's emotions and enhance driving safety. By providing a suitable music environment, the safety hazards caused by emotional fluctuations during driving are effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0018] Figure 1 A flow chart of the multimodal emotion recognition algorithm provided in the present invention;

[0019] Figure 2 This is a framework diagram of the music interaction module provided in the present invention;

[0020] Figure 3 The present invention provides an in-vehicle music interaction system based on context perception. DETAILED DESCRIPTION

[0021] The following is a detailed description of the examples of the present invention in conjunction with the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0022] The present invention provides a context-aware in-vehicle music interaction system, the method flow is as follows: Figure 1 As shown, including:

[0023] S1: In the context-aware in-vehicle music interaction system, collecting the driver's emotional and physiological state data is the basis for the system to respond intelligently. This process is mainly achieved through the in-vehicle camera and microphone, which monitors the driver's facial expressions and voice information in real time to extract features related to his emotional state. The following are the specific implementation details:

[0024] Using computer vision libraries such as OpenCV, the driver's facial image is collected through the on-board camera, and facial detection is first performed. The location of the face in the image is identified, and the key facial feature points are extracted based on the predefined facial feature point model. Common facial feature points include eyes, mouth corners, eyebrows, etc. Through the facial key point detection method in the Dlib library, 68 standard facial feature points can be extracted, including eyebrows, eyes, nose, mouth, etc.

[0025] The driver's voice information is also an important basis for emotion analysis. The onboard microphone can clearly collect the driver's voice signal, and the Python Speech Recognition library can be used for signal preprocessing, such as noise reduction, volume normalization, and feature extraction. Combined with the voice's pitch, speaking speed, volume and other features, the system can further analyze the driver's emotional changes.

[0026] S2: Input the driver's real-time facial expression and voice information into the emotion recognition module, which extracts the driver's emotional state based on the dual-modal emotion recognition method that fuses facial expression and voice information;

[0027] Taking images and voice as input, a multimodal emotion recognition algorithm is used to capture the driver's state in real time. Specifically, after the image data is preprocessed (such as grayscale, cropping, standardization, etc.), a recurrent neural network (RNN) is used to extract facial expression features. After recognizing the facial expression, the network will extract various features in the image, such as the degree of eyebrow rise, the degree of curvature of the mouth corners, the degree of eye opening, etc. These features reflect the driver's emotional state and can be used as input for subsequent emotion classification.

[0028] Mel-Frequency Cipstal Coefficients (MFCC) are used as the main features of speech signals to capture the spectral information of speech. MFCC is a feature widely used in speech processing, which can effectively reflect the spectral information of speech signals. By performing short-time Fourier transform on the speech signal, its cepstral coefficients at the Mel-frequency scale are extracted. MFCC can capture the timbre and emotional changes of speech, so it is very suitable for emotion recognition tasks. In addition, by extracting features such as the pitch, volume, and speaking speed of speech, the basis for emotion recognition is enriched. Finally, in order to improve the accuracy of emotion recognition, the system adopts a multimodal fusion method to combine speech and image features, and uses a multi-layer perceptron (MLP) as a fusion network for processing.

[0029] The Mel-frequency cepstral coefficients are calculated by the following steps: First, pre-emphasis. In speech signals, the energy of the high-frequency part is usually low. In order to balance the spectral energy distribution and enhance the high-frequency components, a high-pass filter needs to be applied to the signal.

[0030] y[n]=x[n]-α·x[n-1]

[0031] Among them, x[n] is the original speech signal, y[n] is the pre-emphasized signal, and the value of α is usually 0.97. This step appropriately enhances the high-frequency components, which helps to capture richer spectral information in subsequent steps. Since the speech signal is a non-stationary signal, dividing it into relatively stable segments (frames) in a short period of time for analysis helps to capture local speech features. Usually the frame length is 20-30 milliseconds, the frame shift is 10 milliseconds, and the frame shift is the difference in starting positions of adjacent frames. This choice ensures that there is partial overlap between frames, making the signal more continuous. Since the signal changes dramatically at the boundary of each frame, a smooth window function needs to be applied to reduce the boundary effect. Apply the window function to each frame, and then calculate the frequency of each frame The FFT result is filtered through the Mel filter to obtain the Mel frequency energy, the logarithm of the Mel frequency energy is taken, and finally, DCT is applied to obtain MFCC.

[0032]

[0033] Then retain the first 12-13 coefficients as the final MFCC features. Intonation changes in speech can express the intensity of emotions, while facial expressions can more directly reflect the type of emotions. Fusion features enable the system to utilize the complementary information of these two types of signals, thereby improving the accuracy of overall emotion recognition. Since speech and image information are two completely different modalities, how to effectively combine the two is the key to enhancing accuracy in emotion recognition. Multimodal feature concatenation is a common feature fusion method because it can simply and efficiently combine information from different modalities into one, allowing subsequent models to capture the dependencies between modalities without complex alignment operations. Therefore, for fusion features, a simple concatenation is sufficient: F fused =[F image ,F audio ], and send the fused features to the MLP multi-layer perceptron. As a basic neural network structure, MLP can capture complex feature relationships from images and speech. As the network depth increases, MLP can more accurately fuse features and extract potential feature relationships between different modalities. The forward propagation formula of the MLP multi-layer perceptron is:

[0034] Z (l) =W (l) ·A (l-1) +b (l)

[0035] A (l) =σ(Z (l) )

[0036] S3: Generates a car music recommendation list based on the driver's emotional state, which includes two parts: the first part is the mapping rule reasoning module, which uses preset rules to infer the appropriate music type based on the emotion recognition results. The second part realizes personalized music recommendation on the basis of ensuring the driver's safe driving, reducing the burden of music selection for the driver.

[0037] This module aims to generate suitable music genres based on real-time emotion recognition results. The core steps include:

[0038] 1. Analysis of emotion recognition results: The system obtains the driver’s real-time emotional state (such as happiness, anger, anxiety, etc.) through a multimodal emotion recognition algorithm.

[0039] 2. Mapping rule definition: Based on emotion theory and psychological research, a set of preset music type mapping rules are designed.

[0040] Happy mood → light pop music, jazz

[0041] Sad mood → Lyrical songs, classical music

[0042] Anxiety → Meditation music, light music

[0043] 3. The rules can be optimized and iterated through expert interviews or data analysis to ensure that they adapt to the preferences of different drivers.

[0044] 4. Build a personalized music preference model based on the driver’s historical music listening data, ratings, and feedback. Use collaborative filtering, content recommendation, or deep learning methods to model user preferences.

[0045] Through the combination of the above two modules, the system can achieve accurate emotional response and personalized recommendations to enhance the driving experience. At the same time, the system has certain safety constraints, that is, during the recommendation process, the system needs to ensure that the recommended music does not cause excessive emotional fluctuations or distract attention.

[0046] S4: The system transmits the classification results of the user's current emotional state to the backend server in real time. The server uses the emotional tags of the music works stored in the database to generate a personalized recommendation list that matches the user's emotions. Specifically, the recommendation process is achieved by calculating the similarity between the emotion vector and the music tag vector, and the cosine similarity method is used to score the matching degree between the two to ensure the emotional fit of the recommended music.

[0047] The classification result of the user's current emotional state is usually expressed in the form of an emotion vector, which contains multi-dimensional emotional features, such as pleasure, energy, tension, etc. Each piece of music also has a corresponding emotion label pre-stored in the database as a feature vector representation of the music. Both the emotion vector E and the music label vector M can be expressed as multi-dimensional vectors. Assuming that they are both n-dimensional, then: E = [e 1 ,e 2 ,…,e n ],M=[m 1 ,m 2 ,…,m n ], where e i and m i Indicates the value of emotion and music tag on the i-th emotion dimension. In order to measure the similarity between the user's emotional state and the music emotion tag, the system uses cosine similarity, that is, calculating the cosine value of the angle between the emotion vector and the music tag vector. The closer the similarity value is to 1, the higher the emotional match between the two. The specific cosine similarity formula is as follows:

[0048]

[0049] S5: When the driver's emotions are detected to be within the normal range, the interaction module takes the driver's historical preferences into consideration;

[0050] After determining the current user's emotional state and related music types, the scoring formula for generating the recommendation list is:

[0051] Score(M j )=α·Similarity(U,M j )+β·Emotion M atch(M j ,E)+γ·User H istory(M j ,H)

[0052] Among them, M j represents the jth piece of music, U represents the user preference vector, E represents the current emotional state, H represents the user's historical preference, and α, β, and γ are weight parameters used to adjust the influence of each feature in the score.

[0053] S6: The system monitors the driver's emotional state in real time through the emotion extraction module and determines whether he is in an extreme emotion (such as anger, anxiety, etc.). When the emotional state reaches the set warning threshold, the system automatically activates the music recommendation process and makes personalized music recommendations based on the pre-designed mapping rules. The recommendation process first gives a comprehensive score to all candidate music works to quantify the degree of match between each piece of music and the current emotional state. After the score is calculated, the system arranges all candidate music in descending order of the comprehensive score, and selects the top N pieces of music with the highest score to generate a recommendation list to ensure that the recommended content not only meets emotional needs but also has a significant soothing or relaxing effect.

[0054] Through the synergy of the above modules, the in-vehicle music recommendation system can realize intelligent emotion perception and music recommendation, which not only meets the personalized needs of users, but also focuses on driving safety. The system adopts different strategies under different emotional states to ensure that the driver can enjoy a personalized music experience while driving safely.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A context-aware in-vehicle music interaction system, characterized in that: The following steps are involved: Step 1: Collect and process the data set: Obtain the driver's facial image and real-time voice through the on-board camera and microphone. Collect the music style and specific music ID that the driver listens to while driving. Process each piece of data into a sequence sample based on the timestamp, where each piece of music is represented by a specific ID. Step 2: Extract the driver's emotional state based on the dual-modal emotion recognition method based on the fusion of facial expression and voice information: Use a recurrent neural network (RNN) to extract facial expression features, and then use Mel-Frequency Cipstal Coefficients (MFCC) to capture the spectral information of the voice signal to enrich the basis for emotion recognition. Finally, use a multi-layer perceptron (MLP) as a fusion network to combine the voice and image features and further process these fused features. Step 3: Encode the driver's emotional state into a specific ID based on the end-to-end learning in step 2. Associate the driver's emotional state with the in-car music style based on the results of psychological classification matching and questionnaire surveys. Step 4: The classification result of the user's current emotional state is transmitted to the background. The server feeds back the recommendation list to the user through the music works with emotional tags stored in the database. Step 5: When the driver emotion extraction module indicates that the driver is in an extreme emotional state, a pre-designed mapping rule can be used for music recommendation. The mapping rule is based on emotion theory and psychological research and provides a set of preset music type matching schemes. Here are some examples: Happy mood → light pop music, jazz Sad mood → Lyrical songs, classical music Anxiety → Meditation music, light music Step 6: When the driver's mood is within the normal range, a personalized music preference model is constructed based on the driver's historical music listening data, ratings, and feedback. The user's preferences are modeled by combining collaborative filtering, content recommendation, and deep learning methods.

2. The context-aware in-vehicle music interaction system according to claim 1 is characterized in that Incorporating contextual information during driving into the in-car music recommendation algorithm enables the in-car music interaction system to not only meet the driver's music needs while driving, but also has the function of regulating the driver's extreme emotions, which has a positive impact on driving safety.

3. The context-aware in-vehicle music interaction system according to claim 1 is characterized in that In step 2, the dual-modal emotion recognition method based on the fusion of facial expressions and voice information can be used as a supplementary source to ensure continuous monitoring of emotional states due to factors such as the driver's limited mobility while driving. The bimodal recognition method can combine the driver's voice and facial features, analyze contextual information, and evaluate a more realistic emotional state. The multi-layer perceptron (MLP) is used to effectively fuse facial expression features and voice features to comprehensively consider multimodal information, thereby improving the robustness and accuracy of emotion recognition.

4. The context-aware in-vehicle music interaction system according to claim 1, characterized in that The music interactive feedback module in step 5 and step 6. This module can intelligently analyze the driver's current state based on the results of emotion recognition and generate a music recommendation list. By mapping rule reasoning and music recommendation, the driver's extreme emotions can be regulated while meeting the driver's personalized needs, thereby improving safe driving.

Images