Intelligent vehicle-mounted entertainment interaction system based on user expression recognition

Abstract

The invention relates to an intelligent vehicle-mounted entertainment interaction system based on user expression recognition, and the system comprises a vehicle-mounted camera which is used for collecting a facial expression image of a user; the expression recognition module is used for acquiring the acquired facial expression image of the user from the vehicle-mounted camera and further processing and recognizing the expression state of the current user; and the AI entertainment screen control module is used for acquiring the current expression state information of the user from the expression recognition module and acquiring entertainment content matched with the current psychological state of the user in the vehicle-mounted terminal system for playing. According to the intelligent vehicle-mounted entertainment interaction system based on user expression recognition, corresponding music, videos and other entertainment contents can be automatically matched and played according to the expression state of the user, the entertainment experience better fitting the current emotion of the user is provided, and the fun and participation degree in the driving process are increased; and the interaction and social experience among passengers in the vehicle are enhanced, so that the journey is more pleasant and interesting.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile electronics, and in particular to an intelligent in-vehicle entertainment interactive system based on user expression recognition. Background Art

[0002] With the development of intelligent vehicles, in-vehicle entertainment systems have become an indispensable part of modern cars. However, most existing in-vehicle entertainment systems can only provide fixed entertainment content, such as only providing local music and video playback in the car or playing music or videos from music and video apps on the in-vehicle entertainment system. The music and video content in existing in-vehicle entertainment systems are mainly played in a targeted manner based on user instructions, and cannot be adjusted according to the real-time emotional state of the driver and passengers. Summary of the invention

[0003] The present invention aims to solve the technical problem in the prior art that entertainment contents such as music and videos are mainly played in a targeted manner according to user instructions, but cannot be adjusted according to the real-time emotional state of the driver and passengers, and provides an intelligent in-vehicle entertainment interactive system based on user expression recognition.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] An intelligent in-vehicle entertainment interactive system based on user expression recognition, comprising:

[0006] An on-board camera for collecting images of the user's facial expressions;

[0007] An expression recognition module is used to obtain the user's facial expression image collected from the vehicle-mounted camera, and further process and recognize the current user's expression state;

[0008] The AI ​​entertainment screen control module is used to obtain the user's current expression state information from the expression recognition module, and obtain entertainment content matching the user's current psychological state for playback in the vehicle system.

[0009] Furthermore: the expression recognition module includes:

[0010] A data preprocessing unit performs denoising, contrast enhancement, and normalization on the image captured by the vehicle-mounted camera to improve image quality; and accurately locates the face area from the complex background, and crops and scales it for subsequent feature extraction and recognition;

[0011] A feature extraction unit, which uses computer vision technology to extract key information that can reflect the facial features from the pre-processed face image, wherein the key information includes the shape, texture and relative position of eyebrows, eyes, mouth and other parts;

[0012] A model inference unit, which sends the features extracted by the feature extraction unit to a well-trained machine learning model for inference; the machine learning model determines the expression state of the face image according to the input feature vector;

[0013] The post-processing and result output unit performs result smoothing and confidence assessment to reduce misrecognition and improve the robustness of the system; outputs one or more most likely expression categories and their corresponding confidence scores.

[0014] Furthermore: the computer vision technology adopted by the feature extraction unit includes: Haar feature technology, local binary pattern or convolutional neural network technology in deep learning.

[0015] Further: the AI ​​entertainment screen control module includes:

[0016] The music playback adjustment unit automatically switches the music APP on the car system to light and cheerful music when it detects that the user shows a happy or joyful expression; if it detects that the user shows a tired or depressed expression, it automatically switches the music APP on the car system to soft and relaxing music; if it detects that the user shows an excited or excited expression, it automatically switches the music APP on the car system to fast-paced and dynamic music.

[0017] Further: the AI ​​entertainment screen control module also includes:

[0018] The picture display adjustment unit controls the vehicle system to display different visual contents according to the user's expression category; when it is detected that the user shows a happy or joyful expression, the screen of the vehicle system is controlled to display tranquil natural scenery or warm family scenes; if it is detected that the user shows an excited or excited expression, the screen of the vehicle system is controlled to display dynamic sports scenes or exciting adventure pictures; if it is detected that the user shows a tired or depressed expression, the screen of the vehicle system is controlled to display warm and inspirational short films or patterns.

[0019] Further: the AI ​​entertainment screen control module also includes:

[0020] The interactive game recommendation unit recommends different types of interactive games based on the user's expression category; when it is detected that the user shows a happy or joyful expression, it recommends more lively and interesting puzzle games or competitive games; if it is detected that the user shows a tired or frustrated expression, it provides easy and simple casual games.

[0021] Further: the AI ​​entertainment screen control module also includes:

[0022] The fatigue detection and warning unit will immediately trigger the warning system if it detects that the user is showing a tired expression, and alert the driver through sound, vibration or visual prompts; the system automatically adjusts parameters such as screen brightness and color temperature to reduce visual fatigue for the driver;

[0023] The distraction detection and content adjustment unit will issue a reminder through the entertainment screen if it detects that the user shows a distracted expression; the system may automatically lower the music volume, simplify the screen display content or pause the current entertainment activity to ensure that the driver's attention is focused on the road conditions.

[0024] Further: Also including:

[0025] The entertainment content database communicates with the AI ​​entertainment screen control module and the vehicle system, and is used to store audio, video, game and picture content used by the vehicle system, and provide corresponding content to the vehicle system according to the instructions of the AI ​​entertainment screen control module.

[0026] The intelligent in-vehicle entertainment interactive system based on user expression recognition provided by the present invention has at least the following beneficial effects or advantages:

[0027] The intelligent in-vehicle entertainment interactive system based on user expression recognition provided by the present invention has an in-vehicle camera configured to collect the user's facial expression image; the expression recognition module obtains the collected user's facial expression image from the in-vehicle camera, and further processes and recognizes the current user's expression state; the AI ​​entertainment screen control module obtains the user's current expression state information from the expression recognition module, and obtains entertainment content matching the user's current psychological state in the vehicle system for playback. The intelligent in-vehicle entertainment interactive system based on user expression recognition can automatically match and play corresponding entertainment content such as music and videos according to the user's expression state, provide an entertainment experience that is more in line with the user's current emotions, increase the fun and participation in the driving process; enhance the interaction and social experience between passengers in the car, and make the journey more pleasant and interesting. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A principle block diagram of an intelligent in-vehicle entertainment interactive system based on user expression recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention aims to solve the technical problem in the prior art that entertainment contents such as music and videos are mainly played in a targeted manner according to user instructions, but cannot be adjusted according to the real-time emotional state of the driver and passengers, and provides an intelligent in-vehicle entertainment interactive system based on user expression recognition.

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] See also Figure 1 The embodiment of the present invention provides an intelligent in-vehicle entertainment interactive system based on user expression recognition, comprising:

[0032] The vehicle-mounted camera is used to collect images of the user's facial expressions.

[0033] The expression recognition module is used to obtain the user's facial expression image collected from the vehicle-mounted camera, and further process and recognize the current user's expression state.

[0034] The AI ​​entertainment screen control module is used to obtain the user's current expression status information from the expression recognition module, and obtain entertainment content matching the user's current psychological state for playback in the vehicle system.

[0035] The entertainment content database communicates with the AI ​​entertainment screen control module and the vehicle system. It is used to store audio, video, game and picture content used by the vehicle system, and provide corresponding content to the vehicle system according to the instructions of the AI ​​entertainment screen control module.

[0036] The intelligent in-vehicle entertainment interaction system based on user expression recognition provided by an embodiment of the present invention needs to perform system initialization when working. At the beginning of the system startup, a series of initialization operations need to be performed. First, start the on-board camera to ensure that its position is correct and the focal length is appropriate so that the facial expressions of the driver and passengers can be clearly captured. Next, load and initialize the expression recognition module to ensure that its algorithm model is ready and can accurately recognize various facial expressions. At the same time, load the AI ​​entertainment screen control module and be ready to dynamically adjust the entertainment content according to the expression recognition results. Finally, initialize the entertainment content database to ensure that various types of entertainment content (such as music, videos, games, etc.) have been correctly classified and stored for subsequent rapid retrieval and updating.

[0037] The following is a further description of the various components of the intelligent in-vehicle entertainment interactive system based on user expression recognition.

[0038] The expression recognition module includes: data preprocessing unit, feature extraction unit, model inference unit, model reasoning unit and post-processing and result output unit. Specifically:

[0039] The original image captured by the vehicle camera needs to be preprocessed to improve the image quality and reduce noise. The preprocessing steps may include graying, denoising, histogram equalization, etc. to enhance the contrast of facial features and prepare for subsequent feature extraction and recognition. In this embodiment, a data preprocessing unit is used to perform denoising, contrast enhancement, and normalization on the image captured by the vehicle camera to improve the image quality; and accurately locate the face area from the complex background, and crop and scale it for subsequent feature extraction and recognition.

[0040] Once the face is detected and located, the algorithm will further extract facial features; these features may include the position, shape and texture information of key points such as facial contours, eyes, mouth, eyebrows, etc.; deep learning models, such as convolutional neural networks (CNNs), perform well in this regard and can automatically learn and extract high-level feature representations. In this embodiment, a feature extraction unit is used and computer vision technology is used to extract key information that can reflect expression features from the pre-processed face image, and the key information includes the shape, texture and relative position of eyebrows, eyes, mouth and other parts. The computer vision technology used by the feature extraction unit includes: Haar feature technology, local binary pattern or convolutional neural network technology in deep learning.

[0041] The model inference unit feeds the features extracted by the feature extraction unit into a well-trained machine learning model for inference; the machine learning model determines the expression state of the face image based on the input feature vector. The extracted facial features are fed into a well-trained classifier, such as a support vector machine (SVM), random forest, or deep neural network, to classify and recognize expressions. This classifier is trained with a large number of datasets with expression labels and can recognize a variety of basic emotions, such as happiness, sadness, anger, surprise, fear, disgust, and neutrality. In addition to basic emotion classification, advanced AI algorithms can also estimate the intensity of emotions. This usually involves further processing of the classifier output, such as using a regression model to predict the intensity of the emotion, thereby providing the system with more delicate emotional feedback. Throughout the process, the algorithm is also optimized based on real-time feedback. For example, if the system finds that the accuracy of a certain recognition is not high, it may automatically adjust certain parameters or retrain the model to improve future recognition performance.

[0042] The post-processing and result output unit is used to perform result smoothing and confidence assessment to reduce misrecognition and improve the robustness of the system; and output one or more most likely expression categories and their corresponding confidence scores.

[0043] The in-vehicle entertainment system uses an integrated expression recognition module to monitor and analyze the facial expressions of the driver and passengers in real time, thereby identifying their emotional states. Based on these emotional data, the system automatically adjusts the entertainment content played to provide an experience that is more in line with the user's current mood. For example, when the user appears tired, the system may play relaxing music or natural scenery videos to help the user relax. In this embodiment, an AI entertainment screen control module is used to implement the above functions. The AI ​​entertainment screen control module includes: a music playback adjustment unit, a screen display adjustment unit, an interactive game recommendation unit, a fatigue detection and warning unit, and a distraction detection and content adjustment unit. Specifically:

[0044] The music playback adjustment unit is used to control the vehicle system to display different music content according to the user's expression category. When it is detected that the user shows a happy or joyful expression, the music APP on the vehicle system will automatically switch to light and cheerful music for playback; if it is detected that the user shows a tired or depressed expression, the music APP on the vehicle system will automatically switch to soft and relaxing music for playback; if it is detected that the user shows an excited or excited expression, the music APP on the vehicle system will automatically switch to brisk and dynamic music for playback.

[0045] The picture display adjustment unit is used to control the vehicle system to display different visual contents according to the user's expression category; when it is detected that the user shows a happy or joyful expression, the screen of the vehicle system is controlled to display tranquil natural scenery or warm family scenes; if it is detected that the user shows an excited or excited expression, the screen of the vehicle system is controlled to display dynamic sports scenes or exciting adventure scenes; if it is detected that the user shows a tired or depressed expression, the screen of the vehicle system is controlled to display warm and inspirational short films or patterns.

[0046] The interactive game recommendation unit recommends different types of interactive games based on the user's expression category; when it is detected that the user shows a happy or joyful expression state, more lively and interesting puzzle games or competitive games are recommended; if it is detected that the user shows a tired or frustrated expression state, easy and simple casual games are provided.

[0047] The fatigue detection and warning unit will issue a warning reminder when it collects a specific user's expression category. If it detects that the user shows a tired expression, the warning system will be triggered immediately to alert the driver through sound, vibration or visual prompts; the system automatically adjusts parameters such as screen brightness and color temperature to reduce visual fatigue for the driver.

[0048] The distraction detection and content adjustment unit will issue warning reminders and automatically adjust the system when it collects specific user expression categories. If it detects that the user shows a distracted expression, a reminder will be issued through the entertainment screen; the system may automatically lower the music volume, simplify the screen display content or pause the current entertainment activity to ensure that the driver's attention is focused on the road conditions.

[0049] In addition, the AI ​​entertainment screen control module also integrates the following functions:

[0050] User preference learning and content recommendation: The in-car entertainment system gradually learns and understands the user's entertainment preferences by collecting and analyzing the user's historical usage data, such as viewing records and song play frequency. Based on these learning results, the system can intelligently recommend entertainment content that suits the user's taste. In addition, users can also actively provide feedback on the recommended content, and the system will continuously optimize the recommendation algorithm based on these feedbacks to improve the accuracy of the recommendation.

[0051] Real-time interaction and feedback mechanism. In order to enhance the user's sense of participation and immersion, the in-vehicle entertainment system provides a wealth of real-time interactive functions. Users can interact with the system through touch screens, voice recognition, etc., such as liking, commenting, sharing, etc. The system will instantly adjust the playback content or recommend related entertainment projects based on the user's interactive behavior. At the same time, user feedback will also be collected and analyzed by the system for subsequent content optimization and recommendation strategy adjustment.

[0052] Multimodal interactive experience: The in-car entertainment system supports multiple modes of interaction, including voice, gesture, touch, etc. This multimodal interaction not only improves the convenience of operation, but also enables users to communicate with the system in a more natural way. For example, users can switch songs or play videos through voice commands, or adjust the volume or progress through gesture control. These interactive methods together create a full range of immersive entertainment environment for users.

[0053] Social functions and content sharing. The in-vehicle entertainment system also integrates social functions, allowing users to share the entertainment content they are enjoying with other in-vehicle system users or friends on social media. This social interaction not only enriches the user's entertainment experience, but also builds a platform for communication and sharing between users. Through social functions, users can discover more interesting entertainment content and share happiness with others.

[0054] Personalized settings and memory functions. In order to meet the personalized needs of different users, the present invention allows users to set the system's response to different emotions according to their preferences. For example, users can choose to play a specific music list or display customized pictures when they are happy. These personalized settings will be learned and memorized by the system to provide more accurate entertainment recommendations in the future.

[0055] The intelligent in-vehicle entertainment interactive system based on user expression recognition provided by the embodiment of the present invention has at least the following beneficial effects or advantages:

[0056] The intelligent in-vehicle entertainment interactive system based on user expression recognition provided by the embodiment of the present invention is configured with an in-vehicle camera to collect the user's facial expression image; the expression recognition module obtains the collected user's facial expression image from the in-vehicle camera, and further processes and recognizes the current user's expression state; the AI ​​entertainment screen control module obtains the user's current expression state information from the expression recognition module, and obtains entertainment content matching the user's current psychological state in the vehicle system for playback. The intelligent in-vehicle entertainment interactive system based on user expression recognition can automatically match and play corresponding entertainment content such as music and videos according to the user's expression state, provide an entertainment experience that is more in line with the user's current mood, increase the fun and participation in the driving process; enhance the interaction and social experience between passengers in the car, and make the journey more pleasant and interesting.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent in-vehicle entertainment interactive system based on user expression recognition, characterized in that: include: An on-board camera for collecting images of the user's facial expressions; An expression recognition module is used to obtain the user's facial expression image collected from the vehicle-mounted camera, and further process and recognize the current user's expression state; The AI ​​entertainment screen control module is used to obtain the user's current expression state information from the expression recognition module, and obtain entertainment content matching the user's current psychological state for playback in the vehicle system.

2. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 1 is characterized in that: The expression recognition module comprises: A data preprocessing unit performs denoising, contrast enhancement, and normalization on the image captured by the vehicle-mounted camera to improve image quality; and accurately locates the face area from the complex background, and crops and scales it for subsequent feature extraction and recognition; A feature extraction unit, which uses computer vision technology to extract key information that can reflect the facial features from the pre-processed face image, wherein the key information includes the shape, texture and relative position of eyebrows, eyes, mouth and other parts; A model inference unit, which sends the features extracted by the feature extraction unit to a well-trained machine learning model for inference; the machine learning model determines the expression state of the face image according to the input feature vector; The post-processing and result output unit performs result smoothing and confidence evaluation to reduce misrecognition and improve the robustness of the system; outputs one or more most likely expression categories and their corresponding confidence scores.

3. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 2 is characterized in that: The computer vision technology adopted by the feature extraction unit includes: Haar feature technology, local binary pattern or convolutional neural network technology in deep learning.

4. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 2 is characterized in that: The AI ​​entertainment screen control module includes: The music playback adjustment unit automatically switches the music APP on the car system to light and cheerful music when it detects that the user shows a happy or joyful expression; if it detects that the user shows a tired or depressed expression, it automatically switches the music APP on the car system to soft and relaxing music; if it detects that the user shows an excited or excited expression, it automatically switches the music APP on the car system to fast-paced and dynamic music.

5. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 4 is characterized in that: The AI ​​entertainment screen control module also includes: The picture display adjustment unit controls the vehicle system to display different visual contents according to the user's expression category; when it is detected that the user shows a happy or joyful expression, the screen of the vehicle system is controlled to display tranquil natural scenery or warm family scenes; if it is detected that the user shows an excited or excited expression, the screen of the vehicle system is controlled to display dynamic sports scenes or exciting adventure pictures; if it is detected that the user shows a tired or depressed expression, the screen of the vehicle system is controlled to display warm and inspirational short films or patterns.

6. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 4 is characterized in that: The AI ​​entertainment screen control module also includes: The interactive game recommendation unit recommends different types of interactive games based on the user's expression category; when it is detected that the user shows a happy or joyful expression, it recommends more lively and interesting puzzle games or competitive games; if it is detected that the user shows a tired or frustrated expression, it provides easy and simple casual games.

7. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 4 is characterized in that: The AI ​​entertainment screen control module also includes: The fatigue detection and warning unit will immediately trigger the warning system if it detects that the user is showing a tired expression, and alert the driver through sound, vibration or visual prompts; the system automatically adjusts parameters such as screen brightness and color temperature to reduce visual fatigue for the driver; The distraction detection and content adjustment unit will issue a reminder through the entertainment screen if it detects that the user shows a distracted expression; the system may automatically lower the music volume, simplify the screen display content or pause the current entertainment activity to ensure that the driver's attention is focused on the road conditions.

8. The intelligent in-vehicle entertainment interactive system based on user expression recognition according to claim 1 is characterized in that: Also includes: The entertainment content database communicates with the AI ​​entertainment screen control module and the vehicle system, and is used to store audio, video, game and picture content used by the vehicle system, and provide corresponding content to the vehicle system according to the instructions of the AI ​​entertainment screen control module.

Images