Vehicle-mounted interaction system and vehicle-mounted interaction method

Abstract

The invention relates to the technical field of intelligent cabins, and discloses a vehicle-mounted interaction system which comprises the steps that when a driver monitoring system recognizes a negative expression of a driver based on a collected first video stream, an emotion recognition request is sent to an in-vehicle passenger detection system, and the first video stream is sent to an emotion interaction system; when the in-vehicle passenger monitoring system receives an emotion recognition request and recognizes a negative body language of the in-vehicle passenger based on a collected second video stream, an emotion detection request is sent to a voice system, and the second video stream is sent to an emotion interaction system; when the voice system receives the emotion detection request and recognizes a negative language based on the collected voice signal, the voice system sends a voice signal to the emotion interaction system; the emotion interaction system determines the current emotion of the passenger in the vehicle based on the first video stream, the second video stream and the voice signal, and adjusts the interaction information in the vehicle based on the current emotion, so that the possibility of misreporting negative emotion is reduced, and the intelligent ability of the vehicle-mounted interaction system is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of intelligent cockpits of vehicles, and in particular to an in-vehicle interaction system and an in-vehicle interaction method. Background Art

[0002] With the rapid development of intelligent driving vehicle technology, people's demand for the intelligence and personalization of intelligent driving vehicles is increasing, and drivers' demand for comfortable and pleasant intelligent driving experience is also increasing. Therefore, how to provide more intelligent interactive services in the intelligent cockpit of intelligent driving vehicles has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, the present disclosure provides an in-vehicle interaction system and an in-vehicle interaction method to solve the problem of providing more intelligent interactive services in an intelligent cockpit of an intelligent driving vehicle.

[0004] In a first aspect, the present disclosure provides an in-vehicle interactive system, which includes: a driver monitoring system, which is used to send an emotion recognition request to an in-vehicle passenger detection system and send a first video stream to an emotional interaction system when a negative expression of the driver is recognized based on a collected first video stream; an in-vehicle passenger monitoring system, which is used to send an emotion detection request to a voice system and send a second video stream to the emotional interaction system when an emotion recognition request is received and negative body language of a passenger in the vehicle is recognized based on a collected second video stream; a voice system, which is used to send a voice signal to the emotional interaction system when an emotion detection request is received and negative language is recognized based on a collected voice signal; and an emotional interaction system, which is used to determine the current emotion of the passenger in the vehicle based on the first video stream, the second video stream and the voice signal, and adjust the interactive information in the vehicle based on the current emotion.

[0005] In a second aspect, the present disclosure provides an in-vehicle interaction method, which includes: using a driver monitoring system to send an emotion recognition request to an in-vehicle passenger detection system and send a first video stream to an emotional interaction system when a negative expression of the driver is recognized based on a collected first video stream; using an in-vehicle passenger monitoring system to send an emotion detection request to a voice system and send a second video stream to an emotional interaction system when an emotion recognition request is received and negative body language of a passenger in the vehicle is recognized based on a collected second video stream; using a voice system to send a voice signal to an emotional interaction system when an emotion detection request is received and negative language is recognized based on a collected voice signal; using the emotional interaction system to determine the current emotion of the passenger in the vehicle based on the first video stream, the second video stream and the voice signal, and adjust the interactive information in the vehicle based on the current emotion.

[0006] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the in-vehicle interaction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the in-vehicle interaction method of the first aspect or any corresponding embodiment thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the in-vehicle interaction method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0009] The in-vehicle interactive system provided by the above-mentioned embodiment of the present disclosure determines the current emotion based on the integrated information of the emotion recognition results of three different data analysts only when the monitoring results of the three different hierarchical channels all indicate that the passengers in the car have negative emotions. Compared with adopting a single channel or a single-layer logic to monitor the emotions of the passengers in the car, the possibility of false positives of negative emotions is reduced, and the accuracy of the current emotions of the passengers in the car is improved. Then, the interactive information in the car is adjusted based on the current emotions, making the interactive information more targeted, thereby enhancing the personalized experience of the passengers and improving the intelligence and humanization of the in-vehicle interactive system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is an exemplary architecture diagram of an in-vehicle interactive system according to an embodiment of the present disclosure;

[0012] Figure 2 is an exemplary structural block diagram of an in-vehicle interactive system according to an embodiment of the present disclosure;

[0013] Figure 3 is a flow chart of an in-vehicle interaction method according to an embodiment of the present disclosure;

[0014] Figure 4 is a schematic flow chart of an in-vehicle interaction method according to an embodiment of the present disclosure;

[0015] Figure 5According to the embodiment of the present disclosure Figure 1 An exemplary working logic of the in-vehicle interactive system in;

[0016] Figure 6 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0018] In this embodiment, a vehicle-mounted interactive system is provided. Figure 1 is an exemplary architecture diagram of an in-vehicle interactive system according to an embodiment of the present disclosure, such as Figure 1 As shown, the vehicle-mounted interactive system includes a data acquisition module, a domain controller (system-on-chip SOC) and an interactive execution module.

[0019] Specifically, the data acquisition module includes the cabin camera and the microphone MIC of the voice system; the domain controller SOC is mainly responsible for algorithm and signal processing; the interactive execution module refers to the cabin environment, scene mode, ambient light and AI voice, etc.

[0020] The cameras in the data acquisition module include the driver monitoring system DMS and the in-vehicle passenger monitoring system OMS. These two and the microphone MIC of the voice system work independently. Only when both meet their respective judgment logics A and B at the same time can the SOC process the signal.

[0021] The SOC domain controller mainly includes a large model for emotion recognition. The processing of camera signals includes but is not limited to time synchronization, feature extraction, image encoding, and key action triggering. The processing of signals collected by the microphone MIC of the voice system includes but is not limited to signal amplification, beamforming, voice encoding, and keyword triggering. After processing the signal, the SOC will make relevant logical judgments on the display screen, such as displaying a pop-up window prompt. The options of the pop-up window include "Clear" and "OK". "Clear" means not starting the emotional interaction function and the program is closed; "OK" means starting the emotional interaction function, which will send a signal to the interactive execution module to make the next control. It should be noted that when different emotions are generated, the current pop-up window information may be ignored. Therefore, if no operation is performed within 3 seconds after the pop-up window information appears, the function logic judgment will start automatically and proceed to the next operation.

[0022] The interactive execution module refers to the functional implementation of emotional interaction, including the implementation of interactive functions such as air conditioning, scene mode, ambient lighting, and AI voice.

[0023] Please refer to Figure 2 , this embodiment provides a vehicle-mounted interactive system, Figure 2 is an exemplary structural block diagram of an in-vehicle interactive system according to an embodiment of the present disclosure, such as Figure 2 As shown, the in-vehicle interaction system includes a driver monitoring system 210 , an in-vehicle passenger monitoring system 220 , a voice system 230 and an emotional interaction system 240 .

[0024] The driver monitoring system 210 is used to send an emotion recognition request to the in-vehicle passenger detection system and send the first video stream to the emotion interaction system when a negative expression of the driver is recognized based on the collected first video stream.

[0025] In this embodiment, the Driver Monitoring System (DMS) is a driver monitoring and warning system based on the physiological reaction characteristics of the human body. It determines the driver's emotional state, fatigue state, and attention status by real-time monitoring and analyzing the driver's facial features, eye movements, head posture and other information to ensure driving safety.

[0026] The DMS can collect and analyze the video stream of the driving position (i.e., the first video stream) to identify the driver's facial expressions and determine whether the driver has negative expressions indicating negative emotions, such as fatigue, anger, or frustration. When the DMS detects these negative expressions, it will send an emotion recognition request to the in-vehicle passenger monitoring system and send the first video stream to the emotional interaction system as a response to the recognition of these negative emotions.

[0027] The in-vehicle passenger monitoring system 220 is used to send an emotion detection request to the voice system and send the second video stream to the emotional interaction system when receiving an emotion recognition request and identifying negative body language of the in-vehicle passenger based on the collected second video stream.

[0028] In this embodiment, the in-vehicle occupancy monitoring system (OMS) collects the behaviors and expressions of the in-vehicle occupants (including humans and other living things with vital signs) through various occupant detection sensors in the vehicle.

[0029] When OMS receives the emotion recognition request from DMS, it will start and collect the second video stream in the car, and by analyzing these video streams, it can identify the body language of passengers, such as posture, gestures and facial expressions, to determine whether there is negative body language, such as happiness, sadness, anger, tension, anxiety, etc. When OMS recognizes negative body language, it will send an emotion detection request to the voice system and send the second video stream to the emotional interaction system as a response to the recognition of these negative body languages.

[0030] The speech system 230 is used to send a speech signal to the emotion recognition module when receiving an emotion detection request and identifying negative language based on the collected speech signal.

[0031] In this embodiment, the voice system is responsible for collecting and analyzing voice signals in the car. When the voice system receives an emotion detection request from the OMS, it starts to monitor and collect voice signals in the car. By analyzing the collected voice signals, the voice system can identify negative language, such as complaints, quarrels, or curses, and then send voice signals to the emotion recognition module to further confirm the emotional state of the passengers in the car.

[0032] The emotional interaction system 240 is used to determine the current emotional state of the passengers in the car based on the first video stream, the second video stream, and the voice signal, and to adjust the interactive information in the car based on the current emotional state.

[0033] In this embodiment, the emotional interaction system can integrate information from the driver monitoring system, the in-car passenger monitoring system and the voice system to determine the current emotional state of the in-car passenger. Based on the current emotional state, the emotional interaction system can adjust the in-car interactive information, such as changing the in-car music, adjusting the lighting atmosphere, AI interactive information or providing personalized driving suggestions, etc., to better adapt to the needs and emotional state of the passenger. In a specific example, the emotional interaction system may include the following: Figure 1 The domain controller and interactive execution module in the architecture diagram shown.

[0034] The in-vehicle interactive system in this embodiment uses the driver monitoring system to identify the driver's facial expressions, and when a negative expression is identified, that is, when negative emotional information is identified for the first time, the in-vehicle passenger monitoring system is triggered to identify the body language of the in-vehicle passenger and send a first video stream to the emotional interaction system; when the in-vehicle passenger monitoring system identifies the negative body language of the in-vehicle passenger, that is, when negative emotional information is identified for the second time, the voice system is triggered to identify the voice signal and send a second video stream to the emotional interaction system; after the voice system identifies negative language, that is, when negative emotional information is identified for the third time, the voice signal is sent to the emotional interaction system; thereafter, the emotional interaction system is used to determine the current emotion of the in-vehicle passenger based on the first video stream, the second video stream and the voice signal, that is, to determine the emotion of the in-vehicle passenger for the fourth time, and adjust the interactive information in the vehicle based on the current emotion. During this process, when the monitoring results of the three different channels in a hierarchical and progressive manner all indicate that the passengers in the car have negative emotions, the current emotions are determined based on the integrated information of the emotion recognition results of three different data analysts. Compared with using a single channel or a single-layer logic to monitor the emotions of the passengers in the car, the possibility of false positives is reduced, and the accuracy of the current emotions of the passengers in the car is improved. The interactive information in the car is then adjusted based on the current emotions, making the interactive information more targeted, thereby enhancing the personalized experience of the passengers and improving the intelligence and humanization of the in-vehicle interactive system.

[0035] In some optional implementations of this embodiment, the emotional interaction system may also include: a personalization engine for generating an in-vehicle environment adjustment strategy based on the current emotions of the passengers in the vehicle, the historical emotional reactions and preferences of the passengers in the vehicle, and adjusting the in-vehicle interaction information according to the in-vehicle environment adjustment strategy.

[0036] The emotional interaction system in this implementation method, by setting up a personalized engine, introduces the historical emotional reactions and preferences of the passengers in the car when generating the in-car environment strategy, thereby generating an in-car environment adjustment strategy based on the accurate analysis of the passengers' current emotions, historical emotional reactions and preferences, and then adjusting the in-car interaction information according to the in-car environment adjustment strategy, making the in-car interaction information more targeted, improving the passengers' riding experience, and enhancing the vehicle's differentiated competitive advantage.

[0037] Please refer to Figure 3 , Figure 3 An exemplary structural diagram of the emotional interaction system in the vehicle-mounted interactive system of the embodiment of the present disclosure is shown. Figure 3 As shown, the emotion interaction system may include: a time synchronization unit 310 , a facial recognition unit 320 , a speech recognition unit 330 , a behavior recognition unit 340 , and a state generation unit 350 .

[0038] The time synchronization unit 310 is used to align the first video stream, the second video stream and the voice signal by using a time synchronization technology.

[0039] In this embodiment, the time synchronization unit can ensure that the first video stream, the second video stream and the voice signal are aligned in time during the emotion analysis process, so as to accurately perform emotion analysis using a time-synchronized data source.

[0040] For example, NTP (Network Time Protocol) or other time synchronization technologies can be used to ensure that the timestamps of all data sources are consistent, so that in the data preprocessing stage, the timestamps of video and voice signals are matched to ensure that these signals are aligned at the same time point.

[0041] The facial recognition unit 320 is used to use the facial recognition model to extract facial expression features for the aligned image frames in the first video stream and the image frames in the second video stream, respectively, to obtain the first facial expression features and the second facial expression features, and to generate a first emotion recognition probability based on the first facial expression features and the second facial expression features.

[0042] In this embodiment, the first emotion recognition probability indicates the probability that the emotion recognition result generated based on the first facial expression feature and the second facial expression feature is each emotion type in the emotion type set.

[0043] The facial recognition model can be trained through deep learning technology, can recognize and analyze subtle facial expression changes, and extract key facial expression features from the image frame. Based on the extracted facial expression features, the facial recognition unit 320 generates a first emotion recognition probability. This probability is a vector or array that contains the probability values ​​of each emotion type in the emotion type set. The emotion type set can include basic emotion types such as happiness, sadness, anger, and surprise.

[0044] The facial recognition unit extracts facial expression features from the image frames in the first video stream and the image frames in the second video stream respectively by using the facial recognition model, and generates emotion recognition probabilities based on these features, thereby improving the accuracy of subsequent emotion analysis and recognition.

[0045] For example, deep learning models (such as convolutional neural networks (CNNs)) can be used for facial recognition and expression feature extraction. Facial recognition models can be pre-trained to recognize specific expression features, such as smiling, frowning, drooping eyes, drooping mouth corners, and tightly closed lips, and associate these features with predefined emotion types.

[0046] The speech recognition unit 330 is used to use a speech analysis algorithm to extract speech emotion features for the aligned speech signal, and generate a second emotion recognition probability based on the speech emotion features; the second emotion recognition probability indicates the probability that the emotion recognition result generated based on the speech emotion features is each emotion type in the emotion type set.

[0047] In this embodiment, the speech recognition unit uses a speech analysis algorithm to extract emotional features in the speech signal, such as intonation, volume, speech speed, etc. These features are used to generate a second emotion recognition probability, that is, the possibility that the emotion recognition result generated based on the speech signal is each emotion type in the emotion type set.

[0048] For example, a speech analysis algorithm can use speech processing techniques (such as Mel-frequency cepstral coefficients MFCC) to extract speech features and adopt a pre-trained machine learning model (such as support vector machine SVM or recurrent neural network RNN) to identify emotional features in speech.

[0049] The behavior recognition unit 340 is used to use a behavior analysis algorithm to identify key emotional actions for image frames in the aligned second video stream, and to generate a third emotion recognition probability based on the key emotional actions; the third emotion recognition probability indicates the probability that the emotion recognition result generated based on the key emotional action is each emotion type in the emotion type set.

[0050] In this embodiment, the behavior recognition unit uses a behavior analysis algorithm to identify key emotional actions in the aligned second video stream, such as gestures, body postures, etc. These actions are used to generate a third emotion recognition probability, that is, the probability that the emotion recognition result generated based on the behavior characteristics is each emotion type in the emotion type set. Computer vision technology (such as target detection and tracking) is used to identify and analyze the behavior in the video. The model is trained to identify key actions associated with specific emotions.

[0051] The state generating unit 350 is used to generate the current emotion of the passenger in the vehicle based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability.

[0052] In this embodiment, the state generation unit combines the first, second and third emotion recognition probabilities and uses a fusion algorithm (such as weighted average, Bayesian fusion, etc.) to generate the current emotional state of the passenger in the vehicle. The current emotional state can be a probability distribution, indicating the possibility of different emotion types; or the emotion type with the maximum probability in the probability distribution is determined as the current emotional state of the passenger in the vehicle.

[0053] For example, the state generation unit may use a fusion algorithm to integrate emotion recognition results from different data sources, and perform weighted processing on the fusion results by considering the reliability and relevance of the different data sources.

[0054] The emotional interaction system in this embodiment integrates multiple technologies such as time synchronization, facial recognition, voice recognition, and behavior recognition to achieve comprehensive analysis and recognition of the emotions of passengers in the car, thereby improving the accuracy of identifying the emotions of passengers in the car. In addition, when the interactive information in the car is adjusted based on the current emotions of the passengers in the car determined by the emotional interaction system, the accuracy of the interactive information provided by the vehicle (such as emotional response, safety warnings, etc.) can be improved, thereby improving the vehicle's intelligent and humanized service capabilities.

[0055] In some optional implementations of this embodiment, the driver monitoring system is further used to: determine the dynamic information of facial expressions based on the image frames in the first video stream; determine the driver's emotion type when the cumulative value of the dynamic information of facial expressions exceeds the expression threshold; and determine whether the driver's negative expression is recognized based on the driver's emotion type.

[0056] In this implementation, the dynamic information may include: smiling, frowning, drooping eyes, drooping mouth corners, tightly closed lips, etc. The first cumulative value may include: duration and / or cumulative number of times. The dynamic information threshold may be set according to actual application scenarios or experience, for example, according to age, music, road conditions, etc.

[0057] The driver monitoring system in this implementation method determines the driver's emotion type when the first cumulative value of the dynamic information of the expression exceeds the dynamic information threshold, and then determines whether the driver's negative expression is recognized based on the driver's emotion type, thereby reducing the erroneous recognition of negative expressions due to misjudgment of occasional or single expressions, and improving the accuracy and timeliness of identifying negative expressions.

[0058] In some optional implementations of this embodiment, the in-vehicle passenger monitoring system is further used to: determine the motion amplitude of the behavior information of the in-vehicle passenger based on the collected image frames in the second video stream; generate the emotion type of the in-vehicle passenger when the motion amplitude of the behavior information meets the motion amplitude threshold interval; the motion amplitude of the behavior information includes at least one of the following: body tilt angle, elbow angle, chin angle, gesture change rate, body movement frequency; based on the emotion type of the in-vehicle passenger, determine whether the negative body language of the in-vehicle passenger is recognized.

[0059] In this implementation, the action amplitude of the behavior information includes at least one of the following: body tilt angle, elbow angle, chin angle, gesture change rate, body movement frequency, etc. The action amplitude threshold interval can indicate the emotion type corresponding to the recognized body language, and can be adjusted according to age, music, road conditions, etc.

[0060] The in-vehicle passenger monitoring system in the present implementation determines the motion amplitude of the behavior information of the in-vehicle passenger based on the image frames in the second video stream, and generates the emotion type of the in-vehicle passenger when the motion amplitude of the behavior information meets the motion amplitude threshold range, and determines whether the negative body language of the in-vehicle passenger is recognized based on the emotion type of the in-vehicle passenger. This process determines the corresponding emotion type based on the motion amplitude of the behavior information and the motion amplitude threshold range, and then determines whether the negative body language of the in-vehicle passenger is recognized, which reduces the erroneous recognition of negative body language due to misjudgment of occasional or single behavior, and improves the accuracy and timeliness of the recognized negative body language.

[0061] In some optional implementations of this embodiment, the speech system is further used to: determine the prosodic feature information of the speech based on the collected speech signal; generate the emotion type corresponding to the speech signal when the ratio of the prosodic feature information to the prosodic threshold meets the target prosodic interval; and determine whether negative language is recognized based on the emotion type corresponding to the speech signal.

[0062] In this implementation, the prosody feature information includes: speech rate and / or volume. The prosody threshold is a threshold of speech rate and / or volume, which can be set according to actual application scenarios or experience, such as adjusting according to age, music, road conditions, etc.

[0063] The in-vehicle passenger monitoring system in this implementation method extracts the rhythmic feature information of the speech based on the collected speech signal, and calculates the ratio of the rhythmic feature information to the rhythmic threshold. When the ratio meets the target rhythmic interval, it generates the emotion type corresponding to the speech signal, and then determines whether negative language is recognized, thereby reducing the incorrect recognition of negative language due to misjudgment and improving the accuracy and timeliness of the recognized negative language.

[0064] In some optional implementations of the present embodiment, the state generation unit is further used to: determine the intensity of the emotion type based on the first emotion recognition probability, the second emotion recognition probability and the third emotion recognition probability; determine the normalization factor of the scores of all emotion types based on the first emotion recognition probability, the second emotion recognition probability and the third emotion recognition probability; take the ratio of the intensity of the emotion type to the normalization factor of the scores of all emotion types as the score of the emotion type; and determine the current emotion of the passengers in the vehicle based on the score of the emotion type.

[0065] In this implementation, the state generation unit determines the intensity of the emotion type based on the first emotion recognition probability, the second emotion recognition probability and the third emotion recognition probability generated by multi-source data, thereby improving the accuracy of the intensity of the determined emotion type; determines the normalization factor of the scores of all emotion types based on the first emotion recognition probability, the second emotion recognition probability and the third emotion recognition probability, thereby improving the accuracy of the normalization factor of the scores of all emotion types; then takes the ratio of the intensity of the emotion type to the normalization factor of the scores of all emotion types as the score of the emotion type, thereby realizing the quantification of the scores of the emotion type; finally, based on the scores of the emotion types, determines the current emotions of the passengers in the car, thereby improving the accuracy and efficiency of determining the current emotions of the passengers in the car.

[0066] As used above, the term "module" may implement a combination of software and / or hardware to implement a predetermined function. Although the apparatus described in the above embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0067] According to an embodiment of the present disclosure, an embodiment of an in-vehicle interaction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here. The method is used in the embodiments and preferred implementations of the above-mentioned in-vehicle interaction system, and those that have been described will not be repeated.

[0068] Please refer to Figure 4 , Figure 4 A schematic flow chart of an in-vehicle interaction method according to an embodiment of the present disclosure is shown, and the method includes:

[0069] Step S401, using the driver monitoring system, when a negative expression of the driver is recognized based on the collected first video stream, an emotion recognition request is sent to the in-vehicle passenger detection system, and the first video stream is sent to the emotion interaction system;

[0070] Step S402, using the in-vehicle passenger monitoring system, upon receiving an emotion recognition request and recognizing negative body language of the in-vehicle passenger based on the collected second video stream, sending an emotion detection request to the voice system and sending the second video stream to the emotion interaction system;

[0071] Step S403, using the voice system, upon receiving an emotion detection request and recognizing negative language based on the collected voice signal, sending a voice signal to an emotion recognition module;

[0072] Step S404, using the emotion interaction system, based on the first video stream, the second video stream, and the voice signal, determine the current emotion of the passengers in the car, and adjust the interaction information in the car based on the current emotion.

[0073] In some optional implementations of the present embodiment, an emotional interaction system is used to determine the current emotions of passengers in the car based on the first video stream, the second video stream, and the voice signal, and based on the current emotions, the interactive information in the car is adjusted, including: using a facial recognition model to extract facial expression features from the image frames in the first video stream and the image frames in the second video stream collected at the target moment, respectively, to obtain the first facial expression features and the second facial expression features, and based on the first facial expression features and the second facial expression features, generating a first emotion recognition probability, the first emotion recognition probability indicating the probability that the emotion recognition result generated based on the first facial expression features and the second facial expression features is each emotion type in the emotion type set; using a speech analysis algorithm to extract facial expression features from the speech collected at the target moment, The method comprises the following steps: using a sound signal, extracting speech emotion features, and generating a second emotion recognition probability based on the speech emotion features; the second emotion recognition probability indicates the probability that the emotion recognition result generated based on the speech emotion features is each emotion type in the emotion type set; utilizing a behavior analysis algorithm, for image frames in a second video stream collected at a target moment, identifying key emotion actions, and generating a third emotion recognition probability based on the key emotion actions; the third emotion recognition probability indicates the probability that the emotion recognition result generated based on the key emotion actions is each emotion type in the emotion type set; generating an emotional state at a target moment based on the first emotion recognition probability, the second emotion recognition probability, and determining the current emotion of the passengers in the vehicle based on the emotional states at multiple target moments within a preset time period.

[0074] In some optional implementations of the present embodiment, a first emotion recognition probability is generated based on the first facial expression feature and the second facial expression feature, including: determining the dynamic information of the expression based on the first facial expression feature and the second facial expression feature; generating the first emotion recognition probability when the cumulative value of the dynamic information of the expression exceeds the dynamic information threshold; the dynamic information includes: intensity and / or duration; the cumulative value includes: duration or cumulative number of times.

[0075] In some optional implementations of the present embodiment, a second emotion recognition probability is generated based on the speech emotion characteristics, including: determining the prosodic feature information of the speech based on the speech emotion characteristics; generating a second emotion recognition probability when the ratio of the prosodic feature information to the prosodic threshold meets the target prosodic interval; the prosodic feature information includes: speaking rate and / or volume.

[0076] In some optional implementations of the present embodiment, a third emotion recognition probability is generated based on the key emotional action, including: determining the motion amplitude of the behavior information based on the key emotional action; generating a third emotion recognition probability when the motion amplitude of the behavior information meets the motion amplitude threshold interval; the motion amplitude of the behavior information includes at least one of the following: body tilt angle, elbow angle, chin angle, gesture change rate, body movement frequency.

[0077] In some optional implementations of the present embodiment, the current emotion of the passengers in the vehicle is generated based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability, including: determining the intensity of the emotion type based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability; determining the normalization factor of the scores of all emotion types based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability; taking the ratio of the intensity of the emotion type to the normalization factor of the scores of all emotion types as the score of the emotion type; and determining the current emotion of the passengers in the vehicle based on the score of the emotion type.

[0078] In some optional implementations of this embodiment, the interactive information in the car is adjusted based on the current emotion, including: using a personalization engine to generate an in-car environment adjustment strategy according to the current emotion of the passengers in the car, the historical emotional reactions and preferences of the passengers in the car, and adjusting the interactive information in the car according to the in-car environment adjustment strategy.

[0079] As a specific application example of the embodiment of the present disclosure, the optimal implementation scheme or the scheme that the inventor most wants to embody is described in combination with a specific application scenario.

[0080] The present disclosure relates to the field of cockpit emotional interaction and involves multiple modules, which mainly introduces the functional logic of human-vehicle emotional interaction and the control method during the execution process.

[0081] The logical implementation of the emotional interaction in the cabin is divided into the sensor module, SOC, and interactive execution module. The sensor module includes the cabin camera and voice MIC; the SOC domain controller is mainly responsible for algorithm and signal processing; the interactive execution module refers to the cabin environment, scene mode, atmosphere light, and AI voice. Figure 1 shown.

[0082] The camera driver monitoring system DMS, the in-vehicle passenger monitoring system OMS and the voice microphone MIC in the sensor module work independently. The domain controller SOC can process the signal only when both of them meet the judgment logic at the same time. The specific logical relationship will be introduced in detail later.

[0083] The SOC domain controller mainly includes a large model for emotion recognition. The signal processing on the camera side includes but is not limited to time synchronization, feature extraction, image encoding, and key action triggering. The processing of the signal collected by the voice microphone MIC includes but is not limited to signal amplification, beamforming, voice encoding, and keyword triggering. After the SOC processes the signal, it will make relevant logical judgments on the display screen. The specific logical relationship will be described in detail later.

[0084] The interactive execution module refers to the realization of emotional interaction functions, including air conditioning, scene mode, ambient light, and AI voice. The four different functional effects will be introduced in detail later.

[0085] The working logic of this solution can be referenced Figure 5 , Figure 5 It shows that Figure 1 The exemplary working logic of the in-vehicle interactive system shown in the figure. Specifically, the logical content corresponding to the working logic is as follows:

[0086] In the cockpit, the camera driver monitoring system DMS and the in-car passenger monitoring system OMS monitor the driver or passenger's negative expressions in real time, including frowning, tightly closed lips, drooping mouth corners, tears, etc. It should be noted that facial expressions are not always accurate indicators of emotions, so a judgment logic is added at this node. If negative emotions are detected, the voice MIC is activated for further detection and judgment; if no negative emotions are detected, the system does not activate the emotional interaction function or the driver manually adjusts the emotional interaction function.

[0087] The driver monitoring system DMS independently identifies the emotional state by analyzing the driver's facial expressions, such as eye opening and closing and changes in the corners of the mouth. The passenger monitoring system OMS also monitors the passenger's facial and body language, extracting key emotional features, such as large waving and hard tapping. When the driver's monitoring system DMS cabin camera detects the driver's potential negative emotions, it activates the voice MIC to further analyze the tone and rhythm in the voice to confirm the emotional state. At the same time, the passenger monitoring system OMS provides the passenger's emotional information at the request of the driver monitoring system DMS to assist in comprehensive judgment. The three systems work not only independently, but also in collaboration through an integrated emotion recognition model, the algorithm of which is based on the collected information of the three to ensure the accuracy of emotion recognition. Once the system confirms the emotional state through comprehensive analysis, whether it is anxiety, fatigue or other emotions, it can automatically adjust the in-car environment, such as adjusting the color of the ambient light, playing soothing music or adjusting the air conditioning temperature, to create a more comfortable and pleasant riding experience.

[0088] In order to make the sensor module recognize accurately, it does not rely solely on the in-cabin camera to make judgments, but adds voice microphone MIC collection. The driver monitoring system uses the in-car camera to capture the facial expressions of the driver or passenger in real time and analyzes these expressions through a deep learning model. The model is trained to recognize specific negative expressions, such as frowning or drooping corners of the mouth. The driver monitoring system sets a specific threshold based on the intensity and duration of the expression to distinguish between occasional changes in expression and true negative emotions. For example, the driver monitoring system may be programmed to consider this as an emotional signal that requires further analysis only when the intensity of the negative expression exceeds a predetermined level and lasts for more than a few seconds. Once these conditions are met, the driver monitoring system automatically activates the voice microphone (MIC) and begins to collect and analyze voice signals to more fully understand the passenger's emotional state. This threshold-based approach ensures that the system does not respond to short-term or non-critical changes in expression, thereby improving the accuracy of emotion recognition and the efficiency of the system. The signal source for waking up the voice MIC is the negative expression detected by the camera. If negative language is detected, the signal is transmitted to the domain controller SOC for processing; if no negative language is detected, no processing is performed, and the camera continues to monitor. It should be noted that if the voice judgment logic is always no, and the cycle after returning to camera monitoring is repeated 3 times, the system function will automatically choose not to start or the driver will manually adjust the emotional interaction function. The emotional interaction function here is the same as the preset emotional mode of the car computer, and you can choose happy, sad, angry and other interactive effects (only for example, other interactive effects can also be selected).

[0089] The cockpit domain controller SOC integrates a large model of emotional interaction. The camera signal processing includes time synchronization, feature extraction, image encoding, key action triggering, etc.; the voice MIC signal processing includes signal amplification, beamforming, voice encoding, keyword triggering, etc.

[0090] The emotion recognition model achieves accurate capture and analysis of the emotional state of the driver and passengers through the designed algorithms and processing steps. First, the system uses time synchronization technology to embed high-precision timestamps in the image and voice data frames to ensure the temporal consistency of the data from the camera and microphone. Then, a deep convolutional neural network (CNN) is used on the image data to extract facial key points and expression features, such as subtle changes in the corners of the eyes and mouth, while on the voice signal, speech analysis techniques such as Mel Frequency Cepstral Coefficients (MFCC) are used to extract emotional features such as speaking speed, pitch and intensity. Through these features, the model adopts a multimodal fusion strategy, combining facial expressions and voice features to make a comprehensive judgment of emotional states, such as happiness, sadness, anger, etc.

[0091] In addition, the model integrates algorithms based on behavioral analysis, which can identify key actions, such as large waving and strong tapping, which may be related to emotional states. In the data processing process, machine learning algorithms such as support vector machine (SVM), random forest and neural network are used to process and analyze the extracted features. SVM classifies by finding a hyperplane that maximizes the boundary between categories, and can handle nonlinear data through kernel techniques. Random forest improves the accuracy and robustness of the model by building multiple decision trees and integrating their prediction results, while also providing an assessment of feature importance. Neural networks automatically learn complex representations of data by simulating the structure of the human brain, and handle nonlinear and high-dimensional data. An optimized process from data preprocessing to feature extraction to emotion classification is implemented. Key technical points include real-time data analysis, adaptive learning algorithms and context-aware capabilities, among which real-time data analysis uses stream processing frameworks (such as Apache Kafka and Apache Flink) to process real-time data streams. Time series databases (such as InfluxDB) are used to store real-time data for fast query and analysis. The adaptive learning algorithm uses online learning algorithms, such as online gradient descent, to enable the model to process continuously incoming data and dynamically update model parameters. The reinforcement learning algorithm is used to enable the emotional interaction model to learn the optimal strategy in the interaction with the environment. The context-aware capability recognizes and understands different environmental states and user behaviors by building context models. Natural language processing (NLP) technology is used to parse and understand contextual information in text data. Sensor data and other external data sources are integrated to provide more comprehensive contextual information. The emotional interaction model can adaptively adjust its parameters according to the current driving environment and historical interaction data to achieve more accurate emotion recognition.

[0092] Finally, the model also includes a personalization engine, which customizes the in-car environment adjustment strategy based on the passenger's historical emotional response and preferences, such as adjusting the color of the ambient light, music playback and air conditioning temperature, to provide a personalized and comfortable riding experience. At the same time, the system can also communicate with passengers in natural language through the AI ​​voice assistant, actively providing help or emotional comfort.

[0093] The multimodal fusion model combines facial expression recognition (FER), speech analysis (SAM) and body language analysis (BLAA) to capture the driver's emotional changes. The output of each module is transformed through a weighted exponential function to emphasize its probability of the final emotional judgment. The basic types of emotions are proposed to be happiness, sadness, and anger. The weights w1, w2, and w3 correspond to these three modules respectively, and are optimized through historical data to reflect the accuracy and reliability of each module in emotion recognition. The softmax function is used to normalize these weighted index scores to obtain a valid probability distribution, indicating the possibility that the current emotional state belongs to each predefined emotional category. The formula is as follows:

[0094]

[0095] Among them, E i represents the probability of emotion i, i can be happiness, sadness or anger, E i,FER ,E i,SAM ,E i,OMS They are the scores of emotion i from the facial expression, voice and body language analysis modules. So in layman's terms, the denominator is the normalization factor of all emotion scores, and the numerator represents the intensity of a specific emotion i.

[0096] The system also sets specific thresholds based on the intensity and duration of the expression to distinguish between occasional changes in expression and true negative emotions. For example, the system may be programmed to consider it an emotional signal that requires further analysis only when the intensity of a negative expression exceeds a certain predetermined level and lasts for more than a few seconds.

[0097] Hardware configurations, such as camera resolution, microphone sensitivity, and the number of sensors, directly affect the quality of data and the accuracy of emotion recognition. A high-resolution camera can capture more subtle changes in facial expressions, while a high-sensitivity microphone can capture voice signals more accurately.

[0098] The setting of the threshold is critical to the performance of the system. If the threshold is set too low, the system may respond to short-term or non-critical changes in expression, resulting in false positives; if the threshold is set too high, the system may miss real emotional changes, resulting in false negatives. Therefore, the setting of the threshold needs to find a balance between false positives and false negatives.

[0099] Through this integrated emotion recognition model, the system can more accurately identify the driver's emotional state and automatically adjust the in-car environment accordingly, such as adjusting the color of the ambient light, playing soothing music or adjusting the air conditioning temperature, to create a more comfortable and pleasant riding experience. This highly integrated control mechanism not only enhances the personalized experience, but also makes driving more intelligent and humane.

[0100] According to the different types of emotions, the classification and setting examples of the emotion analysis results of the three modules are as follows:

[0101]

[0102]

[0103] However, in the actual car-use environment, different factors will affect the judgment of emotion recognition. For example, people of different age groups have different ways of expressing emotions. For children, their emotions are expressed more directly and frequently, so the threshold can be lowered. For the elderly, their emotions are expressed more steadily, so the threshold can be appropriately increased. Different music rhythms and volumes, as well as external noise, may affect emotions. High-volume or fast-paced music may require a higher threshold to distinguish between music and emotional signals. Different driving routes may cause different emotional expressions such as anxiety. For unfamiliar routes, the threshold can be lowered to ensure that the driver can focus on identifying road conditions.

[0104] The interactive logic of its function will first be judged on the display screen, and a pop-up window will be displayed as a prompt. The options of the pop-up window include "Clear" and "Confirm". "Clear" means not starting the emotional interaction function, and the program will be closed; "Confirm" means starting the emotional interaction function, which will send a signal to the interactive execution module to make the next control. It should be noted that when different emotions are generated, the current pop-up information may be ignored, so if no operation is performed within 3 seconds after the pop-up information appears, the function logic will be judged to start automatically and the next operation will be performed.

[0105] The interactive execution module is responsible for the implementation of emotional interaction functions. This solution makes some settings for air conditioning, scene modes, ambient lights and AI voice to achieve the logic of interaction.

[0106] When emotions fluctuate, the air conditioning settings can be adjusted to improve the cabin environment, thereby helping to ease emotions. First, the cabin temperature can be adjusted to a comfortable level according to the emotional state. For example, if you feel anxious or nervous and angry, the air conditioning system will automatically adjust the temperature to a cooler level to help passengers calm down; conversely, if low mood or depression and sadness are identified, the air conditioning will increase the temperature to increase the warmth of the cabin. At the same time, the air conditioning system will maintain the humidity in the cabin between 40%-60% to ensure that the passenger's skin feels comfortable and avoid the air being too dry or humid. And the wind speed and direction of the air conditioning will also be fine-tuned according to the emotional state to avoid blowing directly on the passengers and reduce discomfort. In addition, the air conditioning can improve air circulation, reduce odors, and provide fresh air to better solve the emotional problems of passengers.

[0107] As a smart cockpit product, the atmosphere light can adjust the cabin environment in a variety of ways to relieve emotions. RGBLED lamp beads are used to adjust the color and brightness. The PWM frequency is usually set high enough to avoid the human eye from noticing the flicker, providing a comfortable visual experience by controlling the ratio of red, green and blue and the PWM (pulse width modulation) duty cycle. The atmosphere light can also achieve music rhythm control, changing the brightness and color according to the rhythm and emotion of the music, and can even be combined with the ambient temperature, the driver's emotional state, etc., and automatically adjusted through the intelligent system to adapt to different situations and needs. For example, for anxious and angry emotions, the atmosphere light will be adjusted to a soft blue-green tone, and the brightness will be set to a low-medium level to create a calm and relaxing atmosphere; for sad emotions, the atmosphere light will be adjusted to a soft blue tone, and the brightness will be set to a low level to help relieve sad emotions; and for excited or happy emotions, warm red or yellow tones will be selected, and the brightness will be increased accordingly to enhance the sense of vitality. The ambient light also has an ambient light sensing function, which can automatically adjust the brightness according to the external light intensity to ensure a comfortable visual experience under any lighting conditions, and the brightness of the ambient light can be adjusted by the user. Specific examples are as follows:

[0108]

[0109] In terms of adjustment logic, the ambient light is integrated with the music playback system, and can change the lighting effect in real time according to the rhythm and emotional color of the music, such as increasing the brightness and color change frequency in fast-paced music and reducing the change in slow-paced music to enhance the emotional expression of the music. In addition, the system also allows users to preset the adjustment parameters of the ambient light according to their personal preferences, such as color preference and brightness level. These settings will be memorized by the system and automatically applied after emotion recognition to provide a more personalized lighting experience.

[0110] The role of the scenario mode is to provide an immersive experience to relieve emotions. It has different performance effects according to the emotional state of the passengers. It can be combined with deep breathing, meditation or progressive muscle relaxation techniques to enhance the relaxation effect. In addition, it can also be combined with soft music to achieve a relaxing immersive experience.

[0111] When starting the interactive execution module, the AI ​​voice does not need a wake-up word. It only needs to receive the emotional interaction signal from the SOC to wake up, and actively ask about the passenger's situation and cooperate with the passenger's needs to relieve the pressure of negative emotions. The response mechanism of the AI ​​voice assistant includes the existing technology of real-time monitoring of passengers' answers and instructions, and then quickly analyzes and executes the corresponding in-car environment adjustments. For example, if the passenger agrees to play music, the AI ​​assistant will immediately select and play from the preset relaxation tracks. In addition, the AI ​​assistant will also intelligently adjust other systems in the car according to the needs of the passengers, such as adjusting the air-conditioning temperature, changing the color of the ambient light, and even providing navigation to the nearest rest area when necessary.

[0112] In order to provide more personalized services, the AI ​​voice assistant has increased the ability to learn and adapt, and will continuously optimize its response strategy based on passenger feedback and historical interaction data. In an emergency, if extremely low emotions or potentially dangerous behaviors are identified, the AI ​​assistant will immediately initiate an emergency response procedure, including automatically contacting emergency services or issuing safety warnings to passengers or turning on the double flash lights to alert other vehicles and pedestrians. Throughout the process, the AI ​​assistant always pays attention to the privacy and data security of passengers, ensuring that all interactions are conducted within a strict privacy protection framework. The framework implements a data minimization strategy, only collects data necessary to implement functions, and protects data security through anonymization and advanced encryption technology. (This technology only has clear instructions for the logic and control of emotional interactions, and the implementation strategy of the privacy framework is not detailed here).

[0113] The above introduces the working logic method of emotional interaction, that is, the role of each module. The realization of functional interaction is not a single control. The following content introduces the linkage control method of each function of the interactive execution module: the air conditioning, AI voice, atmosphere light and scene mode inside the car are intelligently linked through a central control system. When the driver triggers the scene mode of emotional interaction, the central control system will automatically adjust the air conditioning to a suitable temperature, adjust the atmosphere light to a soft color, and change the display theme to create a relaxing atmosphere according to the preset instructions. The AI ​​voice assistant allows the driver to control these functions through voice commands without manual operation, which improves the convenience and safety of driving. In addition, sensors inside the vehicle monitor environmental conditions such as temperature and humidity, and feed back data to the central control system so that the system can automatically adjust the settings to ensure the comfort of the driving experience. Under the highly integrated control mechanism, not only the personalized experience is improved, but also driving is more intelligent and humane.

[0114] Advantages of the present disclosure include but are not limited to:

[0115] 1. Personalized experience: The system monitors the driver or passenger’s facial expressions and voice, identifies their emotional state, and provides a personalized interactive experience. This personalized service can meet consumers’ expectations for emotional interactive experience.

[0116] 2. Emotion recognition and response: Using cameras and voice microphones to capture the driver's facial expressions and language, the system can recognize negative emotions and respond accordingly. This helps to improve the driving experience while preventing driving risks caused by emotional fluctuations.

[0117] 3. Environmental adaptability: The system can adjust the cabin environment according to the driver's emotional state, such as temperature, humidity, ambient light color, etc., to help the driver better adapt to different driving environments.

[0118] 4. Health promotion: By improving the cockpit environment, the system helps alleviate the driver's emotional fluctuations and improve physical and mental health.

[0119] 5. Intelligent linkage control: The system realizes intelligent linkage of air conditioning, AI voice, ambient light and scenario mode through the central control system, which improves the convenience and safety of driving.

[0120] 6. No manual operation required: AI voice assistant allows the driver to control the functions in the cockpit through voice commands, reducing the driver's manual operation during driving, thereby improving driving safety.

[0121] 7. Automatic adjustment: Sensors inside the vehicle can monitor environmental conditions and feed data back to the central control system, which can automatically adjust settings to ensure a comfortable driving experience.

[0122] 8. Immersive experience: The scenario mode provides an immersive experience, which, combined with music and environmental adjustments, helps passengers relax and relieve stress.

[0123] 9. Technology Integration: The highly integrated control mechanism not only enhances the personalized experience, but also makes driving more intelligent and humane, reflecting the transformation of the automotive industry from traditional manufacturing to providing comprehensive travel solutions.

[0124] 10. Preventive measures: When the system detects negative emotions, it will prompt the driver to make a choice through a pop-up window. If the driver does not make a choice within the specified time, the system will automatically start the emotional interaction function. This preventive measure helps to respond to the driver's emotional changes in a timely manner.

[0125] In summary, the present disclosure provides an intelligent car cockpit system that can understand, respond to and predict the driver's emotional state through emotional interaction technology, aiming to improve the driving experience, enhance safety, and promote the driver's physical and mental health.

[0126] The technical innovations of this disclosure include but are not limited to:

[0127] 1. Multimodal emotion recognition: The system combines facial expression recognition and speech analysis to identify the driver’s emotional state, improving the accuracy and reliability of emotion recognition.

[0128] 2. Real-time monitoring and feedback: By real-time monitoring of the driver or passenger’s facial expressions and voice, the system can respond quickly to emotional changes and provide corresponding interactive feedback in a timely manner.

[0129] 3. Intelligent environment adjustment control logic: The system can intelligently adjust the temperature, humidity, ambient light color, etc. in the cabin according to the driver's emotional state to improve the driver's experience and mood.

[0130] 4. Intelligent linkage of the central control system: Through the central control system, intelligent linkage of air conditioning, AI voice, ambient lighting and scenario modes is realized, which improves the convenience and safety of driving.

[0131] 5. AI voice assistant that does not require waking up: The AI ​​voice assistant can automatically wake up after receiving emotional interaction signals, without the need to use wake-up words, thus simplifying the interaction process.

[0132] 6. User interface interaction logic: The system provides interaction logic on the display screen, prompting users to make choices through pop-up windows, increasing the flexibility of user control.

[0133] 7. Preventive self-start mechanism: When the user fails to respond to the pop-up prompt in time, the system can automatically start the emotional interaction function to ensure that timely help can be provided when the driver's emotions fluctuate.

[0134] 8. Emergency response procedure: Combined with the AI ​​voice model, the execution of the emergency response procedure is judged to ensure the negative impact of the driver's excessive emotions and ensure the warning effect on driving or the surrounding environment.

[0135] See also Figure 6 , Figure 61 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present disclosure, and the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0136] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0138] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage unit that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0141] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0142] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A vehicle-mounted interactive system, characterized in that: The vehicle-mounted interactive system comprises: The driver monitoring system is configured to send an emotion recognition request to the in-vehicle passenger detection system and send the first video stream to the emotion interaction system when a negative expression of the driver is recognized based on the collected first video stream; The in-vehicle passenger monitoring system is configured to, upon receiving the emotion recognition request and recognizing negative body language of the in-vehicle passenger based on the collected second video stream, send an emotion detection request to the voice system and send the second video stream to the emotion interaction system; The voice system is configured to send the voice signal to the emotion interaction system when receiving the emotion detection request and identifying negative language based on the collected voice signal; The emotional interaction system is used to determine the current emotion of the passengers in the car based on the first video stream, the second video stream, and the voice signal, and to adjust the interactive information in the car based on the current emotion.

2. The in-vehicle interactive system according to claim 1, characterized in that: The emotional interaction system comprises: A time synchronization unit, configured to align the first video stream, the second video stream, and the voice signal using a time synchronization technology; A facial recognition unit, configured to extract facial expression features from the aligned image frames in the first video stream and the aligned image frames in the second video stream using a facial recognition model, respectively, to obtain a first facial expression feature and a second facial expression feature, and to generate a first emotion recognition probability based on the first facial expression feature and the second facial expression feature, wherein the first emotion recognition probability indicates a probability that an emotion recognition result generated based on the first facial expression feature and the second facial expression feature is a respective emotion type in an emotion type set; A speech recognition unit, configured to extract speech emotion features from the aligned speech signal using a speech analysis algorithm, and generate a second emotion recognition probability based on the speech emotion features; the second emotion recognition probability indicates the probability that the emotion recognition result generated based on the speech emotion features is each emotion type in the emotion type set; a behavior recognition unit, configured to use a behavior analysis algorithm to recognize key emotional actions for the image frames in the aligned second video stream, and to generate a third emotion recognition probability based on the key emotional actions; the third emotion recognition probability indicates the probability that the emotion recognition result generated based on the key emotional action is each emotion type in the emotion type set; The state generating unit is used to generate the current emotion of the passenger in the vehicle based on the first emotion recognition probability, the second emotion recognition probability and the third emotion recognition probability.

3. The in-vehicle interactive system according to claim 2, characterized in that: The driver monitoring system is further configured to: Determining dynamic information of the expression based on the image frames in the first video stream; When the accumulated value of the dynamic information of the expression exceeds the expression threshold, determining the emotion type of the driver; The cumulative value includes: duration and / or cumulative number of times; Based on the emotion type of the driver, it is determined whether a negative expression of the driver is recognized.

4. The in-vehicle interactive system according to claim 2, characterized in that: The in-vehicle passenger monitoring system is further used for: Determine the movement amplitude of the behavior information of the passenger in the vehicle based on the image frames in the collected second video stream; When the action amplitude of the behavior information meets the action amplitude threshold range, the emotion type of the passenger in the vehicle is generated; the action amplitude of the behavior information includes at least one of the following: body tilt angle, elbow angle, chin angle, gesture change rate, body movement frequency; Based on the emotion type of the vehicle occupant, it is determined whether negative body language of the vehicle occupant is recognized.

5. The in-vehicle interactive system according to claim 2, characterized in that: The speech system is further used for: Determine the prosodic feature information of the speech based on the collected speech signal; When the ratio of the prosody feature information to the prosody threshold meets the target prosody interval, an emotion type corresponding to the speech signal is generated; the prosody feature information includes: speech speed and / or volume; Based on the emotion type corresponding to the voice signal, it is determined whether negative language is recognized.

6. The in-vehicle interactive system according to claim 2, characterized in that: The state generating unit is further used for: determining the intensity of the emotion type based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability; determining a normalization factor for scores of all emotion types based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability; The ratio of the intensity of the emotion type to the normalization factor of the scores of all emotion types is used as the score of the emotion type; Based on the scores of the emotion types, current emotions of the passengers in the vehicle are determined.

7. The in-vehicle interactive system according to claim 1, characterized in that: The emotional interaction system also includes: The personalization engine is used to generate an in-vehicle environment adjustment strategy based on the current emotions of the in-vehicle passengers, the historical emotional reactions and preferences of the in-vehicle passengers, and to adjust the in-vehicle interactive information based on the in-vehicle environment adjustment strategy.

8. A vehicle-mounted interaction method, characterized in that: The vehicle-mounted interaction method comprises: Using the driver monitoring system, when a negative expression of the driver is recognized based on the collected first video stream, an emotion recognition request is sent to the in-vehicle passenger detection system, and the first video stream is sent to the emotion interaction system; Using the in-vehicle passenger monitoring system, upon receiving the emotion recognition request and recognizing negative body language of the in-vehicle passenger based on the collected second video stream, sending an emotion detection request to the voice system and sending the second video stream to the emotion interaction system; Using the voice system, upon receiving the emotion detection request and recognizing negative language based on the collected voice signal, sending the voice signal to the emotion interaction system; The emotional interaction system is used to determine the current emotions of passengers in the vehicle based on the first video stream, the second video stream, and the voice signal, and to adjust the interactive information in the vehicle based on the current emotions.

9. The vehicle-mounted interaction method according to claim 8, characterized in that: The method of using the emotion interaction system to determine the current emotion of the passenger in the vehicle based on the first video stream, the second video stream, and the voice signal, and adjusting the interaction information in the vehicle based on the current emotion, includes: Used to align the first video stream, the second video stream and the voice signal by using a time synchronization technology; Utilizing a facial recognition model, extracting facial expression features from the aligned image frames in the first video stream and the aligned image frames in the second video stream, respectively, to obtain a first facial expression feature and a second facial expression feature, and generating a first emotion recognition probability based on the first facial expression feature and the second facial expression feature, wherein the first emotion recognition probability indicates a probability that an emotion recognition result generated based on the first facial expression feature and the second facial expression feature is a respective emotion type in an emotion type set; Utilizing a speech analysis algorithm, for the aligned speech signal, extracting speech emotion features, and generating a second emotion recognition probability based on the speech emotion features; the second emotion recognition probability indicates the probability that the emotion recognition result generated based on the speech emotion features is each emotion type in the emotion type set; Using a behavior analysis algorithm, for the aligned image frames in the second video stream, key emotional actions are identified, and based on the key emotional actions, a third emotion recognition probability is generated; the third emotion recognition probability indicates the probability that the emotion recognition result generated based on the key emotional action is each emotion type in the emotion type set; Based on the first emotion recognition probability, the second emotion recognition probability, and the third emotion recognition probability, the current emotion of the passenger in the vehicle is generated.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the in-vehicle interaction method according to any one of claims 8 to 9.

Images