Vehicle intelligent control method and system based on multi-modal emotion recognition and vehicle

Abstract

The invention relates to the field of vehicle personalized services, in particular to a vehicle intelligent control method and device based on multi-modal emotion recognition and a vehicle, and the method comprises the steps that multi-modal data related to the emotion of a user is acquired, and the multi-modal data comprises physiological data of the user, user behavior interaction data and environmental perception data around the vehicle; inputting the data acquired by the multi-source sensor into a pre-trained emotion calculation model for multi-modal information fusion so as to comprehensively output the current emotion state of the user; generating a control instruction according to the current emotional state of the user; and according to the control instruction, at least one functional subsystem of the vehicle is adjusted in a self-adaptive mode. The user emotion is identified through multi-source data, and then the vehicle function can be actively adjusted according to the current mood of the user.

Description

Technical Field

[0001] This application relates to the field of personalized vehicle services, specifically a vehicle intelligent control method, system, and vehicle based on multimodal emotion recognition. Background Technology

[0002] With continuous technological advancements, new energy vehicles are trending towards informatization, connectivity, and intelligence. Vehicle functions need to cover all aspects of passenger needs, including but not limited to driver assistance, voice control, entertainment, and remote control, providing users with a more comfortable and user-friendly driving experience. Vehicles are no longer just a means of transportation, but also intelligent assistants for users on their journeys. Current technology lacks a way to optimally adjust vehicle functions based on the user's current emotional state. Summary of the Invention

[0003] This invention provides a vehicle intelligent control method, system, and vehicle based on multimodal emotion recognition, which aims to proactively adjust vehicle functions according to the user's current mood by recognizing the user's emotions.

[0004] The technical solution of this invention is as follows:

[0005] Firstly, this application provides a vehicle intelligent control method based on multimodal emotion recognition, including:

[0006] Acquire multimodal data related to user emotions, including user physiological data, user behavior interaction data, and environmental perception data around the vehicle;

[0007] The data collected by the multi-source sensors are input into a pre-trained emotion computing model for multimodal information fusion to comprehensively output the user's current emotional state;

[0008] Generate control commands based on the user's current emotional state;

[0009] Based on the control commands, at least one functional subsystem of the vehicle is adaptively adjusted.

[0010] By comprehensively utilizing user physiological data, behavioral interaction data, and environmental perception data around the vehicle for integrated judgment, it overcomes the potential misjudgments and limitations of single data sources, making the identification results of user emotional states more comprehensive and reliable. Based on the identified user's current emotional state, it automatically generates control commands and drives the vehicle's functional subsystems to adjust, enabling the vehicle to transform from a passive tool into an intelligent system that can actively understand and respond to the driver's state, thereby significantly improving the intelligence level of human-vehicle interaction and the personalization level of the driving experience. By linking emotional state with the control of vehicle functional subsystems in a closed loop, it can proactively optimize driving assistance strategies or adjust the cabin environment according to the driver's state, helping to prevent driving risks caused by emotional fluctuations and creating a more comfortable and considerate in-car environment, thus improving comfort while ensuring safety.

[0011] In some embodiments, the step of inputting the data collected by the multi-source sensors into a pre-trained emotion computing model for multimodal information fusion to comprehensively output the user's current emotional state includes:

[0012] From the multimodal data, feature vectors of multiple dimensions are extracted, the feature vectors including:

[0013] Facial expression feature vectors extracted from user facial expressions;

[0014] Brainwave feature vectors extracted from brainwave signals;

[0015] Gesture feature vectors extracted from the user's gesture images;

[0016] Physiological feature vectors extracted from the user's physiological signals;

[0017] Voice feature vectors extracted from voice interaction;

[0018] Feature vectors of social interaction content extracted from social interaction content;

[0019] Environmental feature vectors extracted from environmental perception data;

[0020] Each feature vector is assigned a dynamic weight, which is determined in advance based on the contribution of various features to the judgment of emotional state in historical data;

[0021] A corresponding dynamic weight is assigned to each of the aforementioned features, and the dynamic weight is determined in advance based on the reliability of the judgment of the emotional state of each type of feature in historical data;

[0022] For each emotional state to be identified, a comprehensive confidence score is calculated based on the dynamic weights of each feature and their matching degree with the emotional state.

[0023] The emotional state with the highest overall confidence score is determined as the user's current emotional state.

[0024] By integrating feature vectors from seven dimensions, including facial expressions, brain waves, gestures, physiological signals, speech, social content, and environmental data, a comprehensive emotion perception system was established. This multi-source information fusion mechanism can effectively overcome the limitations of single-modal data and significantly improve the accuracy of emotion state judgment through multi-dimensional cross-validation.

[0025] Furthermore, a dynamic weight allocation mechanism is employed, adjusting the weights based on the reliability of various features in historical data. This design automatically adapts to individual differences among users and adjusts the importance of features in real time according to data quality, ensuring stable recognition performance in complex and ever-changing real-world driving environments.

[0026] By calculating the comprehensive confidence score of each emotional state, a quantitative decision-making basis was established. This decision-making mechanism based on multi-feature weighted fusion makes the process of judging emotional state more objective and scientific, and avoids decision-making bias caused by misjudgment of a single feature.

[0027] In some embodiments, the step of generating control instructions based on the user's current emotional state includes:

[0028] Based on a pre-established mapping relationship between emotional state and control strategy, at least one control strategy that matches the user's current emotional state is invoked.

[0029] Through predefined mapping relationships, abstract emotional states can be instantly transformed into concrete, actionable control strategies. This achieves a seamless transition from perception to execution, enabling vehicles to automatically and precisely adjust functions to best match the user's state without manual intervention.

[0030] In some specific embodiments, the step of invoking at least one control strategy that matches the user's current emotional state, based on a pre-established mapping relationship between emotional states and control strategies, includes:

[0031] When the user's current emotional state is positive, the control strategy executed includes at least one of the following:

[0032] The first control sub-strategy to maintain or enhance the safety monitoring level of the driver assistance system;

[0033] A second control sub-strategy to adjust cabin environment parameters to create a pleasant atmosphere;

[0034] The third control sub-strategy recommends entertainment content that matches a positive emotional state;

[0035] The fourth control sub-strategy is to publish social content on social networks that matches a positive emotional state;

[0036] The fifth control sub-strategy recommends navigation routes with scenic views or landscapes;

[0037] The sixth control sub-strategy drives the virtual assistant to interact with the user in a way that matches their positive emotional state.

[0038] By mapping predefined positive emotional states with multi-level control strategies, a comprehensive response from safety assurance to experience optimization is achieved: while maintaining or enhancing the level of driver assistance monitoring to ensure driving safety, a series of operations such as cabin environment adjustment, entertainment content recommendation, scenic route planning, virtual assistant interaction, and social content publishing are executed in a coordinated manner, constructing an immersive driving environment with safety as the foundation and experience as the core, and ultimately forming an emotion-driven service closed loop.

[0039] In some specific embodiments, the step of invoking at least one control strategy that matches the user's current emotional state, based on a pre-established emotional state control strategy mapping relationship, to drive the corresponding functional subsystem to perform an adjustment operation includes:

[0040] When the user's current emotional state is negative, the control strategy implemented includes at least one of the following:

[0041] The seventh control sub-strategy is to enhance the intervention sensitivity of the driver assistance system or actively take over some driving tasks.

[0042] The eighth control sub-strategy is to adjust cabin environment parameters to help users calm down.

[0043] The ninth control sub-strategy is to provide health advice or initiate external help.

[0044] A tiered response mechanism has been established to address negative emotional states. This mechanism directly prevents safety risks by enhancing the intervention sensitivity of the driver assistance system or proactively taking over driving tasks. Simultaneously, cabin environmental parameters are adjusted to alleviate user emotional fluctuations. When necessary, health advice is provided or external assistance functions are activated, forming a three-tiered protection system from risk warning and emotional soothing to proactive intervention, effectively ensuring driving safety and the user's physical and mental health.

[0045] In some specific embodiments, the first control sub-policy includes:

[0046] Increase the following distance of the adaptive cruise control system and / or enhance the steering wheel correction torque of the lane keeping assist system;

[0047] The second control sub-strategy includes:

[0048] Control the vehicle's ambient lighting system to switch to a preset warm color tone, control the audio system to play upbeat music, control the fragrance system to release a preset fragrance with a calming effect, control the air conditioning system to reduce the fan speed and adjust to a preset comfortable temperature, and control the seat system to activate the massage function in soothing mode.

[0049] The sixth control sub-strategy includes:

[0050] Drive the in-car virtual assistant to interact with users with more dynamic voice and tone, or recommend popular leisure and entertainment venues near your destination.

[0051] By mapping positive emotional states to specific control parameters of various vehicle functional subsystems, a synergistic improvement in safety and comfort is achieved. Increasing following distance and enhancing steering wheel correction torque improve the safety redundancy of longitudinal and lateral control, preventing driving risks caused by emotional excitement from a dynamic perspective. At the same time, the coordinated adjustment of ambient lighting, fragrance, music, air conditioning, and seat functions creates an immersive and comfortable environment through multiple channels of vision, smell, hearing, and touch. Combined with the dynamic interaction and interest-based recommendations of the virtual assistant, a three-layer response architecture of safety enhancement, environmental adjustment, and proactive service is constructed. While ensuring driving safety, the positive emotional experience of the user is continuously enhanced through multimodal interaction.

[0052] In some embodiments, the method further includes:

[0053] Display a prompt message to the user indicating the control command that is about to be executed;

[0054] In response to the user's veto command, the execution of the corresponding control command is cancelled.

[0055] By introducing a human-computer interaction mechanism with prompts and vetoes, the efficiency of automated services is maintained while effectively protecting users' decision-making power. Prompt information is provided to users after generating control commands but before actual execution, ensuring predictability and transparency of behavior. The veto command response mechanism establishes a channel for users to supervise and control the automated system, preventing erroneous operations caused by emotional misjudgments while respecting users' subjective wishes. Ultimately, an effective balance is achieved between automated services and manual control, improving reliability and user experience.

[0056] Secondly, this application also provides a vehicle intelligent control system based on multimodal emotion recognition, including:

[0057] The multi-source sensing module is configured to collect multimodal data related to user emotions, including:

[0058] The first sensing unit is used to collect the user's physiological data and behavioral interaction data, and includes at least one image sensor, at least one audio sensor, and at least one biosignal sensor deployed in the vehicle.

[0059] The second sensing unit is used to collect the user's social interaction data. It includes a vehicle communication module, which is configured to access a designated social platform application interface after obtaining user authorization in order to obtain the user's activity information on the platform.

[0060] An environmental perception module includes at least one environmental sensor deployed outside the vehicle for collecting environmental perception data around the vehicle.

[0061] The processing module is communicatively connected to the multi-source perception module and the environment perception module, and the in-vehicle emotion computing platform is configured as follows:

[0062] Receive multimodal data from the multi-source sensing module and the environment sensing module;

[0063] The received multimodal data is fused and analyzed based on a pre-trained sentiment computing model to identify the user's current sentiment state.

[0064] Based on the emotional state, generate corresponding control commands;

[0065] A strategy execution module is communicatively connected to the processing module. The strategy execution module is configured to receive the control command and drive at least one functional actuator to adaptively adjust at least one of the vehicle's driver assistance system, cabin environment system, human-machine interaction system, or comfort function system.

[0066] In some embodiments, the processing module is specifically configured as follows:

[0067] From the multimodal data, feature vectors of multiple dimensions are extracted, the feature vectors including:

[0068] Facial expression feature vectors extracted from user facial expressions;

[0069] Brainwave feature vectors extracted from brainwave signals;

[0070] Gesture feature vectors extracted from the user's gesture images;

[0071] Physiological feature vectors extracted from the user's physiological signals;

[0072] Voice feature vectors extracted from voice interaction;

[0073] Feature vectors of social interaction content extracted from social interaction content;

[0074] Environmental feature vectors extracted from environmental perception data;

[0075] Each feature vector is assigned a dynamic weight, which is determined in advance based on the contribution of various features to the judgment of emotional state in historical data;

[0076] A corresponding dynamic weight is assigned to each of the aforementioned features, and the dynamic weight is determined in advance based on the reliability of the judgment of the emotional state of each type of feature in historical data;

[0077] For each emotional state to be identified, a comprehensive confidence score is calculated based on the dynamic weights of each feature and their matching degree with the emotional state.

[0078] The emotional state with the highest overall confidence score is determined as the user's current emotional state.

[0079] Thirdly, this application also provides a vehicle including the aforementioned vehicle intelligent control system based on multimodal emotion recognition. Attached Figure Description

[0080] Figure 1 This is a structural block diagram of the vehicle in the embodiments of this application;

[0081] Figure 2 This is a flowchart of a vehicle intelligent control method based on multimodal emotion recognition in an embodiment of this application. Detailed Implementation

[0082] The following detailed description, with reference to the accompanying drawings, describes a vehicle intelligent control method, system, and vehicle based on multimodal emotion recognition according to embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0083] The core of this application's embodiments lies in providing an in-vehicle emotional intelligence assistant, which comprehensively judges the emotional state of users (especially drivers) through multimodal perception technology, and adaptively adjusts various functions of the vehicle accordingly, aiming to improve the driving experience, ensure driving safety, and provide personalized comfort services.

[0084] Figure 1According to one embodiment of the present invention, a vehicle may be, but is not limited to, a pure electric vehicle (PEV / BEV), a hybrid electric vehicle (HEV), a range-extended electric vehicle (REEV), a plug-in hybrid electric vehicle (PHEV), a new energy vehicle, or a fuel vehicle.

[0085] The vehicle includes a vehicle intelligent control system based on multimodal emotion recognition. For example... Figure 1 As shown, the system 100 mainly includes: a multi-source sensing module 110, an environmental sensing module 120, a processing module 130, and a policy execution module 140. These modules are connected for communication via the vehicle CAN bus or in-vehicle Ethernet.

[0086] The multi-source sensing module 110 includes a first sensing unit 1101 and a second sensing unit 1102.

[0087] The first sensing unit 1101 includes, for example, at least one image sensor, at least one audio sensor, and at least one biosignal sensor deployed inside the vehicle.

[0088] The image sensor includes, for example, a near-infrared camera deployed in front of the driver's seat. This camera captures images of the driver's face, which are then analyzed using image analysis algorithms to identify facial expression features. Simultaneously, the image sensor can also capture the user's gestures, thereby recognizing their hand gesture characteristics.

[0089] The audio sensor is an in-vehicle microphone array used to collect the user's voice data, and then analyze the tone, speed and key features of the voice to help judge the user's emotions.

[0090] Biosignal sensors include, for example, a heart rate sensor integrated into the steering wheel grip, which can monitor the user's heart rate characteristics in a contact manner; an electroencephalogram (EEG) acquisition device for monitoring the user's central nervous system activity, which can acquire the user's brain waves; and a sports watch for establishing communication with the vehicle, which can detect the user's blood pressure and respiratory rate.

[0091] In this embodiment, the second sensing unit 1102 is, for example, a vehicle network module, which integrates a baseband processing chip, a radio frequency front-end circuit and a corresponding antenna system to jointly realize the connection with the cellular mobile communication network.

[0092] Within the vehicle's infotainment system, an authorization management interface is provided. Users can securely bind one or more social media accounts to the vehicle system using a standard authorization protocol. After authorization, the vehicle network module can periodically or be triggered by specific events to call the open application programming interfaces provided by the social media platform through its built-in network protocol stack. Requests are sent to the mobile network via the baseband processing chip and radio frequency front-end circuitry, carrying an encrypted authentication token to securely obtain the user's social interaction data within a specified time window. This social interaction data includes, but is not limited to: user-posted image and text updates, and user comments and replies to other users' content.

[0093] The hardware foundation of the environmental perception module 120 is a set of sensors deployed outside the vehicle. Its core purpose is to acquire contextual environmental information that can indirectly infer or influence the user's emotions. Specifically, the environmental perception module 120 includes:

[0094] Ambient light sensor: Ambient light sensor is usually deployed near the base of the rearview mirror inside the windshield of a vehicle to accurately measure the ambient light intensity (unit: Lux).

[0095] External temperature / humidity sensors are typically deployed in well-ventilated areas such as the front bumper grille or rearview mirror housing to measure ambient temperature and relative humidity.

[0096] External noise sensors are typically waterproof microphones deployed outside the vehicle (such as inside the side skirts or rearview mirrors) to monitor ambient noise levels (in decibels).

[0097] In this embodiment, the processing module is communicatively connected to the multi-source perception module and the environment perception module. The in-vehicle emotion computing platform is configured to: receive multimodal data from the multi-source perception module and the environment perception module; perform fusion analysis on the received multimodal data based on a pre-trained emotion computing model to identify the user's current emotional state; and generate corresponding control commands based on the emotional state.

[0098] In this embodiment, the sentiment computing model is essentially a multimodal feature fusion and classification decision system. It operates based on a multidimensional feature sentiment state mapping database, which is trained with a large amount of data and can be optimized online based on user feedback.

[0099] The model first preprocesses and extracts features from the input multimodal raw data, transforming various data types into a unified, quantifiable feature vector:

[0100] Extract facial expression feature vector F_face from the user's facial expressions, including but not limited to: the angle of the corners of the mouth, the degree of contraction of the orbicularis oculi muscle, and the eyebrow shape encoding.

[0101] Extract the EEG feature vector F_eeg from the EEG signal, including but not limited to: the ratio of alpha wave power in the left prefrontal cortex to that in the right prefrontal cortex, and the average power of beta waves.

[0102] Extract the gesture feature vector F_gesture from the user's gesture image. It is a one-hot encoded vector used to represent the specific gestures that are recognized (such as thumbs up or fist clenching).

[0103] Extract physiological feature vector F_physio from the user's physiological signals, including but not limited to: heart rate value, heart rate variability, systolic blood pressure, and respiratory rate.

[0104] Extract the voice feature vector F_voice from the voice interaction, including but not limited to: average speech rate, fundamental frequency standard deviation, and the frequency of occurrence of sentiment keywords in the speech recognition text.

[0105] Extract the social interaction content feature vector F_social from the social interaction content, including but not limited to: the text sentiment polarity score obtained by the natural language processing model, and the count ratio of positive keywords to negative keywords.

[0106] Extract feature vector F_env from environmental perception data, including but not limited to: ambient light intensity, the difference between ambient temperature and human comfort temperature, and ambient noise decibel value.

[0107] Furthermore, each feature vector F_i is assigned a dynamic weight W_i, which reflects the contribution or reliability of such features to the judgment of sentiment state in historical data.

[0108] For each emotion state to be classified, Emotion_j (e.g., happy, angry, sad, calm), the model calculates a comprehensive confidence score S_j:

[0109] S_j=Σ(W_i*M_ij(F_i))

[0110] Here, M_ij is the matching degree function for the features F_i of the emotion state Emotion_j. This function can be a distance-based metric or a small sub-classifier.

[0111] Ultimately, the model selects the emotional state Emotion_j with the highest overall confidence score S_j as the output of the user's current emotional state.

[0112] Specifically, the methods for determining dynamic weights, as mentioned above, include:

[0113] Weights are assigned based on the classification contribution of features in historical data, with features having higher classification accuracy being given higher weights.

[0114] The weighting is dynamically adjusted based on the quality of the feature data, with features having higher signal-to-noise ratios and better data integrity being assigned higher weights.

[0115] The system dynamically adjusts based on the timeliness of features, assigning higher weights to features with greater real-time timeliness.

[0116] The methods for calculating the matching degree include:

[0117] Calculate the similarity between the current feature vector and the standard feature vectors of each emotional state;

[0118] Matching degree assessment is performed based on a pre-defined emotional state feature template;

[0119] The probability of a feature belonging to each emotional state is calculated using a trained classifier.

[0120] By integrating feature vectors from seven dimensions, including facial expressions, brain waves, gestures, physiological signals, speech, social content, and environmental data, a comprehensive emotion perception system was established. This multi-source information fusion mechanism can effectively overcome the limitations of single-modal data and significantly improve the accuracy of emotion state judgment through multi-dimensional cross-validation.

[0121] Furthermore, a dynamic weight allocation mechanism is employed, adjusting the weights based on the reliability of various features in historical data. This design automatically adapts to individual differences among users and adjusts the importance of features in real time according to data quality, ensuring stable recognition performance in complex and ever-changing real-world driving environments.

[0122] By calculating the comprehensive confidence score of each emotional state, a quantitative decision-making basis was established. This decision-making mechanism based on multi-feature weighted fusion makes the process of judging emotional state more objective and scientific, and avoids decision-making bias caused by misjudgment of a single feature.

[0123] For the processing module, the steps for generating control commands based on the user's current emotional state include:

[0124] Based on the pre-established mapping relationship between emotional state and control strategy, at least one control strategy that matches the user's current emotional state is invoked.

[0125] The processing module pre-stores a mapping relationship between emotional states and control strategies. This database defines the correspondence between specific emotional states and specific control strategies.

[0126] When a user's current emotional state is identified as a positive emotional state (such as happy, relaxed, or excited), the control policy invoked by the system includes at least one of the following:

[0127] The first control sub-strategy is used to maintain or enhance the safety monitoring level of the driver assistance system. Specifically, the first control sub-strategy includes increasing the following distance of the adaptive cruise control system and / or enhancing the steering wheel correction torque of the lane keeping assist system to ensure that the user can drive safely even when in a positive mood.

[0128] The second control sub-strategy is used to adjust cabin environment parameters to create a pleasant atmosphere. This is a comprehensive adjustment, and the second control sub-strategy specifically includes: controlling the vehicle's ambient lighting system to switch to a preset warm color tone (such as yellow or orange), controlling the audio system to play upbeat music, controlling the fragrance system to release a preset fragrance with a calming effect (such as a fresh fruity scent), controlling the air conditioning system to reduce the fan speed and adjust to a preset comfortable temperature (such as 24°C), and controlling the seat system to activate the massage function in the soothing mode.

[0129] The third control sub-strategy is used to recommend entertainment content that matches a positive emotional state. For example, recommending upbeat music playlists or podcasts on the in-car infotainment screen.

[0130] The fourth control sub-strategy is used to publish social content on social networks that matches a positive emotional state. For example, content can be automatically generated based on a preset template (such as "Feeling great, encountered something wonderful on the road!") and published through authorized accounts.

[0131] The fifth control sub-strategy is used to recommend navigation routes with scenery or landscapes. When planning routes, it prioritizes or suggests routes that pass through parks, lakes, or scenic roads.

[0132] The sixth control sub-strategy is used to drive the virtual assistant to interact with the user in a way that matches a positive emotional state. For example, the in-vehicle virtual assistant might interact with the user with a more energetic voice and tone, or recommend popular leisure and entertainment venues near the destination.

[0133] When a user's current emotional state is identified as a negative emotional state (such as anger, anxiety, sadness), the control policy invoked by the system includes at least one of the following:

[0134] The seventh control sub-strategy is used to enhance the intervention sensitivity of the driver assistance system or actively take over some driving tasks. For example, it can enable the automatic emergency braking system to issue warnings earlier, or automatically control the vehicle to decelerate and pull over in extreme situations.

[0135] The eighth control sub-strategy is used to adjust cabin environment parameters to help users calm down. For example, it can adjust the ambient lighting to blue, play soothing music, and release lavender fragrance.

[0136] The ninth control sub-strategy is used to provide health advice or initiate external assistance. For example, it can suggest to the user via voice, "You seem to be feeling tired, we recommend you rest at the service area ahead," or automatically connect to emergency rescue services when serious abnormal health indicators are detected.

[0137] In this embodiment of the application, the vehicle's functional subsystems include, for example, a driver assistance system, a cabin environment system, a human-machine interaction system, or a comfort function system.

[0138] The vehicle intelligent control system described in this embodiment of the application realizes the transformation of the vehicle from a traditional means of transportation to an emotionally intelligent partner by constructing a complete closed loop of multimodal perception, emotion computing, and strategy execution technologies. The system first comprehensively collects the user's facial expressions, gestures, voice, physiological signals, and social interaction data through a multi-source perception module (including a first perception unit integrating hardware such as near-infrared cameras, microphone arrays, and heart rate / EEG sensors, and a second perception unit based on a baseband processing chip and radio frequency front-end circuit). At the same time, it obtains information about the vehicle's surrounding environment through the ambient light sensor, temperature and humidity sensor, and noise sensor of the environmental perception module. This multi-source heterogeneous data is transmitted to the processing module through the vehicle's CAN bus or in-vehicle Ethernet, where it is fused and analyzed by a pre-trained emotion computing model.

[0139] Secondly, the model extracts features from seven dimensions: facial expression feature vectors (such as the angle of the corners of the mouth), electroencephalogram (such as the alpha / beta wave power ratio), gesture feature vectors (one-hot encoding), physiological feature vectors (such as heart rate variability), speech feature vectors (such as the fundamental frequency standard deviation), social interaction content feature vectors (such as text sentiment polarity score), and environmental feature vectors (such as light intensity). It then employs a dynamic weight allocation mechanism (based on classification contribution, data quality, and timeliness) and a comprehensive confidence score calculation (using the matching degree function Mi_ij) to ultimately output an accurate sentiment state judgment. This multi-source information fusion and weighted decision-making mechanism effectively overcomes the limitations of single-modal data, significantly improving the accuracy of sentiment recognition and its robustness in complex driving environments.

[0140] Furthermore, based on emotion recognition results, a pre-established emotional state-control strategy mapping relationship drives the strategy execution module to achieve precise functional adjustments. When a positive emotional state is identified, the system strengthens safety through the first control sub-strategy (increasing ACC following distance, etc.), creates a pleasant atmosphere through the second control sub-strategy (coordinating multi-channel adjustments such as ambient lighting, fragrance, music, air conditioning, and seat massage), and enhances the user experience through the sixth control sub-strategy (virtual assistant interactive features). When a negative emotional state is identified, the system prevents safety risks through the seventh control sub-strategy (improving the sensitivity of driver assistance intervention) and calms the user's emotions through the eighth control sub-strategy (adjusting environmental parameters), forming a complete protection system from risk warning and emotional soothing to proactive intervention. This mechanism, which deeply links emotional states with functional subsystems such as the vehicle's driver assistance system, cabin environment system, and human-machine interaction system, not only ensures driving safety but also creates a highly personalized intelligent driving experience.

[0141] Reference Figure 2 This application also provides a vehicle intelligent control method based on multimodal emotion recognition, including:

[0142] S101, acquire multimodal data related to user emotions, including user physiological data, user behavior interaction data, and environmental perception data around the vehicle;

[0143] S102, input the data collected by multiple sensors into the pre-trained emotion computing model to perform multimodal information fusion, so as to comprehensively output the user's current emotional state;

[0144] S103, Generate control commands based on the user's current emotional state;

[0145] S104, adaptively adjust at least one functional subsystem of the vehicle according to control commands.

[0146] The step S102, which involves inputting data collected from multiple sensors into a pre-trained emotion computing model for multimodal information fusion to comprehensively output the user's current emotional state, includes:

[0147] Extract multi-dimensional feature vectors from multimodal data. The feature vectors include:

[0148] Facial expression feature vectors extracted from user facial expressions;

[0149] Brainwave feature vectors extracted from brainwave signals;

[0150] Gesture feature vectors extracted from the user's gesture images;

[0151] Physiological feature vectors extracted from the user's physiological signals;

[0152] Voice feature vectors extracted from voice interaction;

[0153] Feature vectors of social interaction content extracted from social interaction content;

[0154] Environmental feature vectors extracted from environmental perception data;

[0155] Each feature vector is assigned a dynamic weight, which is determined in advance based on the contribution of various features to the judgment of sentiment state in historical data.

[0156] Each feature is assigned a corresponding dynamic weight, which is determined in advance based on the reliability of the emotional state judgment based on various features in historical data.

[0157] For each emotional state to be identified, a comprehensive confidence score is calculated based on the dynamic weights of each feature and their matching degree with the emotional state.

[0158] The sentiment state with the highest overall confidence score is determined as the user's current sentiment state.

[0159] The step of generating control instructions S103 based on the user's current emotional state includes:

[0160] Based on the pre-established mapping relationship between emotional state and control strategy, at least one control strategy that matches the user's current emotional state is invoked.

[0161] The step of invoking at least one control strategy that matches the user's current emotional state, based on a pre-established mapping relationship between emotional states and control strategies, includes:

[0162] When a user's current emotional state is positive, the control strategy implemented includes at least one of the following:

[0163] The first control sub-strategy to maintain or enhance the safety monitoring level of the driver assistance system;

[0164] A second control sub-strategy to adjust cabin environment parameters to create a pleasant atmosphere;

[0165] The third control sub-strategy recommends entertainment content that matches a positive emotional state;

[0166] The fourth control sub-strategy is to publish social content on social networks that matches a positive emotional state;

[0167] The fifth control sub-strategy recommends navigation routes with scenic views or landscapes;

[0168] The sixth control sub-strategy drives the virtual assistant to interact with the user in a way that matches their positive emotional state.

[0169] The steps of invoking at least one control strategy that matches the user's current emotional state, based on a pre-established mapping relationship of emotional state control strategies, to drive the corresponding functional subsystem to perform adjustment operations include:

[0170] When a user's current emotional state is negative, the control strategy implemented includes at least one of the following:

[0171] The seventh control sub-strategy is to enhance the intervention sensitivity of the driver assistance system or actively take over some driving tasks.

[0172] The eighth control sub-strategy is to adjust cabin environment parameters to help users calm down.

[0173] The ninth control sub-strategy is to provide health advice or initiate external help.

[0174] The first control sub-policy includes:

[0175] Increase the following distance of the adaptive cruise control system and / or enhance the steering wheel correction torque of the lane keeping assist system;

[0176] The second control sub-strategy includes:

[0177] Control the vehicle's ambient lighting system to switch to a preset warm color tone, control the audio system to play upbeat music, control the fragrance system to release a preset fragrance with a calming effect, control the air conditioning system to reduce the fan speed and adjust to a preset comfortable temperature, and control the seat system to activate the massage function in soothing mode.

[0178] The sixth control sub-policy includes:

[0179] Drive the in-car virtual assistant to interact with users with more dynamic voice and tone, or recommend popular leisure and entertainment venues near your destination.

[0180] The methods also include:

[0181] S105, output a prompt message to the user regarding the control command that is about to be executed;

[0182] S106, in response to the user's veto command, cancel the execution of the corresponding control command.

[0183] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Those skilled in the art can understand that implementing all or part of the processes of the above embodiments and making equivalent changes according to the claims of this application still fall within the scope of this application.

Claims

1. A vehicle intelligent control method based on multimodal emotion recognition, characterized in that, include: Acquire multimodal data related to user emotions, including user physiological data, user behavior interaction data, and environmental perception data around the vehicle; The data collected by the multi-source sensors are input into a pre-trained emotion computing model for multimodal information fusion to comprehensively output the user's current emotional state; Generate control commands based on the user's current emotional state; Based on the control commands, at least one functional subsystem of the vehicle is adaptively adjusted.

2. The vehicle intelligent control method based on multimodal emotion recognition according to claim 1, characterized in that, The steps of inputting the data collected by the multi-source sensors into a pre-trained emotion computing model for multimodal information fusion to comprehensively output the user's current emotional state include: From the multimodal data, feature vectors of multiple dimensions are extracted, the feature vectors including: Facial expression feature vectors extracted from user facial expressions; Brainwave feature vectors extracted from brainwave signals; Gesture feature vectors extracted from the user's gesture images; Physiological feature vectors extracted from the user's physiological signals; Voice feature vectors extracted from voice interaction; Feature vectors of social interaction content extracted from social interaction content; Environmental feature vectors extracted from environmental perception data; Each feature vector is assigned a dynamic weight, which is determined in advance based on the contribution of various features to the judgment of emotional state in historical data; A corresponding dynamic weight is assigned to each of the aforementioned features, and the dynamic weight is determined in advance based on the reliability of the judgment of the emotional state of each type of feature in historical data; For each emotional state to be identified, a comprehensive confidence score is calculated based on the dynamic weights of each feature and their matching degree with the emotional state. The emotional state with the highest overall confidence score is determined as the user's current emotional state.

3. The vehicle intelligent control method based on multimodal emotion recognition according to claim 1, characterized in that, The steps for generating control instructions based on the user's current emotional state include: Based on a pre-established mapping relationship between emotional state and control strategy, at least one control strategy that matches the user's current emotional state is invoked.

4. The method according to claim 3, characterized in that, Based on a pre-established mapping relationship between emotional states and control strategies, the step of invoking at least one control strategy that matches the user's current emotional state includes: When the user's current emotional state is positive, the control strategy executed includes at least one of the following: The first control sub-strategy to maintain or enhance the safety monitoring level of the driver assistance system; A second control sub-strategy to adjust cabin environment parameters to create a pleasant atmosphere; The third control sub-strategy recommends entertainment content that matches a positive emotional state; The fourth control sub-strategy is to publish social content on social networks that matches a positive emotional state; The fifth control sub-strategy recommends navigation routes with scenic views or landscapes; The sixth control sub-strategy drives the virtual assistant to interact with the user in a way that matches their positive emotional state.

5. The method according to claim 3, characterized in that, Based on a pre-established mapping relationship of emotional state control strategies, the steps of invoking at least one control strategy that matches the user's current emotional state to drive the corresponding functional subsystem to perform adjustment operations include: When the user's current emotional state is negative, the control strategy implemented includes at least one of the following: The seventh control sub-strategy is to enhance the intervention sensitivity of the driver assistance system or actively take over some driving tasks. The eighth control sub-strategy is to adjust cabin environment parameters to help users calm down. The ninth control sub-strategy is to provide health advice or initiate external help.

6. The method according to claim 4, characterized in that, The first control sub-policy includes: Increase the following distance of the adaptive cruise control system and / or enhance the steering wheel correction torque of the lane keeping assist system; The second control sub-strategy includes: Control the vehicle's ambient lighting system to switch to a preset warm color tone, control the audio system to play upbeat music, control the fragrance system to release a preset fragrance with a calming effect, control the air conditioning system to reduce the fan speed and adjust to a preset comfortable temperature, and control the seat system to activate the massage function in soothing mode. The sixth control sub-strategy includes: Drive the in-car virtual assistant to interact with users with more dynamic voice and tone, or recommend popular leisure and entertainment venues near your destination.

7. The method according to claim 1, characterized in that, The method further includes: Display a prompt message to the user indicating the control command that is about to be executed; In response to the user's veto command, the execution of the corresponding control command is cancelled.

8. A vehicle intelligent control system based on multimodal emotion recognition, characterized in that, include: The multi-source sensing module is configured to collect multimodal data related to user emotions, including: The first sensing unit is used to collect the user's physiological data and behavioral interaction data, and includes at least one image sensor, at least one audio sensor, and at least one biosignal sensor deployed in the vehicle. The second sensing unit is used to collect the user's social interaction data. It includes a vehicle communication module, which is configured to access a designated social platform application interface after obtaining user authorization in order to obtain the user's activity information on the platform. An environmental perception module includes at least one environmental sensor deployed outside the vehicle for collecting environmental perception data around the vehicle. The processing module is communicatively connected to the multi-source perception module and the environment perception module, and the in-vehicle emotion computing platform is configured as follows: Receive multimodal data from the multi-source sensing module and the environment sensing module; The received multimodal data is fused and analyzed based on a pre-trained sentiment computing model to identify the user's current sentiment state. Based on the emotional state, generate corresponding control commands; A strategy execution module is communicatively connected to the processing module. The strategy execution module is configured to receive the control command and drive at least one functional actuator to adaptively adjust at least one of the vehicle's driver assistance system, cabin environment system, human-machine interaction system, or comfort function system.

9. The vehicle intelligent control system based on multimodal emotion recognition according to claim 8, characterized in that, The processing module is specifically configured as follows: From the multimodal data, feature vectors of multiple dimensions are extracted, the feature vectors including: Facial expression feature vectors extracted from user facial expressions; Brainwave feature vectors extracted from brainwave signals; Gesture feature vectors extracted from the user's gesture images; Physiological feature vectors extracted from the user's physiological signals; Voice feature vectors extracted from voice interaction; Feature vectors of social interaction content extracted from social interaction content; Environmental feature vectors extracted from environmental perception data; Each feature vector is assigned a dynamic weight, which is determined in advance based on the contribution of various features to the judgment of emotional state in historical data; A corresponding dynamic weight is assigned to each of the aforementioned features, and the dynamic weight is determined in advance based on the reliability of the judgment of the emotional state of each type of feature in historical data; For each emotional state to be identified, a comprehensive confidence score is calculated based on the dynamic weights of each feature and their matching degree with the emotional state. The emotional state with the highest overall confidence score is determined as the user's current emotional state.

10. A vehicle, characterized in that, The vehicle intelligent control system based on multimodal emotion recognition as described in claim 8 or 9.

Images