Automatic adaptation method and system for multi-climate-zone emotional response strategy

Abstract

The invention provides an automatic adaptation method and system for a multi-climate zone emotional response strategy. The method comprises the steps that S1, the type of a climate zone where a vehicle is located currently is recognized; s2, collecting multi-dimensional environment data reflecting the environment where the vehicle is located, and constructing a structured environment state vector based on the multi-dimensional environment data; s3, collecting multi-modal data of the target object, wherein the multi-modal data comprises a facial image, voice information and physiological information; extracting a feature vector reflecting an emotional state based on the collected multi-modal data, and identifying a current emotion category based on the extracted feature vector reflecting the emotional state; and S4, according to the identified climatic zone type, environment state vector and emotion category of the current vehicle, searching and matching an optimal strategy template in an emotion response strategy library, and generating equipment control parameters.

Description

Technical Field

[0001] The present invention relates to the field of automobile technology, and in particular to a method and system for automatically adapting emotional response strategies in multiple climate zones. Background Art

[0002] With the rapid development of smart cockpit technology, in-vehicle emotional interaction systems have gradually become a core direction for improving the driving experience. However, existing technologies mostly focus on emotional responses in a single environment or static scene, lacking adaptability to complex climate conditions and regional differences.

[0003] Patent document CN114132328B (application number: 202111507451.0) discloses an assisted driving system, method, and storage medium for automatically adjusting the driving environment. The system includes: a state acquisition module for collecting real-time physiological and emotional state information of the driver during driving; an environment acquisition module for collecting real-time vehicle internal and external environmental information during driving; an information processing module for determining information on influencing factors that cause changes in the driver's emotional state based on the physiological, emotional, and internal and external environmental information; and an environment adjustment module for interacting with the state acquisition module, environment acquisition module, and information processing module via a bus. The environment adjustment module is configured to adjust the target vehicle's internal environment, driving mode, and media functions based on the influencing factors determined by the information processing module. While this patent considers real-time climate information, it does not incorporate a global climate zone adaptation mechanism and lacks regional difference determination. Furthermore, the strategy relies on rule-based settings and lacks self-learning and personalization capabilities.

[0004] Patent document CN112455370A (application number: 202011325496.1) discloses an emotion management and interaction system and method based on a multi-dimensional data arbitration mechanism, including a microphone, an audio host, an emotion management engine, a speaker, a central control screen, an environmental atmosphere system, and a face camera; the microphone and the audio host are connected by a wiring harness; the audio host and the emotion management engine are connected by a software interface; the emotion management engine is respectively connected to the speaker, the central control screen, and the environmental atmosphere system by a wiring harness; the face camera and the audio host are connected by a wiring harness. In this invention, during the interaction between the driver and the actual vehicle, the actual vehicle can detect the driver's emotional state in real time and conduct emotional arbitration. According to the results of the arbitration, feedback is provided to the driver's vision, hearing, and smell, and the driver's emotions are dynamically adjusted to achieve the purpose of improving driving safety. The emotion recognition of this patent relies on a multimodal arbitration mechanism but does not combine geographical climate information, and cannot perform regional response optimization. The control logic is relatively fixed and does not support dynamic adjustment of strategies under situational awareness. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for automatically adapting emotional response strategies in multiple climate zones.

[0006] According to the present invention, a method for automatically adapting a multi-climate zone emotional response strategy includes:

[0007] Step S1: Identify the climate zone type that the vehicle is currently in;

[0008] Step S2: collecting multi-dimensional environmental data reflecting the environment in which the vehicle is located, and constructing a structured environmental state vector based on the multi-dimensional environmental data;

[0009] Step S3: collecting multimodal data of the target object, including facial images, voice information, and physiological information; extracting feature vectors reflecting the emotional state based on the collected multimodal data; and identifying the current emotion category based on the extracted feature vectors reflecting the emotional state;

[0010] Step S4: According to the identified climate zone type, environmental state vector, and emotion category of the current vehicle, the optimal matching strategy template is retrieved from the emotion response strategy library to generate device control parameters.

[0011] Preferably, the step S1 includes: obtaining the vehicle's geographic location information in real time through a vehicle-mounted positioning module; outputting a climate zone label based on the vehicle's geographic location information and matching it with a climate zone classification database;

[0012] The climate zone label includes: climate zone code, climate zone name, regional distribution and climate characteristics.

[0013] Preferably, step S2 includes:

[0014] Step S2.1: Acquire multi-dimensional environmental perception data through vehicle-mounted sensors, wherein the multi-dimensional environmental perception data includes: vehicle interior temperature, vehicle interior humidity, CO2 concentration, light intensity, vehicle exterior temperature, and vehicle exterior humidity;

[0015] Step S2.2: Accessing the cloud meteorological platform through the communication module to obtain remote meteorological data, wherein the remote meteorological data includes: real-time weather conditions, external temperature and relative humidity, wind speed risk, ultraviolet intensity, and precipitation probability;

[0016] Step S2.3: constructing an environmental state vector based on the multi-dimensional environmental perception data and remote meteorological data by weighted averaging method;

[0017] Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code];

[0018] Step S2.4: Correct the constructed environment state vector using the credibility correction strategy to obtain a corrected environment state vector.

[0019] Preferably, step S3 includes:

[0020] Step S3.1: Acquire a facial image of the target object, locate the facial region using a face detection model based on the acquired facial image, and extract the facial expression vector using a lightweight neural network model based on the facial region location;

[0021] Step S3.2: Acquire the sound signal of the target subject during natural interaction or subjective speech input, perform noise reduction and frame processing on the acquired sound signal to obtain a processed sound signal; extract audio feature parameters reflecting the emotional state based on the processed sound signal, including: Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation index; encode the extracted audio feature parameters reflecting the emotional state into a sound feature vector with a length that meets preset requirements;

[0022] Step S3.3: Acquire physiological signals of the target subject, the physiological signals including heart rate, galvanic skin response, and body surface temperature; perform preprocessing on the acquired physiological signals, including filtering, normalization, and detrending, to obtain preprocessed physiological signals; and construct a physiological state feature vector having a length of several dimensions based on the preprocessed physiological signals;

[0023] Step S3.4: Construct a multimodal fusion vector based on the facial expression vector, the sound feature vector, and the physiological state feature vector, input the constructed multimodal fusion vector into the multimodal emotion recognition model for emotion classification, and obtain the emotional state label of the target object.

[0024] Preferably, the constructing of a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector includes: constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector through feature-level splicing, decision-level weighted fusion or introducing an attention mechanism to weight the credibility of different modalities.

[0025] Preferably, when using a multimodal emotion recognition model to perform emotion classification to obtain the emotional state label of the target object, the corresponding confidence score is obtained. When the confidence score is lower than the set threshold, the current emotion result is not used as the basis for strategy triggering; when the confidence score is not lower than the set threshold, it triggers the retrieval of the optimal matching strategy template in the emotion response strategy library and generates device control parameters to enable intelligent control of the device.

[0026] According to the present invention, an automatic adaptation system for multi-climate zone emotional response strategies is provided, comprising:

[0027] Module M1: Identify the climate zone type the vehicle is currently in;

[0028] Module M2: collects multi-dimensional environmental data reflecting the vehicle's environment and constructs a structured environmental state vector based on the multi-dimensional environmental data;

[0029] Module M3: Collects multimodal data of the target subject, including facial images, voice information, and physiological information; extracts feature vectors reflecting the emotional state based on the collected multimodal data; and identifies the current emotion category based on the extracted feature vectors reflecting the emotional state;

[0030] Module M4: Based on the identified climate zone type, environmental state vector, and emotion category of the current vehicle, the optimal matching strategy template is retrieved from the emotion response strategy library to generate device control parameters.

[0031] Preferably, the module M1 includes: obtaining the vehicle's geographical location information in real time through a vehicle positioning module; outputting a climate zone label based on the vehicle's geographical location information and matching it with a climate zone classification database;

[0032] The climate zone label includes: climate zone code, climate zone name, regional distribution and climate characteristics.

[0033] Preferably, the module M2 includes:

[0034] Module M2.1: Acquires multi-dimensional environmental perception data through onboard sensors, including: in-vehicle temperature, in-vehicle humidity, CO2 concentration, light intensity, and outside temperature and humidity.

[0035] Module M2.2: Access the cloud meteorological platform through the communication module to obtain remote meteorological data, including real-time weather conditions, external temperature and relative humidity, wind speed risk, ultraviolet intensity, and precipitation probability;

[0036] Module M2.3: Constructing the environmental state vector using the weighted average method based on multi-dimensional environmental perception data and remote meteorological data;

[0037] Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code];

[0038] Module M2.4: Correct the constructed environment state vector through the credibility correction strategy to obtain the corrected environment state vector.

[0039] Preferably, the module M3 includes:

[0040] Module M3.1: Collect facial images of the target subject, locate the facial region using a face detection model based on the collected facial image, and extract facial expression vectors using a lightweight neural network model based on the facial region location.

[0041] Module M3.2: Collect the target subject's sound signals during natural interaction or subjective speech input, perform noise reduction and frame processing on the collected sound signals to obtain processed sound signals; extract audio feature parameters reflecting the emotional state based on the processed sound signals, including: Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation indicators; encode the extracted audio feature parameters reflecting the emotional state into a sound feature vector with a length that meets preset requirements;

[0042] Module M3.3: Acquire physiological signals from the target subject, including heart rate, galvanic skin response, and body surface temperature; perform preprocessing on the acquired physiological signals, including filtering, normalization, and detrending, to obtain preprocessed physiological signals; and construct a physiological state feature vector having a length of several dimensions based on the preprocessed physiological signals.

[0043] Module M3.4: Construct a multimodal fusion vector based on facial expression vectors, voice feature vectors, and physiological state feature vectors. Input the constructed multimodal fusion vector into a multimodal emotion recognition model for emotion classification to obtain the target object's emotional state label.

[0044] The method of constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector includes: constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector through feature-level splicing, decision-level weighted fusion or introducing an attention mechanism to weight the credibility of different modalities.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention introduces a climate recognition mechanism based on GPS positioning and a climate zone geographic information database, addressing the technical issues of existing cabin emotional response systems, which are unable to distinguish climate characteristics in different geographical regions and whose response strategies are not adaptable. This mechanism automatically determines the climate zone category based on the vehicle's current location and dynamically adjusts the parameter benchmarks of the emotional control strategy, providing a foundational contextual basis for subsequent response strategies. This achieves a shift in strategy control from "fixed logic" to "regional adaptation."

[0047] 2. This invention integrates multi-parameter sensor data from the vehicle's internal and external environments, including temperature, humidity, sunlight intensity, PM2.5 concentration, and in-vehicle carbon dioxide concentration, to build a complete environmental perception subsystem. This addresses the problem of existing systems collecting insufficient parameter dimensions and failing to reflect the actual riding environment. This method enables real-time monitoring and coordinated invocation of environmental factors under extreme climate conditions, providing more accurate and time-sensitive environmental context support for policy selection.

[0048] 3. By jointly modeling multimodal emotion recognition results with climate zone types and current environmental parameters, the present invention achieves context-aware adaptation of occupant emotional response strategies, addressing the problem of traditional policy libraries with a single adjustment method and lack of personalized responses. This method combines the occupant's emotional state identified through image, voice, and physiological modalities to match the most appropriate response plan in the policy library, achieving a precise response tailored to the individual and the location. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0050] Figure 1 Flowchart of the automatic adaptation method for emotional response strategies in multiple climate zones.

[0051] Figure 2 Flowchart of the human factors engineering testing method for the emotional cockpit.

[0052] Figure 3 Flowchart of the dynamic calibration method for the in-vehicle emotion recognition system.

[0053] Figure 4 Flowchart of the subscription service approach for the Emotion Cockpit feature. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0055] Example 1

[0056] According to the present invention, a method for automatically adapting a multi-climate zone emotional response strategy is provided. Figure 1 Shown, including:

[0057] Step S1: Identify the climate zone type that the vehicle is currently in;

[0058] Step S2: collecting multi-dimensional environmental data reflecting the environment in which the vehicle is located, and constructing a structured environmental state vector based on the multi-dimensional environmental data;

[0059] Step S3: collecting multimodal data of the target object, including facial images, voice information, and physiological information; extracting feature vectors reflecting the emotional state based on the collected multimodal data; and identifying the current emotion category based on the extracted feature vectors reflecting the emotional state;

[0060] Step S4: According to the identified climate zone type, environmental state vector, and emotion category of the current vehicle, the optimal matching strategy template is retrieved from the emotion response strategy library to generate device control parameters.

[0061] Specifically, step S1 includes: obtaining the vehicle's geographic location information in real time through the vehicle positioning module; matching the climate zone classification database based on the vehicle's geographic location information to output a climate zone label;

[0062] The climate zone label includes: climate zone code, climate zone name, regional distribution and climate characteristics.

[0063] Specifically, step S2 includes:

[0064] Step S2.1: Acquire multi-dimensional environmental perception data through vehicle-mounted sensors, wherein the multi-dimensional environmental perception data includes: vehicle interior temperature, vehicle interior humidity, CO2 concentration, light intensity, vehicle exterior temperature, and vehicle exterior humidity;

[0065] Step S2.2: Accessing the cloud meteorological platform through the communication module to obtain remote meteorological data, wherein the remote meteorological data includes: real-time weather conditions, external temperature and relative humidity, wind speed risk, ultraviolet intensity, and precipitation probability;

[0066] Step S2.3: constructing an environmental state vector based on the multi-dimensional environmental perception data and remote meteorological data by weighted averaging method;

[0067] Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code];

[0068] Step S2.4: Correct the constructed environment state vector using the credibility correction strategy to obtain a corrected environment state vector.

[0069] Specifically, step S3 includes:

[0070] Step S3.1: Acquire a facial image of the target object, locate the facial region using a face detection model based on the acquired facial image, and extract the facial expression vector using a lightweight neural network model based on the facial region location;

[0071] Step S3.2: Acquire the sound signal of the target subject during natural interaction or subjective speech input, perform noise reduction and frame processing on the acquired sound signal to obtain a processed sound signal; extract audio feature parameters reflecting the emotional state based on the processed sound signal, including: Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation index; encode the extracted audio feature parameters reflecting the emotional state into a sound feature vector with a length that meets preset requirements;

[0072] Step S3.3: Acquire physiological signals of the target subject, the physiological signals including heart rate, galvanic skin response, and body surface temperature; perform preprocessing on the acquired physiological signals, including filtering, normalization, and detrending, to obtain preprocessed physiological signals; and construct a physiological state feature vector having a length of several dimensions based on the preprocessed physiological signals;

[0073] Step S3.4: Construct a multimodal fusion vector based on the facial expression vector, the sound feature vector, and the physiological state feature vector, input the constructed multimodal fusion vector into the multimodal emotion recognition model for emotion classification, and obtain the emotional state label of the target object.

[0074] Specifically, the constructing of a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector includes: constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector through feature-level splicing, decision-level weighted fusion or introducing an attention mechanism to weight the credibility of different modalities.

[0075] Specifically, when using a multimodal emotion recognition model to perform emotion classification to obtain the emotional state label of the target object, its corresponding confidence score is obtained. When the confidence score is lower than the set threshold, the current emotion result is not used as the basis for strategy triggering; when the confidence score is not lower than the set threshold, it triggers the retrieval of the optimal matching strategy template in the emotion response strategy library and generates device control parameters to enable intelligent control of the device.

[0076] The present invention also provides an automatic adaptation system for a multi-climate zone emotional response strategy. The automatic adaptation system for a multi-climate zone emotional response strategy can be implemented by executing the process steps of the automatic adaptation method for a multi-climate zone emotional response strategy. That is, those skilled in the art can understand the automatic adaptation method for a multi-climate zone emotional response strategy as a preferred implementation of the automatic adaptation system for a multi-climate zone emotional response strategy.

[0077] Example 2

[0078] Example 2 is a preferred example of Example 1

[0079] According to the present invention, a method and system for automatically adapting emotional response strategies for multiple climate zones are provided. By combining vehicle positioning information with the global climate classification system, the type of climate zone in which the vehicle is located can be intelligently identified, and environmental perception and emotion recognition results can be integrated to achieve dynamic matching and regional differentiated regulation of emotional response strategies. The present invention supports emotion-driven linkage adjustment of multiple devices such as in-vehicle temperature, lighting, and fragrance, and has strong environmental adaptability, high response accuracy, and personalized evolution capabilities, significantly improving cabin comfort and emotional regulation intelligence in complex climate conditions.

[0080] The automatic adaptation method of the multi-climate zone emotional response strategy includes:

[0081] Step 1: Based on the locally applied Köppen climate classification ( Climate Classification) combines GNSS geolocation with preset climate zone distribution data to achieve refined identification of the climate zone type at the vehicle's current location, providing support for regional adaptation of subsequent emotional response strategies;

[0082] Step 2: Environmental parameter collection: Using onboard sensors and an external meteorological platform, we collect multi-dimensional sensory data reflecting the occupant's surroundings, forming a structured environmental state vector as one of the inputs for emotion strategy adaptation. This environmental data collection is divided into two sub-steps: local sensor collection and remote meteorological data fusion.

[0083] Step 3: Multimodal emotion recognition: By collecting multimodal data such as the occupant's images, voice, and physiological signals, the feature vector reflecting the emotional state is extracted. The fusion model then outputs the occupant's current emotion category and confidence score to drive the subsequent selection of emotional response strategies.

[0084] Step 4: Strategy matching and response decision. Based on the climate zone label obtained in step 1, the environmental state vector obtained in step 2, and the emotion recognition result output in step 3, the optimal matching strategy template is retrieved from the emotion response strategy library and the device control parameters are generated to achieve joint emotion regulation of multiple subsystems such as air conditioning, lighting, fragrance, and seating.

[0085] Step 5: Response Execution and Strategy Optimization. Based on the emotion response strategy parameter set generated in Step 4, corresponding control instructions are issued to the vehicle control system, driving the cabin environment control modules to perform corresponding operations. The strategy parameter set includes control variables for the air conditioning system, lighting system, fragrance system, and seat comfort system. The specific content depends on the current climate zone, environmental conditions, and occupant emotion recognition results. Based on these parameters, the system constructs standardized control instructions and transmits them to the corresponding execution modules via the vehicle communication bus, completing the multi-channel coordinated adjustment of the vehicle interior environment.

[0086] This embodiment uses a closed-loop control process based on climate perception, environmental understanding, and emotional reasoning. Through comprehensive judgment of climate zones and the environment inside and outside the vehicle, it accurately identifies the occupants' current emotional state and dynamically matches the optimal control parameters in a multi-strategy library to achieve emotion-driven coordinated adjustment of on-board air conditioning, lighting, fragrances, seats, and other equipment. It has technical advantages such as strong cross-regional adaptability, stable response links, accurate emotion recognition, and flexible policy control.

[0087] Specifically, step 1 includes the following steps:

[0088] Step 1.1: Get the vehicle's geographic location information;

[0089] In this embodiment, the vehicle's latitude and longitude coordinates are acquired in real time through the on-board GNSS module, with a positioning accuracy of no less than ±5 meters and a sampling frequency of no less than 1Hz. The positioning module is integrated into the T-BOX or IVI main control, and the acquired position data is output in the standard WGS-84 format. Among them, the on-board GNSS modules include: GPS, Beidou, and GLONASS;

[0090] Step 1.2: Match the climate zone classification database and output the climate zone label;

[0091] The matching process outputs climate zone codes that serve as part of the environmental input vector in subsequent emotional response strategies.

[0092]

[0093]

[0094] The step 2 comprises the following steps:

[0095] Step 2.1: Local environment sensor acquisition;

[0096] Multiple environmental sensors are preset inside and outside the vehicle cabin to collect the following data:

[0097] Parameter name Collection range Accuracy requirements Update frequency In-car temperature -10～50℃ ±0.5℃ ≥1Hz Humidity in the car 0~100% RH ±3%RH ≥1Hz <![CDATA[CO2 concentration]]> 0~5000ppm ±50ppm ≥0.5Hz Light intensity 0~100,000 lux ±5% (full scale) ≥1Hz Outside temperature -10～50℃ ±0.5℃ ≥1Hz Humidity outside the vehicle 0~100% RH ±3%RH ≥1Hz

[0098] The data is transmitted locally via the CAN or LIM bus to the central processing unit (DCU) and is updated every second.

[0099] The collected data uses timestamp as the primary key.

[0100] Step S2.2: Remote meteorological data fusion;

[0101] To address the limitations of onboard sensors, the system accesses a cloud-based meteorological platform through a communication module, integrating macro-climate parameters. Platform data is periodically acquired via a RESTful API, with a recommended update frequency of at least 10 minutes. Remote meteorological data includes real-time weather conditions, outside temperature and relative humidity, wind speed and direction, UV intensity, and precipitation probability.

[0102] The local data is integrated with the remote data, using a weighted average method and a credibility correction strategy. The default weight distribution is: local sensor data weight (70%), remote meteorological data weight (30%).

[0103] Generate environmental state vector: Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code]

[0104] For example: [24.7,45.2,820,650,18,5,2.3,03];

[0105] Weather type code

[0106] 01 02 03 04 05 06 fog Negative clear light rain thunderstorms heavy snow

[0107] The step 3 comprises the following steps:

[0108] Step 3.1: Image modality emotion feature extraction;

[0109] The system collects occupant facial image data in real time through cameras deployed in the front row or center console of the vehicle, and processes the image stream at a rate of no less than 25 frames per second. The image resolution is recommended to be no less than 720p to ensure that facial detail areas can be stably extracted. The system uses a face detection algorithm to locate the facial area and further extracts feature points of key expression areas such as eyebrows, eyes, and corners of the mouth. After completing face alignment and cropping, the system uses a lightweight neural network model to extract image expression vector features. The output dimension can be an embedding vector of 64 to 128 dimensions, which is used to represent the expression and emotional tendency in the current image state. This process can continuously process multiple time frames to construct dynamic expression change trends under image modality and serve as subsequent fusion input.

[0110] Step 3.2: Extract speech modality emotion features;

[0111] During speech modality recognition, the in-vehicle microphone array collects occupant sound signals during natural interactions or subjective speech input. A sampling frequency of at least 16kHz is recommended. The system first performs noise reduction and frame processing on the speech signal, then extracts audio feature parameters reflecting emotional state, including Mel-Frequency Cepstral Coefficients (MFCCs), intonation contour, fundamental frequency variation, speech rate, and energy variation. These features are encoded into a 32- to 64-dimensional vector and input into the speech emotion recognition sub-model for classification. Model training can utilize supervised learning based on an emotion corpus to identify states such as anger, calmness, excitement, and fatigue. In actual use, the emotion judgment results of the speech modality will be jointly analyzed with the image modality.

[0112] Step 3.3: Extraction of physiological modality emotion features;

[0113] If the vehicle is equipped with a physiological signal acquisition module (such as an embedded heart rate belt, ECG contact electrodes or seat sensors), the system can collect multiple physiological signals of the occupants in real time. The collected signals include but are not limited to: heart rate (bpm), skin electrical response (conductivity), body surface temperature (°C), etc. The rate of change of heart rate, the intensity of skin electrical fluctuations, and the trend of body temperature drift can all reflect the emotional stress level of the occupants. After the system pre-processes various signals (filtering, standardization, detrending), it constructs a physiological state feature vector of several dimensions in length to supplement the non-subjective information of image and voice modalities and improve the robustness of the system in high noise or image occlusion conditions. If the system is not configured with a physiological modality, this step can be automatically skipped.

[0114] Step 3.4: Multimodal fusion and emotional state output;

[0115] The feature vectors extracted from each of the above modalities are unified and integrated to construct a multimodal fusion input. The fusion method can adopt feature-level splicing (early fusion) or decision-level weighted fusion (late fusion), or an attention mechanism can be introduced to weight the credibility of different modalities. The final fusion result is input into the multimodal emotion recognition model for emotion classification, and the occupant's current emotional state label and its corresponding confidence score are output. Emotion categories may include calm, happy, angry, anxious, surprised, tired, etc., and the confidence level is used to measure the degree of recognition reliability. If the confidence level is lower than the system-set threshold (for example, 0.6), the current emotional result will not be used as the basis for policy triggering, and only the emotional fluctuation trend will be recorded. The recognition result will be transmitted to the policy matching module to drive the subsequent environmental control logic.

[0116] The step 4 comprises the following steps:

[0117] Step 4.1: Joint input vector construction;

[0118]

[0119] Example of joint input vectors:

[0120] Input_Vector=["Cfa",24.7,45.2,820,650,18,5,2.3,"03","anxious",0.84]

[0121] Step 3.2: Strategy matching mechanism;

[0122]

[0123] Strategy matching can be done using a rule engine, mapping climate zones and sentiment labels into conditional combinations, and then matching the corresponding strategy templates. Some commonly used templates are listed below:

[0124]

[0125] Example 3

[0126] Example 3 is a preferred example of Example 1

[0127] According to the human factors engineering testing method of an emotional cockpit provided by the present invention, Figure 2 Shown, including:

[0128] Step A1: Acquire multi-source data, including facial images, voice signals, and physiological signals; and pre-process the acquired multi-source data to obtain pre-processed multi-source data;

[0129] Step A2: performing feature extraction based on the preprocessed facial image, speech signal, and physiological signal to obtain a facial expression feature vector, an audio feature vector, and a physiological state feature vector;

[0130] Step A3: constructing a multimodal fusion vector based on the facial expression feature vector, the audio feature vector, and the physiological state feature vector;

[0131] Step A4: Construct a joint emotion recognition model, and use the multimodal fusion vector to perform emotion classification using the constructed joint emotion recognition model;

[0132] Step A5: The AI system automatically adjusts the cabin environment parameters based on the emotion category and the environment adjustment strategy;

[0133] The joint emotion recognition model uses a method of multimodal feature fusion and deep learning classification to identify the emotional state of the user in the car and achieve the purpose of intelligent dynamic adjustment of the cabin environment accordingly.

[0134] Specifically, step A1 includes:

[0135] Step A1.1: Capture a facial image of the target subject using an in-vehicle camera, and perform preprocessing on the captured facial image, including image denoising, alignment, and cropping, to obtain a preprocessed facial image;

[0136] Step A1.2: collecting the target subject's voice information through the microphone array, and performing pre-processing including noise reduction and frame segmentation on the collected voice information to obtain pre-processed voice information;

[0137] Step A1.3: collecting physiological information of the target subject through the somatosensory monitoring device, and performing preprocessing including denoising and normalization on the collected physiological information to obtain preprocessed physiological information;

[0138] The voice information includes the sound signal of the target object during natural interaction or subjective voice input; the physiological information includes: heart rate, skin electrical response and body surface temperature.

[0139] Specifically, step A2 includes:

[0140] Step A2.1: Recognize facial key points based on the preprocessed facial image, and extract facial expression feature vectors based on the recognized facial key points;

[0141] Step A2.2: Extract audio feature vectors that can reflect emotional state based on the preprocessed speech information, including Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation indicators;

[0142] Step A2.3: Construct a physiological state feature vector with a length of several dimensions based on the preprocessed physiological information.

[0143] Step A3 includes: constructing a multimodal fusion vector based on facial expression feature vectors, audio feature vectors, and physiological state feature vectors by weighting the credibility of different modalities through feature-level splicing, decision-level weighted fusion, or introducing an attention mechanism; and performing emotion classification on the multimodal fusion vector through a joint emotion recognition model.

[0144] Specifically, the method further includes: collecting the target subject's feedback on cabin environment comfort in real time, and automatically adjusting the environment adjustment strategy according to the feedback;

[0145] At the same time, the joint emotion recognition model is optimized based on the collected target subject’s self-reported emotion evaluation.

[0146] Specifically, the method further includes: encrypting the collected multi-source data, and locally storing the encrypted multi-source data.

[0147] This embodiment deeply integrates emotion recognition with in-car environment adjustment, analyzes the driver's emotional state in real time and adjusts the in-car environment to ensure that the driver is always in the most comfortable state.

[0148] Example 4

[0149] Example 4 is a preferred example of Example 1

[0150] According to the present invention, a dynamic calibration method for a vehicle-mounted emotion recognition system is provided. Figure 3 Shown, including:

[0151] Step M1: Acquire multi-source data, including facial images, voice signals, and physiological signals; and pre-process the acquired multi-source data to obtain pre-processed multi-source data;

[0152] Step M2: performing feature extraction based on the preprocessed facial image, speech signal, and physiological signal to obtain a facial expression feature vector, an audio feature vector, and a physiological state feature vector;

[0153] Step M3: Evaluate the current environment credibility based on the environment credibility evaluation function;

[0154] Step M4: Dynamically assign weights to multi-source data based on the current environmental credibility and real-time scenario;

[0155] Step M5: Based on the dynamically assigned weights of the multi-source data, a multimodal fusion vector is constructed according to the facial expression feature vector, the audio feature vector, and the physiological state feature vector, and emotion recognition is performed using the constructed emotion recognition model based on the multimodal fusion vector;

[0156] Step M6: The emotion category automatically adjusts the cabin environment parameters through the AI system according to the environment adjustment strategy.

[0157] This embodiment continuously optimizes the emotion recognition model by collecting driving environment data, driver biometrics, and vehicle scene status information in real time, combined with a dynamic weight allocation algorithm and a closed-loop feedback mechanism. It is suitable for personalized emotion monitoring in complex driving scenarios and can be integrated into the in-vehicle infotainment system (IVI) or advanced driver assistance system (ADAS). It improves the stability and robustness of emotion recognition and maintains a high recognition rate even in complex cabin environment changes such as lighting changes and noise interference.

[0158] Specifically, the step M1 includes:

[0159] Step M1.1: Capture a facial image of the target subject using an in-vehicle camera, and perform preprocessing on the captured facial image, including image denoising, alignment, and cropping, to obtain a preprocessed facial image;

[0160] Step M1.2: collecting the target subject's voice information through the microphone array, and performing preprocessing including noise reduction and frame segmentation on the collected voice information to obtain preprocessed voice information;

[0161] Step M1.3: collecting physiological information of the target subject through the somatosensory monitoring device, and performing preprocessing including denoising and normalization on the collected physiological information to obtain preprocessed physiological information;

[0162] The voice information includes the sound signal of the target object during natural interaction or subjective voice input; the physiological information includes: heart rate, skin electrical response and body surface temperature.

[0163] Specifically, the step M2 includes:

[0164] Step M2.1: Recognize facial key points based on the preprocessed facial image, and extract facial expression feature vectors based on the recognized facial key points;

[0165] Step M2.2: Extract audio feature vectors that can reflect emotional state based on the preprocessed speech information, including Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation indicators;

[0166] Step M2.3: Construct a physiological state feature vector with a length of several dimensions based on the preprocessed physiological information.

[0167] Specifically, the step M3 includes: constructing an environment credibility evaluation function, and using the constructed environment credibility evaluation function to evaluate the credibility of the current environment;

[0168]

[0169] Where L represents the light intensity; R valid Indicates the effective frame rate.

[0170] Specifically, the step M4 includes:

[0171] Step M4.1: Adjust the weights of multi-source data based on the assessed credibility of the current environment and the real-time scenario;

[0172] Step M4.2: Use the improved DS evidence theory to calculate the confidence of the multi-source data weights respectively; when the confidence of a multi-source data weight is less than the preset value, introduce the driving scenario correction coefficient to correct the multi-source data weight to obtain the corrected multi-source data weight.

[0173] Specifically, the weights of the multi-source data in step M4.1 include: face weight W f , speech weight W s and physiological information weight Wp ;W f +W s +W p =1(0≤W f ,W s ,W p ≤1);

[0174] The facial weight W f include:

[0175]

[0176] Among them, W f (0) Assign weight to the base face; k1 is the vehicle speed influence coefficient; Q v,th is the visual quality critical value; h represents the vehicle speed threshold; Z is the normalization factor; in this embodiment, k1=0.1; Q v,th =0.6;

[0177] The speech weight W s include:

[0178]

[0179] Among them, W s (0) Assign weight to basic voice; SNR0 is the reference signal-to-noise ratio; SNR is the signal-to-noise ratio; η∈[0,1] represents the degree of window closing; where 0 = fully open; 1 = fully closed; in this embodiment, SNR0 = 15dB;

[0180] The physiological information weight W p include:

[0181]

[0182] Among them, W p (0) Assign weights to basic physiological information; when deceleration a>0.4g a>0.4g is detected, the weight increment of the physiological signal is: R C is the road complexity index; γ(R C )={0.05R C ,R C ≤4; 0.2+0.1(R C -4), R C >4.

[0183] Specifically, the method further includes: establishing an emotion-behavior association rule base, and when the target object's emotion is identified as "anger" and rapid acceleration occurs within a preset time, the corresponding emotion confidence is increased by ΔC;

[0184]

[0185] Among them, γ is the experience adjustment coefficient; A indicates that rapid acceleration occurs within the preset time, and E indicates that the target object’s emotion is recognized as “anger”.

[0186] This embodiment solves the problems of static calibration models' adaptability in dynamic driving environments, conflicts and misjudgments of multimodal sensor data, and error accumulation caused by the lack of closed-loop optimization through three major mechanisms: dynamic calibration of environmental perception, intelligent arbitration of evidence conflicts, and feedback-driven continuous evolution.

[0187] Example 5

[0188] Example 5 is a preferred example of Example 1

[0189] According to the present invention, a subscription service method for the emotional cockpit function is provided, such as Figure 4 Shown, including:

[0190] Step B1: Collecting multimodal data of the target object, including facial images, voice information, and physiological information; identifying the target object's emotion category based on the collected multimodal data;

[0191] Step B2: Based on the target object's emotion category and the user subscription policy, call the preset adjustment solution corresponding to the current emotion category;

[0192] Step B3: Based on the preset adjustment scheme corresponding to the current emotion category, the corresponding cabin experience service package is activated based on the user's selection.

[0193] Specifically, step B1 includes:

[0194] Step B1.1: Acquire a facial image of a target subject and perform preprocessing on the acquired facial image of the target subject, including denoising, alignment, and cropping, to obtain a preprocessed facial image; perform facial key point recognition based on the preprocessed facial image, and extract a facial expression feature vector based on the recognized facial key points;

[0195] Step B1.2: Acquire the target subject's voice signal and perform noise reduction and frame preprocessing on the acquired voice signal to obtain a preprocessed voice signal; extract audio feature vectors that reflect the emotional state based on the preprocessed voice information, including Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation indicators;

[0196] Step B1.3: Acquire physiological signals of the target subject, perform preprocessing including denoising and normalization on the acquired physiological information to obtain preprocessed physiological information; construct a physiological state feature vector with a length of several dimensions based on the preprocessed physiological information.

[0197] Step B1.4: Construct a multimodal fusion vector based on the facial expression feature vector, audio feature vector, and physiological state feature vector, input the constructed multimodal fusion vector into the multimodal emotion recognition model for emotion classification, and obtain the emotional state label of the target object.

[0198] Specifically, the method also includes: triggering an emotion continuous warning mechanism when the emotion classification is negative for a continuous preset time; and pushing an emotion depth adjustment package that meets the preset requirements through the vehicle interface.

[0199] Specifically, the method further includes: obtaining adjustment feedback of the target object, and adjusting the user subscription strategy according to the adjustment feedback to optimize the intelligent matching.

[0200] This embodiment achieves precise, personalized, and continuous cabin emotion management through the deep integration of multimodal emotion perception and dynamic closed-loop regulation technologies, multi-dimensional data collaboration, intelligent subscription services, and a continuous feedback optimization mechanism. This subscription model also significantly differs from traditional full-featured services by breaking down and packaging different functions into distinct packages, guiding users to experience the entire service system from a single point to a comprehensive level.

[0201] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0202] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for automatically adapting emotional response strategies in multiple climate zones, characterized in that: include: Step S1: Identify the climate zone type that the vehicle is currently in; Step S2: collecting multi-dimensional environmental data reflecting the environment in which the vehicle is located, and constructing a structured environmental state vector based on the multi-dimensional environmental data; Step S3: Collecting multimodal data of the target object, including facial images, voice information, and physiological information; Extracting a feature vector reflecting the emotional state based on the collected multimodal data, and identifying the current emotion category based on the extracted feature vector reflecting the emotional state; Step S4: According to the identified climate zone type, environmental state vector, and emotion category of the current vehicle, the optimal matching strategy template is retrieved from the emotion response strategy library to generate device control parameters.

2. The automatic adaptation method of multi-climate zone emotional response strategy according to claim 1 is characterized in that: The step S1 includes: obtaining the vehicle's geographic location information in real time through the vehicle positioning module; matching the climate zone classification database based on the vehicle's geographic location information to output a climate zone label; The climate zone label includes: climate zone code, climate zone name, regional distribution and climate characteristics.

3. The automatic adaptation method of multi-climate zone emotional response strategy according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Acquire multi-dimensional environmental perception data through vehicle-mounted sensors, wherein the multi-dimensional environmental perception data includes: vehicle interior temperature, vehicle interior humidity, CO2 concentration, light intensity, vehicle exterior temperature, and vehicle exterior humidity; Step S2.2: Accessing the cloud meteorological platform through the communication module to obtain remote meteorological data, wherein the remote meteorological data includes: real-time weather conditions, external temperature and relative humidity, wind speed risk, ultraviolet intensity, and precipitation probability; Step S2.3: constructing an environmental state vector based on the multi-dimensional environmental perception data and remote meteorological data by weighted averaging method; Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code]; Step S2.4: Correct the constructed environment state vector using the credibility correction strategy to obtain a corrected environment state vector.

4. The automatic adaptation method of multi-climate zone emotional response strategy according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Acquire a facial image of the target object, locate the facial region using a face detection model based on the acquired facial image, and extract the facial expression vector using a lightweight neural network model based on the facial region location; Step S3.2: Acquire the sound signal of the target subject during natural interaction or subjective speech input, perform noise reduction and frame processing on the acquired sound signal to obtain a processed sound signal; extract audio feature parameters reflecting the emotional state based on the processed sound signal, including: Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation index; encode the extracted audio feature parameters reflecting the emotional state into a sound feature vector with a length that meets preset requirements; Step S3.3: Acquire physiological signals of the target subject, the physiological signals including heart rate, galvanic skin response, and body surface temperature; perform preprocessing on the acquired physiological signals, including filtering, normalization, and detrending, to obtain preprocessed physiological signals; and construct a physiological state feature vector having a length of several dimensions based on the preprocessed physiological signals; Step S3.4: Construct a multimodal fusion vector based on the facial expression vector, the sound feature vector, and the physiological state feature vector, input the constructed multimodal fusion vector into the multimodal emotion recognition model for emotion classification, and obtain the emotional state label of the target object.

5. The automatic adaptation method of multi-climate zone emotional response strategy according to claim 4, characterized in that: The method of constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector includes: constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector through feature-level splicing, decision-level weighted fusion or introducing an attention mechanism to weight the credibility of different modalities.

6. The automatic adaptation method of multi-climate zone emotional response strategy according to claim 4, characterized in that: When using the multimodal emotion recognition model to perform emotion classification to obtain the emotional state label of the target object, the corresponding confidence score is obtained. When the confidence score is lower than the set threshold, the current emotion result is not used as the basis for strategy triggering; when the confidence score is not lower than the set threshold, it triggers the retrieval of the optimal matching strategy template in the emotion response strategy library and generates device control parameters to enable intelligent control of the device.

7. An automatic adaptation system for multi-climate emotional response strategies, characterized in that: include: Module M1: Identify the climate zone type the vehicle is currently in; Module M2: collects multi-dimensional environmental data reflecting the vehicle's environment and constructs a structured environmental state vector based on the multi-dimensional environmental data; Module M3: Collects multimodal data of the target object, including facial images, voice information, and physiological information; Extracting a feature vector reflecting the emotional state based on the collected multimodal data, and identifying the current emotion category based on the extracted feature vector reflecting the emotional state; Module M4: Based on the identified climate zone type, environmental state vector, and emotion category of the current vehicle, the optimal matching strategy template is retrieved from the emotion response strategy library to generate device control parameters.

8. The automatic adaptation system for multi-climate zone emotional response strategies according to claim 7, characterized in that: The module M1 includes: obtaining the vehicle's geographic location information in real time through the vehicle positioning module; matching the climate zone classification database based on the vehicle's geographic location information to output a climate zone label; The climate zone label includes: climate zone code, climate zone name, regional distribution and climate characteristics.

9. The automatic adaptation system for multi-climate zone emotional response strategies according to claim 7, characterized in that: The module M2 includes: Module M2.1: Acquires multi-dimensional environmental perception data through onboard sensors, including: in-vehicle temperature, in-vehicle humidity, CO2 concentration, light intensity, and outside temperature and humidity. Module M2.2: Access the cloud meteorological platform through the communication module to obtain remote meteorological data, including real-time weather conditions, external temperature and relative humidity, wind speed risk, ultraviolet intensity, and precipitation probability; Module M2.3: Constructing the environmental state vector using the weighted average method based on multi-dimensional environmental perception data and remote meteorological data; Env_Vector = [temperature, humidity, light, CO2, PM2.5, UV index, wind speed, weather type code]; Module M2.4: Correct the constructed environment state vector through the credibility correction strategy to obtain the corrected environment state vector.

10. The automatic adaptation system for multi-climate zone emotional response strategies according to claim 7, characterized in that: The module M3 includes: Module M3.1: Collect facial images of the target subject, locate the facial region using a face detection model based on the collected facial image, and extract facial expression vectors using a lightweight neural network model based on the facial region location. Module M3.2: Collect the target subject's sound signals during natural interaction or subjective speech input, perform noise reduction and frame processing on the collected sound signals to obtain processed sound signals; extract audio feature parameters reflecting the emotional state based on the processed sound signals, including: Mel-frequency cepstral coefficients, intonation contour, fundamental frequency variation, speech rate, and energy variation indicators; encode the extracted audio feature parameters reflecting the emotional state into a sound feature vector with a length that meets preset requirements; Module M3.3: Acquire physiological signals from the target subject, including heart rate, galvanic skin response, and body surface temperature; perform preprocessing on the acquired physiological signals, including filtering, normalization, and detrending, to obtain preprocessed physiological signals; and construct a physiological state feature vector having a length of several dimensions based on the preprocessed physiological signals. Module M3.4: Construct a multimodal fusion vector based on facial expression vectors, voice feature vectors, and physiological state feature vectors. Input the constructed multimodal fusion vector into a multimodal emotion recognition model for emotion classification to obtain the target object's emotional state label. The method of constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector includes: constructing a multimodal fusion vector based on the image expression vector, the sound feature vector and the physiological state feature vector through feature-level splicing, decision-level weighted fusion or introducing an attention mechanism to weight the credibility of different modalities.

Images