In-vehicle environment self-adaptive regulation and control method and system based on emotion recognition

Abstract

The invention provides an in-vehicle environment self-adaptive regulation and control method and system based on emotion recognition. The method comprises the following steps: S1, constructing an emotion recognition model; and S2, according to the emotion category output by the emotion recognition model, adjusting in-vehicle environment parameters through a self-adaptive control algorithm. The emotion recognition model is constructed by fusing multi-modal data such as facial expressions, voice features and physiological signals, and the limitation of a single recognition mode is solved. Based on the recognized emotional state of the driver, the system automatically and accurately adjusts environmental parameters such as in-vehicle temperature, air outlet amount, air outlet mode, seat mode and music playing, and driving experience and driving safety are improved. The method is characterized in that an emotion monitoring technology and an intelligent regulation and control strategy are deeply combined, a personalized and intelligent in-vehicle environment solution is customized for a user by analyzing historical preference data of a driver, and whole-process intelligent regulation and control from emotion perception to environment adaptation are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Vehicles, in particular, to an in-vehicle environment adaptive regulation method and system based on emotion recognition. BACKGROUND

[0002] With the continuous innovation of intelligent cockpit technology, realizing intelligent regulation of in-vehicle environment through emotion recognition has become an important research direction to improve driving experience and driving safety. However, current vehicle-mounted emotion recognition and environment regulation technology still faces many bottlenecks, which restricts its application effect in actual scenarios.

[0003] In the emotion recognition link, traditional technology mostly relies on single modal data, such as emotion judgment through facial expression or voice features only. This single recognition method is obviously disturbed by environmental factors. In complex lighting conditions and large noise interference driving scenarios, it is easy to have recognition errors, resulting in low emotion judgment accuracy and poor environmental adaptability, making it difficult to accurately capture the real emotional state of the driver, and further affecting the effectiveness of the subsequent environment regulation strategy. In addition, single modal recognition lacks comprehensive consideration of the physiological state of the driver, cannot fully reflect the emotional changes, and is difficult to meet the high-precision recognition demand in complex driving scenarios.

[0004] In terms of in-vehicle environment regulation, existing systems generally have the problems of inaccurate regulation response and insufficient individual adaptation ability. Most systems can only make extensive regulation based on preset rules, and cannot dynamically optimize according to the real-time emotional state and individual preferences of the driver. For example, when the driver is in different emotional states, the system cannot accurately match the appropriate temperature, air volume, seat mode and music type, resulting in a disconnection between environment regulation and the actual needs of the driver, and making it difficult to effectively alleviate negative emotions and improve driving comfort and safety. At the same time, due to the lack of real-time feedback mechanism, the existing system cannot timely perceive the regulation effect, and is difficult to adaptively adjust according to the emotional changes of the driver, reducing the intelligent level of environment regulation.

[0005] Through the retrieval of patent documents, it is found that the patent with the publication number CN202410090120.9 discloses an intelligent networked vehicle driver emotion regulation system. The method includes driver emotion detection based on in-vehicle sensors, driver emotion regulation based on a human-machine interface, and communication between vehicles based on vehicle-to-vehicle (V2V) contact and a cloud layer. The cloud layer includes driver emotion detection and the creation of emotion regulation strategies. In addition, an emotion analysis model predicts the driver's emotions from the geometric features extracted from facial skin information and the heart rate extracted from the RGB component changes. The regulation quality of the driver's emotions is analyzed from the driver's subjective experience, behavior, and physiology. The evaluation method of the experimental data of this patent lacks long-term verification in actual driving scenarios, and the system mode is relatively single, which may not meet the emotional regulation needs in complex driving environments.

[0006] In summary, in view of the problems of the prior art, it is a key task to be solved at present to study an in-vehicle environment adaptive regulation method and system based on emotion recognition. SUMMARY

[0007] In view of the defects in the prior art, the purpose of the present application is to provide an in-vehicle environment adaptive regulation method and system based on emotion recognition.

[0008] According to the in-vehicle environment adaptive regulation method based on emotion recognition provided by the present application, the following steps are included: Step S1, constructing an emotion recognition model to recognize the emotion category of the driver through multi-modal data fusion; Step S2, adjusting the in-vehicle environment parameters through an adaptive control algorithm according to the emotion category output by the emotion recognition model, the adaptive control algorithm including the following sub-steps: Step S2.2.1, calculating the environment parameter deviation; Step S2.2.2, generating control instructions based on the environment parameter deviation; Step S2.2.3, combining the generated control instructions into a unified instruction set.

[0009] Preferably, step S1 includes the following sub-steps: Step S1.1, collecting multi-modal data, including facial images, voice signals, and heart rate data, and dividing the multi-modal data into a training set, a validation set, and a test set according to a preset ratio; Step S1.2, extracting facial expression features from the facial images using a convolutional neural network, extracting voice emotion features from the voice signals using a long short-term memory network, and extracting physiological emotion features from the heart rate data using a support vector machine; Step S1.3, performing weighted average fusion of the facial expression features, voice emotion features, and physiological emotion features to generate a comprehensive feature vector; Step S1.4, constructing a fully connected neural network, taking the comprehensive feature vectors of the training set as input features, and taking the corresponding multi-modal data labeled emotion categories as supervision labels, performing model training, and outputting an emotion category including happy, sad, angry, surprised, scared, disgusted, or neutral; Step S1.5, inputting the comprehensive feature vectors of the verification set into the trained fully connected neural network to optimize the model parameters; Step S1.6, solidifying the optimized fully connected neural network parameters, using the test set to evaluate the model performance, and when the emotion recognition accuracy reaches a preset threshold, deploying the complete model including the network structure and optimized parameters to the car machine end to form a runnable emotion recognition model.

[0010] Preferably, in step S1.1, the facial image is captured by a camera; the voice signal is captured by a vehicle-mounted microphone; and the heart rate variability data is obtained by a vehicle-mounted heart rate sensor.

[0011] Preferably, in step S1.4, the facial image emotion category is labeled based on the Facial Action Coding System standard, the voice signal emotion category is labeled based on the standard speech emotion database, and the heart rate data emotion category is labeled based on the heart rate variability index.

[0012] Preferably, step S1.5 includes the following sub-steps: Step S1.5.1, adjusting the neural network weights using the Adam optimization algorithm; Step S1.5.2, dynamically adjusting the learning rate based on the verification set accuracy, and optimizing the hidden layer number and neuron number configuration through the validation loss function value; Step S1.5.3, using cross-validation and regularization methods to prevent overfitting.

[0013] Preferably, step S2 includes the following sub-steps: Step S2.1, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm, and the adaptive control algorithm matches the corresponding target environment parameter set from the preset rule library; Step S2.2, based on the target environment parameter set, the adaptive control algorithm generates a control instruction to adjust the in-vehicle environment parameters; Step S2.3, based on the real-time collected multi-modal data and the emotion recognition model, the emotion category is re-identified, and when the difference value between the output emotion category and the target state exceeds a preset threshold, the adaptive control algorithm generates a correction instruction and triggers a secondary adjustment cycle; Step S2.4, recording the emotion state changes and environment parameter execution results during the adjustment process for updating the strategy parameters of the adaptive control algorithm.

[0014] Preferably, in step S2.1, the emotion category outputted by the emotion recognition model is transmitted to the adaptive control algorithm through the vehicle communication bus, and the target environment parameter set comprises a target temperature value, a target air volume value, a target air mode code, a target seat mode code and a target music playlist identifier.

[0015] Preferably, in step S2.2, the control instructions generated by the adaptive control algorithm are transmitted to each actuator through the vehicle CAN bus: the air conditioning system controls the compressor / heater power according to the target temperature value; the fan control system adjusts the motor speed according to the target air volume value; the damper actuator adjusts the angle of the air deflector according to the target air mode code; the seat controller activates the corresponding massage and ventilation functions according to the target seat mode code; and the vehicle audio system switches the music content according to the music playlist identifier.

[0016] Preferably, in step S2.2.1, for continuous parameters, the difference between the target value and the current value is calculated: ,

[0017] wherein, represents the temperature deviation, represents the air volume deviation, is the target indoor temperature, is the current indoor temperature, is the target air volume, is the current air conditioning air volume. For discrete parameters, it is determined whether the target value and the current value are consistent, and if not, a switching operation is performed. The judgment function is as follows:

[0018]

[0019]

[0020] wherein, is the current air mode code, is the current seat mode code, is the current music playlist identifier, is the target air mode code. is the target seat mode code. is the target music playlist identifier. In step S2.2.2, for continuous variable processing, a proportional regulation method is used to generate control instructions:

[0021]

[0022] wherein, are the proportional adjustment coefficients of temperature and air volume respectively.

[0023] For discrete variables, if the target value is inconsistent with the current value, an instruction switching to the target mode is directly generated:

[0024]

[0025]

[0026] In the step S2.2.3, the instruction set is represented as:

[0027] The instruction set is sent to the corresponding actuator through the vehicle communication bus to complete the adjustment of the in-vehicle environmental parameters.

[0028] The application also provides an in-vehicle environment adaptive control system based on emotion recognition, comprising: Module M1, an emotion recognition model is constructed, and the emotion categories of the driver are recognized through multi-modal data fusion; Module M2, according to the emotion categories output by the emotion recognition model, the in-vehicle environmental parameters are adjusted through an adaptive control algorithm, and the adaptive control algorithm comprises the following sub-modules: Module M2.2.1, calculating the environmental parameter deviation; Module M2.2.2, generating control instructions based on the environmental parameter deviation; Module M2.2.3, combining the generated control instructions into a unified instruction set.

[0029] Compared with the prior art, the application has the following beneficial effects: 1. The multi-modal emotion recognition model constructed by the application integrates facial expressions, speech features and physiological signals, effectively solving the problem of low accuracy and poor environmental adaptability of single emotion recognition method. The model uses deep learning algorithm to comprehensively analyze the driver's facial micro-expression, speech tone change and heart rate and other physiological parameters, realizes deep fusion of multi-source data, improves the accuracy and robustness of emotion recognition, and can still accurately judge the driver's emotional state in complex light, noisy and other harsh environments, providing reliable data support for subsequent in-vehicle environment adjustment.

[0030] 2、The application successfully solves the problems of inaccurate response and insufficient personalized adaptation ability of in-vehicle environment adjustment after emotion recognition by the application of the adaptive control algorithm. The algorithm dynamically adjusts key environmental parameters such as in-vehicle temperature, air volume, air mode, seat mode and music playing based on the recognized driver's emotional state (such as happy, sad, angry, etc.). At the same time, combined with the historical preference data of the driver, the adjustment strategy is optimized individually. Through the real-time feedback mechanism, the system can continuously monitor the effect of environmental adjustment, and flexibly adjust the control strategy according to the real-time changes of the driver's emotion, to ensure that the in-vehicle environment quickly and accurately meets the psychological and physiological needs of the driver, significantly improving the driving comfort and driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0031] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, made with reference to the accompanying drawings: Figure 1 A flowchart of the in-vehicle environment adaptive control system based on emotion recognition in the embodiment of the present application; Figure 2 An emotion recognition model flowchart in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of changes and improvements can be made. These all belong to the protection scope of the present application.

[0033] The present application solves the limitations of single recognition mode by fusing facial expression, speech features and physiological signals and other multi-modal data to build a high-precision emotion recognition model. Based on the recognized driver's emotional state, the system automatically and accurately adjusts the environmental parameters such as in-vehicle temperature, air volume, air mode, seat mode and music playing, to improve the driving experience and driving safety. The core innovation lies in the deep combination of emotion monitoring technology and intelligent control strategy, which analyzes the historical preference data of the driver to customize personalized and intelligent in-vehicle environment solutions for users, realizing the whole-process intelligent control from emotion perception to environment adaptation.

[0034] Example 1: Figure 1 A flowchart of the in-vehicle environment adaptive control system based on emotion recognition in the embodiment of the present application.

[0035] As shown in Figure 1 , the present embodiment proposes an in-vehicle environment adaptive control method based on emotion recognition, including the following steps: Step S1, construct an emotion recognition model to recognize the emotion category of the driver through multi-modal data fusion.

[0036] In this embodiment, emotion feature extraction is performed by fusing facial expression, speech feature and physiological signal (such as heart rate), and a deep learning algorithm is used to complete classification, thereby constructing an emotion recognition model suitable for driving scenarios. The emotion recognition model aims to improve the accuracy of emotion recognition and enhance the adaptability to different driving environments and individual differences, providing reliable data support for subsequent in-vehicle environment regulation.

[0037] Figure 2 The flowchart of the emotion recognition model in the embodiment of the application.

[0038] As shown in Figure 2 Step S1 includes the following sub-steps: Step S1.1, collect multi-modal data, including facial images, speech signals and heart rate data, establish an emotion feature database, and divide the multi-modal data into a training set, a validation set and a test set according to a preset ratio.

[0039] Further, in step S1.1, facial images are collected by a camera, speech signals are collected by a vehicle-mounted microphone, and heart rate variability data are obtained by a vehicle-mounted heart rate sensor.

[0040] Step S1.2, facial expression features are extracted from facial images using a convolutional neural network (CNN), speech emotion features are extracted from speech signals using a long short-term memory network (LSTM), and physiological emotion features are extracted from heart rate data using a support vector machine.

[0041] Specifically, computer vision technology is used to extract facial key points to capture subtle features reflecting emotional changes such as smiling, frowning and opening mouth; since the emotional expression of speech is usually reflected in tone, speech rate, volume, etc., features such as mel-frequency cepstral coefficient (MFCC) and speech tone are extracted to represent the changes in emotion, and the fluctuation of heart rate can directly reflect the emotional categories of the driver such as tension, anxiety or relaxation.

[0042] Step S1.3, weighted average fusion of facial expression features, speech emotion features and physiological emotion features to generate a comprehensive feature vector.

[0043] Step S1.4, construct a fully connected neural network, use the comprehensive feature vector of the training set as the input feature, and use the emotion category labeled by the corresponding multi-modal data as the supervision label to perform model training.

[0044] Specifically, the fully connected neural network comprises 3 hidden layers, adopts ReLU activation function, and the output layer is a 7-dimensional softmax classifier to generate emotion categories including happy, sad, angry, surprised, scared, disgusted or neutral.

[0045] Further, in step S1.4, the facial image emotion category is labeled based on the Facial Action Coding System (FACS) standard, the speech signal emotion category is labeled based on the standard speech emotion database, and the heart rate data emotion category is labeled based on the heart rate variability (HRV) index.

[0046] In step S1.5, the comprehensive feature vector of the verification set is input into the trained fully connected neural network to optimize the model parameters.

[0047] Further, step S1.5 includes the following sub-steps: Step S1.5.1, adjusting the neural network weight using the Adam optimization algorithm; Step S1.5.2, dynamically adjusting the learning rate based on the verification set accuracy, and optimizing the hidden layer number and neuron number configuration through the verification loss function value; Step S1.5.3, using cross-validation and regularization methods to prevent overfitting.

[0048] In step S1.6, the optimized fully connected neural network parameters are solidified, and the model performance is evaluated using the test set. When the emotion recognition accuracy reaches the preset threshold, the complete model containing the network structure and optimized parameters is deployed to the car machine end to form a runnable emotion recognition model.

[0049] In step S2, according to the emotion category output by the emotion recognition model, the in-vehicle environment parameters are adjusted through an adaptive control algorithm to improve driving safety and comfort.

[0050] Specifically, step S2 includes the following sub-steps: Step S2.1, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm, and the adaptive control algorithm matches the corresponding target environment parameter set from the preset rule library.

[0051] Specifically, various sensors are arranged in the vehicle to obtain the driver state and in-vehicle environment information. The driver's facial expression is collected by a camera, the speech features are collected by a speech collection device, and the heart rate data and other physiological signals are obtained by a heart rate sensor, so as to comprehensively perceive the driver's emotion category; at the same time, the in-vehicle environment parameters are collected in real time, including the in-vehicle temperature value detected by the temperature sensor, the air volume value detected by the air volume sensor, the sitting posture state detected by the seat pressure sensor, and the music playing state obtained by the audio collection device.

[0052] In this embodiment, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm through the vehicle communication bus, and the target environment parameter set includes a target temperature value, a target air volume value, a target air mode code, a target seat mode code, and a target music playlist identifier.

[0053] Specifically, the adaptive control algorithm receives the following input parameters: current emotion category (output by the emotion recognition module), a current real-time environment parameter set: wherein : current vehicle interior temperature; : current air conditioner air volume; : current air mode code (discrete numerical value, such as 0 = face blowing, 1 = foot blowing, etc.); : current seat mode code (discrete numerical value); : current music playlist identifier.

[0054] According to the emotion category , the corresponding target environment parameter set is found in the preset rule base: wherein : target vehicle interior temperature; : target air volume; : target air mode code (discrete numerical value, such as 0 = face blowing, 1 = foot blowing, etc.); target seat mode code (discrete numerical value); : target music playlist identifier.

[0055] In this embodiment, the preset rule base is shown in the following table:

[0056] In this embodiment, the control instructions generated by the adaptive control algorithm are transmitted to each execution mechanism through the vehicle CAN bus: the air conditioning system controls the compressor / heater power according to the target temperature value; the fan control system adjusts the motor speed according to the target air volume value; the damper actuator adjusts the angle of the air deflector according to the target air mode code; the seat controller activates the corresponding massage and ventilation functions according to the target seat mode code; and the vehicle audio system switches the music content according to the music playlist identifier.

[0057] Specifically, step S2.2 includes the following sub-steps: Step S2.2.1, calculate the environment parameter deviation: For continuous parameters (temperature, air volume), calculate the difference between the target value and the current value:​ ,

[0058] wherein, represents the temperature deviation, represents the air volume deviation.

[0059] For discrete parameters (mode code, playlist identification), determine whether the target value is consistent with the current value, and if not, perform the switching operation. The judgment function is as follows:

[0060]

[0061]

[0062] Step S2.2.2, generate control instructions: 1) For continuous variable processing (temperature, air volume), generate control instructions using proportional regulation:

[0063]

[0064] wherein, are the proportional regulation coefficients of temperature and air volume, respectively.

[0065] 2) For discrete variables (air outlet mode, seat mode, music playlist), if the target value is not consistent with the current value, generate an instruction to switch to the target mode directly:

[0066]

[0067]

[0068] That is: If the current air outlet mode and the target mode are inconsistent, switch to the target mode; If the current seat mode and the target mode are inconsistent, switch to the target mode; If the current music playing mode and the target mode are inconsistent, switch to the target mode.

[0069] Step S2.2.3, combine all generated control instructions into a unified instruction set:

[0070] And send the instruction set to the corresponding actuator through the vehicle communication bus to complete the adjustment of the in-vehicle environment parameters.

[0071] Step S2.3, based on real-time collected multi-modal data and emotion recognition model, perform emotion category re-identification, when the difference value of the output emotion category and the target state exceeds the preset threshold, the adaptive control algorithm generates a correction instruction and triggers a secondary adjustment cycle.

[0072] In this embodiment, after the adjustment action is performed, the sensor continuously monitors the driver's state and the change of the in-vehicle environment. If it is found that the driver's emotion has not improved as expected, or the in-vehicle environment parameters have not reached the set standard, the sensor will feed back the new data to the controller. The controller re-evaluates and adjusts the control instruction to drive the actuator to act again, forming a closed-loop dynamic adjustment process to ensure that the in-vehicle environment always adapts to the driver's emotion.

[0073] Step S2.4, record the changes of emotional state and the execution results of environmental parameters during the adjustment process, for updating the strategy parameters of the adaptive control algorithm.

[0074] In this embodiment, the driver's emotion category data and the reaction data after the in-vehicle environment adjustment are continuously collected during the entire adjustment process. These data are stored in the vehicle-mounted data storage unit for subsequent learning and optimization. The control system is built-in with machine learning algorithm, which continuously adjusts and optimizes the control strategy of the adaptive control algorithm through analysis and learning of the stored data, to better adapt to the preferences and emotional reactions of different drivers.

[0075] Embodiment 2: The application also provides an in-vehicle environment adaptive control system based on emotion recognition, which can be realized by performing the flow steps of the in-vehicle environment adaptive control method based on emotion recognition, that is, the in-vehicle environment adaptive control method based on emotion recognition can be understood by those skilled in the art as the preferred embodiment of the in-vehicle environment adaptive control system based on emotion recognition.

[0076] Specifically, the in-vehicle environment adaptive control system based on emotion recognition comprises: Module M1, constructing an emotion recognition model to identify the emotion category of the driver through multi-modal data fusion.

[0077] Module M2, adjusting the in-vehicle environment parameters through an adaptive control algorithm according to the emotion category output by the emotion recognition model, the adaptive control algorithm comprising the following sub-modules: Module M2.2.1, calculating the environmental parameter deviation; Module M2.2.2, generating a control instruction based on the environmental parameter deviation; Module M2.2.3, combining the generated control instructions into a unified instruction set.

[0078] In particular, the module M1 comprises the following sub-modules: The module M1.1 collects multi-modal data including facial images, voice signals and heart rate data, establishes an emotional feature database, and divides the multi-modal data into a training set, a validation set and a test set according to a preset ratio.

[0079] Further, in the module M1.1, the facial images are collected by a camera, the voice signals are collected by a vehicle-mounted microphone, and the heart rate variability data are obtained by a vehicle-mounted heart rate sensor.

[0080] The module M1.2 extracts facial expression features from facial images using a convolutional neural network (CNN), extracts voice emotion features from voice signals using a long short-term memory network (LSTM), and extracts physiological emotion features from heart rate data using a support vector machine.

[0081] In particular, computer vision technology is used to extract facial key points to capture subtle features reflecting emotional changes such as smiling, frowning and opening mouth. Since the emotional expression of voice is usually reflected in tone, speed, volume, etc., the change of emotion is represented by extracting Mel frequency cepstral coefficients (MFCC) and voice tone features. The fluctuation of heart rate can directly reflect the emotional categories of the driver such as tension, anxiety or relaxation.

[0082] The module M1.3 performs weighted average fusion of the facial expression features, voice emotion features and physiological emotion features to generate a comprehensive feature vector.

[0083] The module M1.4 constructs a fully connected neural network, takes the comprehensive feature vector of the training set as the input feature, and takes the emotional category labeled by the corresponding multi-modal data as the supervision label to perform model training.

[0084] In particular, the fully connected neural network contains 3 hidden layers, uses a ReLU activation function, and the output layer is a 7-dimensional softmax classifier to generate emotional categories including happy, sad, angry, surprised, frightened, disgusted or neutral.

[0085] Further, in the module M1.4, the facial image emotional categories are labeled based on the Facial Action Coding System (FACS) standard, the voice signal emotional categories are labeled based on the standard speech emotion database, and the heart rate data emotional categories are labeled based on the heart rate variability (HRV) index.

[0086] The module M1.5 inputs the comprehensive feature vector of the validation set into the trained fully connected neural network to optimize the model parameters.

[0087] Further, the module M1.5 comprises the following sub-modules: The module M1.5.1 adjusts the neural network weights using the Adam optimization algorithm. Module M1.5.2, dynamically adjusting learning rate based on validation set accuracy, optimizing hidden layer number and neuron number configuration through validation loss function value; Module M1.5.3, using cross-validation and regularization method to prevent overfitting.

[0088] Module M1.6, solidifying the optimized fully connected neural network parameters, using the test set to evaluate the model performance, when the emotion recognition accuracy reaches the preset threshold, deploying the complete model containing network structure and optimization parameters to the car end to form a runnable emotion recognition model.

[0089] Module M2, according to the emotion category output by the emotion recognition model, adjusting the in-vehicle environment parameters through the adaptive control algorithm to improve driving safety and comfort.

[0090] Specifically, module M2 includes the following sub-modules: Module M2.1, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm, and the adaptive control algorithm matches the corresponding target environment parameter set from the preset rule library.

[0091] Specifically, various sensors are arranged in the vehicle to obtain driver state and in-vehicle environment information. The driver's facial expression is collected by a camera, the voice features are collected by a voice collection device, and the heart rate data and other physiological signals are obtained by a heart rate sensor, so as to comprehensively perceive the emotion category of the driver; at the same time, the in-vehicle environment parameters are collected in real time, including the in-vehicle temperature value detected by the temperature sensor, the air volume value detected by the air volume sensor, the sitting posture state detected by the seat pressure sensor, and the music playing state obtained by the audio collection device.

[0092] In this embodiment, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm through the vehicle-mounted communication bus, and the target environment parameter set includes target temperature value, target air volume value, target air volume mode code, target seat mode code and target music playlist identifier.

[0093] Specifically, the adaptive control algorithm receives the following input parameters: Current emotion category (output by the emotion recognition module), current real-time environment parameter set: , wherein : current in-vehicle temperature; : current air conditioner air volume; : current air volume mode code (discrete numerical value, such as 0 = face blowing, 1 = foot blowing, etc.); : current seat mode code (discrete numerical value); : current music playlist identifier.

[0094] According to the emotion category , find the corresponding target environment parameter set in the preset rule base: , wherein : target indoor temperature; : target air volume; : target air outlet mode code (discrete numerical value, such as 0 = face blowing, 1 = foot blowing, etc.); target seat mode code (discrete numerical value); : target music playlist identifier.

[0095] In this embodiment, the preset rule base is shown in the following table:

[0096] In this embodiment, the control instructions generated by the adaptive control algorithm are transmitted to each actuator through the vehicle CAN bus: the air conditioning system controls the compressor / heater power according to the target temperature value; the fan control system adjusts the motor speed according to the target air volume value; the damper actuator adjusts the angle of the air deflector according to the target air outlet mode code; the seat controller activates the corresponding massage and ventilation functions according to the target seat mode code; the car audio system switches the music content according to the music playlist identifier.

[0097] Specifically, the module M2.2 includes the following sub-modules: Module M2.2.1, calculate the environment parameter deviation: For continuous parameters (temperature, air volume), calculate the difference between the target value and the current value: ,

[0098] wherein, represents the temperature deviation, represents the air volume deviation.

[0099] For discrete parameters (mode code, playlist identifier), determine whether the target value and the current value are consistent, and perform switching operation when they are not consistent. The judgment function is as follows:

[0100]

[0101]

[0102] Module M2.2.2, generate control instructions: 1) For continuous variable processing (temperature, air volume), use proportional regulation to generate control instructions:​

[0103]

[0104] wherein, are the proportional adjustment coefficients of temperature and air volume respectively.

[0105] 2) For discrete variables (air mode, seat mode, music playlist), if the target value is inconsistent with the current value, directly generate an instruction to switch to the target mode:

[0106]

[0107]

[0108] That is: If the current air mode and the target mode are inconsistent, switch to the target mode; If the current seat mode and the target mode are inconsistent, switch to the target mode; If the current music playing mode and the target mode are inconsistent, switch to the target mode.

[0109] Module M2.2.3, combine all generated control instructions into a unified instruction set:

[0110] And send the instruction set to the corresponding actuator through the vehicle communication bus to complete the adjustment of the in-vehicle environment parameters.

[0111] Module M2.3, based on real-time collected multi-modal data and emotion recognition model, perform emotion category re-identification, when the difference value between the output emotion category and the target state exceeds the preset threshold, the adaptive control algorithm generates a correction instruction and triggers a secondary adjustment cycle.

[0112] In this embodiment, after the adjustment action is executed, the sensor continuously monitors the driver's state and the change of the in-vehicle environment. If it is found that the driver's emotion has not improved as expected, or the in-vehicle environment parameters have not reached the set standard, the sensor will feedback the new data to the controller. The controller re-evaluates and adjusts the control instruction, and drives the actuator to act again, forming a closed-loop dynamic adjustment process, ensuring that the in-vehicle environment is always adapted to the driver's emotion.

[0113] Module M2.4, record the changes of emotional state and environmental parameter execution results during the adjustment process, for updating the strategy parameters of the adaptive control algorithm.

[0114] In this embodiment, the driver's emotional category data is continuously collected throughout the adjustment process, as well as the reaction data after the adjustment of the in-vehicle environment. These data are stored in the on-board data storage unit for subsequent learning and optimization. The regulation system is built-in with machine learning algorithms that continuously adjust and optimize the control strategy of the adaptive control algorithm through analysis and learning of the stored data, to better adapt to the preferences and emotional reactions of different drivers.

[0115] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module, unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module, unit thereof can also be realized in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module, unit thereof can be considered as a hardware component, and the devices, modules, units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules, units for achieving various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0116] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A method for adaptive control of the in-vehicle environment based on emotion recognition, characterized in that, Includes the following steps: Step S1: Construct an emotion recognition model to identify the driver's emotion category through multimodal data fusion; Step S2: Based on the emotion category output by the emotion recognition model, adjust the in-vehicle environment parameters using an adaptive control algorithm. The adaptive control algorithm includes the following sub-steps: Step S2.2.1: Calculate the environmental parameter deviations; Step S2.2.2: Generate control commands based on the environmental parameter deviations; Step S2.2.3: Combine the generated control instructions into a unified instruction set.

2. The in-vehicle environment adaptive control method based on emotion recognition according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S1.1: Collect multimodal data, which includes facial images, voice signals and heart rate data. The multimodal data is divided into training set, validation set and test set according to a preset ratio. Step S1.2: Use a convolutional neural network to extract facial expression features from the facial image, use a long short-term memory network to extract speech emotion features from the speech signal, and use a support vector machine to extract physiological emotion features from the heart rate data. Step S1.3: The facial expression features, the voice emotion features, and the physiological emotion features are weighted and averaged to generate a comprehensive feature vector; Step S1.4: Construct a fully connected neural network, using the comprehensive feature vector of the training set as the input feature and the corresponding multimodal data labeled emotion category as the supervision label, to train the model. The output layer generates emotion categories including happy, sad, angry, surprised, fearful, disgusted, or neutral. Step S1.5: Input the comprehensive feature vector of the validation set into the trained fully connected neural network to optimize the model parameters; Step S1.6: Solidify the optimized fully connected neural network parameters, evaluate the model performance using the test set, and when the emotion recognition accuracy reaches a preset threshold, deploy the complete model, including the network structure and optimized parameters, to the vehicle terminal to form a runnable emotion recognition model.

3. The in-vehicle environment adaptive control method based on emotion recognition according to claim 2, characterized in that, In step S1.1, facial images are captured by a camera; voice signals are captured by a vehicle microphone; and heart rate variability data are obtained by a vehicle heart rate sensor.

4. The in-vehicle environment adaptive control method based on emotion recognition according to claim 2, characterized in that, In step S1.4, facial image emotion categories are labeled based on the facial motion coding system standard, speech signal emotion categories are labeled based on the standard speech emotion database, and heart rate data emotion categories are labeled based on the heart rate variability index.

5. The in-vehicle environment adaptive control method based on emotion recognition according to claim 2, characterized in that, Step S1.5 includes the following sub-steps: Step S1.5.1: Adjust the neural network weights using the Adam optimization algorithm; Step S1.5.2: Dynamically adjust the learning rate based on the accuracy of the validation set, and optimize the configuration of the number of hidden layers and neurons by using the validation loss function value; Step S1.5.3: Cross-validation and regularization are used to prevent overfitting.

6. The in-vehicle environment adaptive control method based on emotion recognition according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: The emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm, and the adaptive control algorithm matches the corresponding target environment parameter set from the preset rule base; Step S2.2: Based on the target environmental parameter set, the adaptive control algorithm generates control commands to adjust the in-vehicle environmental parameters; Step S2.3: Based on the real-time collected multimodal data and the emotion recognition model, the emotion category is re-identified. When the difference between the output emotion category and the target state exceeds a preset threshold, the adaptive control algorithm generates a correction instruction and triggers a secondary adjustment cycle. Step S2.4: Record the changes in emotional state and the execution results of environmental parameters during the adjustment process, which are used to update the strategy parameters of the adaptive control algorithm.

7. The in-vehicle environment adaptive control method based on emotion recognition according to claim 6, characterized in that, In step S2.1, the emotion category output by the emotion recognition model is transmitted to the adaptive control algorithm through the vehicle communication bus. The target environment parameter set includes the target temperature value, the target air volume value, the target air outlet mode code, the target seat mode code, and the target music playlist identifier.

8. The in-vehicle environment adaptive control method based on emotion recognition according to claim 6, characterized in that, In step S2.2, the control commands generated by the adaptive control algorithm are transmitted to each actuator via the vehicle CAN bus: the air conditioning system controls the compressor / heater power according to the target temperature value; the fan control system adjusts the motor speed according to the target airflow value; and the damper actuator adjusts the air guide plate angle according to the target airflow mode code. The seat controller activates the corresponding massage and ventilation functions according to the target seat mode code; the in-vehicle audio system switches music content according to the music playlist identifier.

9. The in-vehicle environment adaptive control method based on emotion recognition according to claim 8, characterized in that, In step S2.2.1, for continuous parameters, the difference between the target value and the current value is calculated: ， in, Indicates temperature deviation. Indicates the deviation in air volume. Target interior temperature, This is the current interior temperature of the vehicle. For the target air volume, Current air conditioning fan speed; For discrete parameters, it is determined whether the target value is consistent with the current value. If they are inconsistent, a switching operation is performed. The determination function is as follows: in, This is the code for the current airflow mode. This is the current seat mode code. This serves as an identifier for the current music playlist. Code for the target airflow mode; For the target seat mode code; Identify the target music playlist; In step S2.2.2, for continuous variable processing, a proportional adjustment method is used to generate control commands: in, These are the proportional adjustment coefficients for temperature and air volume, respectively. For discrete variables, if the target value is inconsistent with the current value, an instruction to switch to the target mode is generated directly: In step S2.2.3, the instruction set is represented as follows: The set of instructions is sent to the corresponding actuators via the vehicle communication bus to adjust the in-vehicle environmental parameters.

10. An in-vehicle environment adaptive control system based on emotion recognition, employing the in-vehicle environment adaptive control method based on emotion recognition as described in any one of claims 1-9, characterized in that, include: Module M1 constructs an emotion recognition model to identify the driver's emotion category through multimodal data fusion; Module M2 adjusts the in-vehicle environment parameters based on the emotion category output by the emotion recognition model using an adaptive control algorithm. The adaptive control algorithm includes the following sub-modules: Module M2.2.1 calculates the deviation of environmental parameters; Module M2.2.2 generates control commands based on the environmental parameter deviations; Module M2.2.3 combines the generated control commands into a unified instruction set.

Images