New energy automobile driver emotion adjusting method and system based on multi-modal interaction

Abstract

The invention provides a new energy automobile driver emotion adjusting method and system based on multi-modal interaction, and belongs to the technical field of new energy automobile control. Comprising the following steps: acquiring physiological data of a driver and running data of a vehicle to form a multi-modal original data stream; carrying out encryption transmission and preprocessing on the multi-modal original data stream, and identifying the emotional state of the driver through a time sequence-space dual-channel fusion network; and determining an optimal emotion regulation scheme from a preset emotion regulation strategy library in combination with the personal preference of the driver and the current driving scene, controlling equipment in the vehicle to perform corresponding execution, and dynamically updating an emotion regulation strategy according to the execution condition. According to the method, accurate recognition and personalized adjustment of the driver emotion can be realized, and the driving safety and the user experience are improved.

Description

Technical Field

[0001] This invention belongs to the field of new energy vehicle control technology, and in particular relates to a method and system for regulating the emotions of new energy vehicle drivers based on multimodal interaction. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of new energy vehicles and smart cockpit technology, the impact of driver's emotional state on driving safety and passenger experience is receiving increasing attention. Existing technologies include some solutions for driver emotion recognition based on a single modality (such as facial recognition or voice analysis alone), and attempts to intervene in emotions by playing music, adjusting lights, and other methods.

[0004] However, such existing technologies still have obvious drawbacks: First, the emotion recognition method is singular and easily interfered with in complex in-vehicle environments, resulting in inaccurate recognition results and poor stability; second, there is a lack of linkage with dynamic driving scenarios (such as rapid acceleration and emergency braking), which may recommend unsafe adjustment schemes (such as loud music interfering with driving), which may lead to the risk of distracting the driver and affecting driving safety. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a method and system for regulating the emotions of new energy vehicle drivers based on multimodal interaction, which can achieve accurate identification and personalized regulation of driver emotions, thereby improving driving safety and user experience.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for regulating the emotions of new energy vehicle drivers based on multimodal interaction.

[0007] A method for regulating the emotions of new energy vehicle drivers based on multimodal interaction includes: Acquire driver physiological data and vehicle operational data to form a multimodal raw data stream; The multimodal raw data stream is encrypted and preprocessed, and the driver's emotional state is identified through a time-space dual-channel fusion network. At the same time, the driver's sensitive data during the identification process is encrypted and stored locally, and only the de-identified emotion tags are uploaded. Based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario, the optimal emotional regulation plan is determined from the preset emotional regulation strategy library, and the corresponding in-vehicle equipment is controlled to execute accordingly. The effectiveness of the regulation strategy is evaluated by monitoring changes in the driver's emotions in real time, and the emotion regulation strategy is dynamically updated based on the evaluation results.

[0008] Furthermore, the physiological data includes: visual data including the driver's facial image and posture data; voice data including the driver's voice signal and voiceprint features; and physiological manifestation data including the driver's heart rate and skin conductance data. The vehicle's operational data includes: vehicle speed, acceleration, and navigation data.

[0009] Furthermore, the temporal-spatial dual-channel fusion network includes a temporal channel and a spatial channel; wherein, the temporal channel is used to process temporal data including speech signals and physiological signals; the spatial channel is used to process spatial data including facial images and posture data, and outputs emotional states through weighted fusion via an attention mechanism.

[0010] Furthermore, the emotion regulation strategy library includes music recommendation, ambient lighting adjustment, fragrance release, seat linkage, and voice interaction.

[0011] Furthermore, the selection of the optimal emotion regulation scheme is adjusted based on the safety level of the driving scenario to limit the intensity of the emotion regulation scheme; wherein, the safety level is divided according to vehicle speed, road conditions and weather conditions.

[0012] Furthermore, the safety levels of the driving scenarios include safe scenarios, medium-risk scenarios, and high-risk scenarios; in high-risk scenarios, non-intrusive adjustment methods are preferred.

[0013] Furthermore, the dynamic updating of emotion regulation strategies based on the evaluation results includes: short-term feedback monitoring and long-term feedback analysis. The short-term feedback monitoring monitors changes in the emotion index and switches alternative strategies within a preset short period, while the long-term feedback analysis analyzes user preference migration patterns within a preset long period and updates the emotion regulation strategy library.

[0014] The second aspect of this invention provides a new energy vehicle driver emotion regulation system based on multimodal interaction.

[0015] A new energy vehicle driver emotion regulation system based on multimodal interaction includes: The multimodal data acquisition module is configured to acquire the driver's physiological data and the vehicle's operational data to form a multimodal raw data stream; The edge computing and emotion recognition module is configured to: encrypt and preprocess the multimodal raw data stream, and identify the driver's emotional state through a time-space dual-channel fusion network; at the same time, the driver's sensitive data during the recognition process is locally encrypted and stored, and only the desensitized emotion tags are uploaded. The intelligent recommendation module is configured to determine the optimal emotion regulation scheme from a preset emotion regulation strategy library based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario. The execution module is configured to control the in-vehicle devices to execute the selected optimal emotion regulation scheme. The feedback evaluation module is configured to evaluate the regulation effect by monitoring the driver's emotional changes in real time, and dynamically update the emotion regulation strategy based on the evaluation results. A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the method for regulating the emotions of a new energy vehicle driver based on multimodal interaction as described in the first aspect of the present invention.

[0016] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the method for regulating the emotions of a new energy vehicle driver based on multimodal interaction as described in the first aspect of the present invention.

[0017] The above one or more technical solutions have the following beneficial effects: This invention first acquires the driver's physiological data and the vehicle's operational data to form a multimodal raw data stream. Then, it encrypts and preprocesses the multimodal raw data stream, and identifies the driver's emotional state through a temporal-spatial dual-channel fusion network. By comprehensively utilizing multi-dimensional data such as the driver's facial images, voice signals, and physiological signals, and performing comprehensive analysis based on the temporal-spatial dual-channel fusion network, it overcomes the perceptual limitations of single modalities in specific scenarios (such as insufficient lighting or noisy environments), significantly reduces environmental interference, and thus achieves more accurate and stable identification of the driver's emotional state.

[0018] This invention determines the optimal emotion regulation scheme from a pre-set emotion regulation strategy library based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario. By acquiring real-time vehicle operation data and environmental information, the current driving scenario is classified into safety levels, and the recommendation logic of the emotion regulation strategy is dynamically adjusted accordingly. In high-risk scenarios, potentially distracting adjustment methods (such as loud music) will be automatically avoided, and non-intrusive safety solutions (such as ambient lighting adjustment and low-frequency voice prompts) will be prioritized, thereby improving the driver's mood while fundamentally ensuring their driving safety.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart of the new energy vehicle driver emotion regulation method based on multimodal interaction in Embodiment 1 of the present invention.

[0022] Figure 2 This is a modular flowchart of the new energy vehicle driver emotion regulation system based on multimodal interaction in Embodiment 2 of the present invention. Detailed Implementation

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] Example 1 This embodiment discloses a method for regulating the emotions of new energy vehicle drivers based on multimodal interaction.

[0027] like Figure 1 As shown, the method for regulating the emotions of new energy vehicle drivers based on multimodal interaction includes: Step S1: Acquire the driver's physiological data and the vehicle's operational data to form a multimodal raw data stream; Step S2: Encrypt and preprocess the multimodal raw data stream, and identify the driver's emotional state through a time-space dual-channel fusion network; at the same time, the driver's sensitive data during the identification process is locally encrypted and stored, and only the desensitized emotion tags are uploaded. Step S3: Based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario, determine the optimal emotional regulation scheme from the preset emotional regulation strategy library, and control the corresponding execution of the in-vehicle equipment; Step S4: Evaluate the regulation effect by monitoring the driver's emotional changes in real time, and dynamically update the emotion regulation strategy based on the evaluation results.

[0028] Based on the above process, this invention can accurately identify and personalize driver emotions, thereby improving driving safety and user experience. To facilitate understanding of the technical solution of this invention, the specific implementation methods of this invention will be further explained and described below.

[0029] In step S1, the driver's physiological data and the vehicle's operational data are acquired to form a multimodal raw data stream.

[0030] Visual data, including the driver's facial image and posture data, is acquired through a visual sensor. Specifically, an in-vehicle DMS (Driver Monitoring System) camera with a resolution of 1080P and a frame rate of 30fps is used as the visual sensor. It is installed behind the steering wheel and uses a wide-angle lens to cover the driver's face and upper body posture for visual data acquisition.

[0031] The system acquires voice data, including the driver's voice signal and voiceprint features, through a voice sensor. Specifically, an array microphone (4 microphones in a ring layout) is used as the voice sensor and integrated into the roof console for voice data acquisition. This array microphone supports active noise cancellation and sound source localization.

[0032] Physiological sensors are used to acquire physiological data, including the driver's heart rate and skin conductance data. Specifically, the physiological sensors used include a capacitive heart rate sensor and a skin conductance sensor. The capacitive heart rate sensor is embedded in the steering wheel grip area and collects heart rate signals through contact electrodes at a sampling rate of 100Hz. The skin conductance sensor is integrated into the surface of the seat armrest and uses flexible electrode patches to measure changes in skin conductance of the driver's hands.

[0033] Simultaneously, vehicle operation data, including vehicle speed, acceleration, steering wheel angle, gear position, and navigation data (including traffic congestion level), is acquired via the vehicle's CAN bus. Finally, the acquired driver physiological data and vehicle operation data are used as the raw multimodal data stream for subsequent analysis.

[0034] In step S2, the multimodal raw data stream is encrypted and preprocessed, and the driver's emotional state is identified through a time-space dual-channel fusion network. At the same time, the driver's sensitive data during the identification process is encrypted and stored locally, and only the desensitized emotion tags are uploaded.

[0035] The original multimodal data stream is encrypted and transmitted to the edge computing and emotion recognition module, where it undergoes preprocessing operations such as noise filtering and data alignment. It should be noted that noise filtering and data alignment are not key innovative points of this invention, and therefore will not be specifically explained in this embodiment.

[0036] After preprocessing the multimodal raw data stream, the driver's emotional state, such as happiness, sadness, anger, fear, surprise, and disgust, is identified through a temporal-spatial dual-channel fusion network. This temporal-spatial dual-channel fusion network includes a temporal channel, a spatial channel, and a fusion layer, specifically: The temporal channel (LSTM + 1D-CNN) is used to process temporal data including speech signals (MFCC features) and physiological signals (heart rate (HRV time-domain and frequency-domain features), skin conductance (SCL, SCR)). Specifically, the temporal channel first extracts Mel-frequency cepstral coefficients (MFCC) features from the speech signal, performs time-domain and frequency-domain feature analysis on the heart rate signal (HRV), and extracts skin conductance level (SCL) and skin conductance response (SCR) features from the skin conductance signal. Subsequently, a Long Short-Term Memory (LSTM) network is used to capture the long-term dependencies of these temporal features, and a one-dimensional convolutional neural network (1D-CNN) is combined for local feature enhancement, ultimately outputting a temporal feature vector.

[0037] The spatial channel (ResNet50 + Transformer) is used to process spatial data including facial images (facial action units) and pose data (OpenPose skeleton keypoints). It then uses an attention mechanism to weightedly fuse and output emotional states. Specifically, the spatial channel detects and analyzes facial action units (AUs) in facial images, while simultaneously extracting human skeleton keypoints from pose images using the OpenPose model. These spatial features are first extracted using the ResNet50 backbone network for deep feature extraction, and then the Transformer architecture captures the global relationships between features, ultimately outputting a spatial feature vector.

[0038] The fusion layer dynamically assigns modal weights through an attention mechanism and outputs emotion labels. Specifically, the fusion layer receives feature vectors from both temporal and spatial channels. A trainable attention mechanism network dynamically calculates the weight coefficients of each modality feature based on the real-time environmental context (e.g., judging insufficient lighting by evaluating the average pixel value of the image), and then fuses the weighted features. Finally, the fused feature vector is passed through a fully connected layer and a Softmax classifier to output emotion labels corresponding to happy, sad, angry, fearful, surprised, disgusted, and neutral states.

[0039] Meanwhile, differential privacy technology is used to encrypt and store sensitive driver data locally during the recognition process. Only anonymized emotion tags are uploaded to the cloud to prevent user privacy leaks. This involves adding Laplace noise to driver behavior data to ensure that it is impossible to infer an individual's identity from the data. Specifically, after emotion recognition is completed, facial images, raw voice signals, and physiological signal waveforms are all encrypted and stored in local memory for no more than 24 hours, after which they are automatically and permanently deleted, ensuring that this type of data is never transmitted outside the vehicle system. For data that needs to be uploaded to the cloud, differential privacy technology is used to privatize this dataset before uploading. Specifically, after data aggregation, random noise conforming to a Laplace distribution (noise scale parameter set to 0.1) is added to it, thereby ensuring that the identity or behavioral records of any individual driver cannot be inferred from the published aggregated data while preserving the statistical utility of the data.

[0040] In step S3, based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario, the optimal emotional regulation scheme is determined from the preset emotional regulation strategy library, and the corresponding in-vehicle equipment is controlled to execute.

[0041] The system includes a pre-defined emotion regulation strategy library, encompassing music recommendations, ambient lighting adjustment, fragrance release, seat interaction, and voice interaction. After recognizing the driver's emotional state, the system combines the driver's personal preferences and the current driving scenario to determine the optimal emotion regulation solution from this library. Specifically, the pre-defined emotion regulation strategy library is stored in a database, recording specific regulation actions (such as playing a certain type of music or setting a certain ambient lighting color) and metadata about those actions (such as the type of emotion targeted and the intensity level of regulation). The intelligent recommendation module, upon receiving emotion tags from the edge computing and emotion recognition modules, combines this with the driver's personal preferences synchronized from the cloud (such as favorite music genres and frequently used fragrances) and the safety level of the current driving scenario to make a comprehensive decision. The decision-making logic is as follows: First, all regulation strategies matching the currently identified emotion are selected from the strategy library; second, unsafe strategies are filtered out based on the safety level of the current driving scenario (e.g., filtering out "playing intense music" in high-risk scenarios); finally, among the remaining candidate strategies, the strategy with the highest match to the driver's historical preferences is selected as the current optimal emotion regulation solution.

[0042] Furthermore, the selection of the optimal emotion regulation scheme is adjusted based on the safety level of the driving scenario to limit the intensity of the emotion regulation scheme. The safety level of the driving scenario includes safe, medium-risk, and high-risk scenarios. In high-risk scenarios, non-intrusive regulation methods are prioritized. Specifically, data is continuously acquired from the vehicle's CAN bus, and the safety level (safe, medium-risk, high-risk) of the current driving scenario is calculated in real time according to the classification rules in Table 1. This safety level directly affects the decision logic of the intelligent recommendation module to limit the intensity and form of the emotion regulation scheme. Specifically: in safe scenarios, full-function regulation, including music and fragrance, is allowed; in medium-risk scenarios, to avoid excessive distraction of the driver's attention, only medium-intensity non-intrusive regulation is enabled, such as ambient light color changes and gentle seat massage; in high-risk scenarios, to ensure absolute driving safety, the intensity of regulation is strictly limited, allowing only the lowest intensity, prompting non-intrusive regulation, such as low-frequency safety prompts via voice assistant, while all entertainment functions are disabled.

[0043] Table 1. Examples of Safety Level Classification Rules for Driving Scenarios

[0044] The system controls in-vehicle devices to selectively perform music recommendations, ambient lighting adjustments, fragrance releases, seat linkages, and voice interactions. For example, it adjusts the RGB color and brightness of the ambient lighting based on mood (e.g., cool blue light relieves anxiety), controls the type and concentration of fragrances via micro-pumps (lavender for calming, mint for invigorating), and adjusts the massage intensity and temperature via the seat's pneumatic module (e.g., strong pressure triggered by anger). Specifically, after receiving the adjustment strategy instructions from the intelligent recommendation module, the execution module parses them into specific device control commands. Ambient lighting control: By calling the application programming interface (API) of the vehicle ambient lighting controller, the color and brightness of the RGB LEDs are adjusted according to policy instructions. For example, to alleviate anxiety, the color is set to a cool blue light with a wavelength of approximately 450nm, and the brightness is adjusted to a preset 30%.

[0045] Fragrance release: The release rate of specific essential oils (such as lavender or peppermint) can be precisely controlled by a micrometer pump integrated into the fragrance system. For example, the release rate can be controlled within the corresponding range of 0.1 to 0.5 ml / min according to the concentration level (0-3) of the strategy instructions.

[0046] Seat linkage: By sending commands to the seat domain controller, the pneumatic massage module can be controlled to execute different massage modes and intensities (adjustable from 0 to 5 levels), and can also adjust the massage intensity based on the currently set temperature. The temperature of the seat heating or ventilation can be finely adjusted within a certain range.

[0047] Music and Voice: The in-vehicle infotainment system can play preset playlists that match the mood, and the volume can be dynamically controlled (such as automatically limiting the maximum volume when the mood is excited); the voice interaction system plays pre-recorded soothing or prompting voices according to the strategy.

[0048] In step S4, the effect of the regulation is evaluated by monitoring the driver's emotional changes in real time, and the emotion regulation strategy is dynamically updated based on the evaluation results.

[0049] The effect of emotion regulation is evaluated by real-time monitoring of driver emotion changes, and the emotion regulation strategy is dynamically updated based on the evaluation results. This includes short-term feedback monitoring and long-term feedback analysis. Short-term feedback monitoring monitors changes in the emotion index over a preset short period and switches alternative strategies accordingly. Long-term feedback analysis analyzes user preference migration patterns over a preset long period and updates the emotion regulation strategy library in conjunction with a reinforcement learning model. This can be achieved through the following methods: Short-term feedback monitoring: The system receives the emotion index output by the emotion recognition module in real time, with a 10-second cycle. If the driver's emotion index does not improve as expected or worsens after two consecutive cycles of monitoring the current adjustment plan, an instruction is immediately sent to the intelligent recommendation module to trigger the execution of a preset alternative adjustment strategy.

[0050] Long-term feedback analysis and strategy optimization: On a weekly basis, all anonymized user data is aggregated and analyzed, including sentiment change trends, active feedback to different regulation strategies (such as skipping a song), and passive reactions. This data is used to train a reinforcement learning model based on proximal policy optimization (PPO). The model uses "emotional state, scene level, and user historical preferences" as its state space, "choosing which regulation strategy" as its action space, and a comprehensive index composed of "emotion improvement index and user active rating" as its reward function. After training in the cloud, the updated strategy weights are periodically (e.g., every 24 hours) synchronized to the vehicle's intelligent recommendation module, thereby achieving continuous personalized optimization of the sentiment regulation strategy library.

[0051] For example, this embodiment provides the following two scenarios to further understand the present invention: 1) Scenario 1: Relieving fatigue during long-distance driving on highways Data input: facial recognition (frequent blinking, head drooping), heart rate dropped to 55 bpm, vehicle speed maintained at 120 km / h; Scenario determination: High-risk scenario (vehicle speed exceeding the limit), but minimum intensity adjustment is allowed; Execution strategy: The ambient light is switched to blue cool light (wavelength 450nm, which inhibits melatonin secretion). Fragrance releases peppermint essential oil (concentration 0.3ml / min); Voice prompt: "Fatigue driving detected, lane centering assist has been activated for you."

[0052] 2) Scenario 2, Family Users' Parent-Child Mode User preferences: Knowing which children are frequently in the car through account synchronization; Emotion regulation: When a child is detected crying (high-frequency voice features + facial expression), the animation theme song is played automatically, and the ambient light is dimmed to a warm yellow.

[0053] Obviously, based on the above design method, this invention can achieve the following breakthroughs: 1) By integrating multi-dimensional data such as facial expressions, voice tone, and physiological indicators, the accuracy and robustness of emotion recognition can be improved; 2) Personalized adjustment for a better experience: Personalized emotion adjustment solutions can be provided based on the driver's personal preferences and real-time emotional state, enhancing the user experience; 3) Scene adaptation for safety and reliability: By combining vehicle operation data and driving scenarios, safe and reliable emotion adjustment solutions can be intelligently recommended to ensure driving safety; 4) Closed-loop feedback for continuous optimization: Emotional changes can be monitored in real time, the adjustment effect can be evaluated, and the strategy can be dynamically adjusted to achieve closed-loop control of emotion regulation.

[0054] Example 2 This embodiment discloses a new energy vehicle driver emotion regulation system based on multimodal interaction.

[0055] like Figure 2 As shown, the new energy vehicle driver emotion regulation system based on multimodal interaction includes: The multimodal data acquisition module is configured to acquire the driver's physiological data and the vehicle's operational data to form a multimodal raw data stream; The edge computing and emotion recognition module is configured to: encrypt and preprocess the multimodal raw data stream, and identify the driver's emotional state through a time-space dual-channel fusion network; at the same time, the driver's sensitive data during the recognition process is locally encrypted and stored, and only the desensitized emotion tags are uploaded. The intelligent recommendation module is configured to determine the optimal emotion regulation scheme from a preset emotion regulation strategy library based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario. The execution module is configured to control the in-vehicle devices to execute the selected optimal emotion regulation scheme. The feedback evaluation module is configured to evaluate the regulation effect by monitoring the driver's emotional changes in real time, and dynamically update the emotion regulation strategy based on the evaluation results. Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0056] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for regulating the emotions of a new energy vehicle driver based on multimodal interaction as described in Embodiment 1 of this disclosure.

[0057] Example 4 The purpose of this embodiment is to provide an electronic device.

[0058] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the method for regulating the emotions of a new energy vehicle driver based on multimodal interaction as described in Embodiment 1 of this disclosure.

[0059] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0060] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0061] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for regulating the emotions of new energy vehicle drivers based on multimodal interaction, characterized in that, include: Acquire driver physiological data and vehicle operational data to form a multimodal raw data stream; The multimodal raw data stream is encrypted and preprocessed, and the driver's emotional state is identified through a time-space dual-channel fusion network. At the same time, the driver's sensitive data during the identification process is encrypted and stored locally, and only the de-identified emotion tags are uploaded. Based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario, the optimal emotional regulation plan is determined from the preset emotional regulation strategy library, and the corresponding in-vehicle equipment is controlled to execute accordingly. The effectiveness of the regulation strategy is evaluated by monitoring changes in the driver's emotions in real time, and the emotion regulation strategy is dynamically updated based on the evaluation results.

2. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 1, characterized in that, The physiological data includes: visual data including the driver's facial image and posture data; voice data including the driver's voice signal and voiceprint features; and physiological manifestation data including the driver's heart rate and skin conductance data. The vehicle's operational data includes: vehicle speed, acceleration, and navigation data.

3. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 1, characterized in that, The temporal-spatial dual-channel fusion network includes a temporal channel and a spatial channel; wherein, the temporal channel is used to process temporal data including speech signals and physiological signals; the spatial channel is used to process spatial data including facial images and posture data, and outputs emotional states through weighted fusion using an attention mechanism.

4. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 1, characterized in that, The emotion regulation strategy library includes music recommendations, ambient lighting adjustment, fragrance release, seat linkage, and voice interaction.

5. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 1, characterized in that, The selection of the optimal emotion regulation scheme is adjusted based on the safety level of the driving scenario to limit the intensity of the emotion regulation scheme; wherein, the safety level is divided according to vehicle speed, road conditions and weather conditions.

6. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 5, characterized in that, The safety levels of the driving scenarios include safe scenarios, medium-risk scenarios, and high-risk scenarios; in high-risk scenarios, non-intrusive adjustment methods are preferred.

7. The method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in claim 1, characterized in that, The dynamic updating of emotion regulation strategies based on evaluation results includes: short-term feedback monitoring and long-term feedback analysis. Short-term feedback monitoring monitors changes in the emotion index and switches alternative strategies over a preset short period, while long-term feedback analysis analyzes user preference migration patterns over a preset long period and updates the emotion regulation strategy library.

8. A new energy vehicle driver emotion regulation system based on multimodal interaction, characterized in that, include: The multimodal data acquisition module is configured to acquire the driver's physiological data and the vehicle's operational data to form a multimodal raw data stream; The edge computing and emotion recognition module is configured to: encrypt and preprocess the multimodal raw data stream, and identify the driver's emotional state through a time-space dual-channel fusion network; at the same time, the driver's sensitive data during the recognition process is locally encrypted and stored, and only the desensitized emotion tags are uploaded. The intelligent recommendation module is configured to determine the optimal emotion regulation scheme from a preset emotion regulation strategy library based on the identified emotional state, combined with the driver's personal preferences and the current driving scenario. The execution module is configured to control the in-vehicle devices to execute the selected optimal emotion regulation scheme. The feedback evaluation module is configured to evaluate the regulation effect by monitoring the driver's emotional changes in real time, and dynamically update the emotion regulation strategy based on the evaluation results.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in any one of claims 1-7.

10. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for regulating the emotions of new energy vehicle drivers based on multimodal interaction as described in any one of claims 1-7.

Images