Intelligent cockpit-oriented real-time emotion perception and voice interaction system

Abstract

The application discloses a real-time emotion perception and voice interaction system for an intelligent cockpit, comprising: a multi-modal data acquisition module for synchronously acquiring facial images, voice signals and text inputs of a driver; a visual feature enhancement unit for repairing low-quality images and extracting emotion distribution; an audio noise reduction and feature extraction unit for extracting voice emotion features; a text emotion coding unit for fusing relative position coding and context semantic information; a cross-modal fusion module for outputting emotion classification results by integrating visual, audio and text features; a personalized emotion database for storing user historical emotion data and performing emotion trend prediction and early warning judgment; a large language model feedback module for generating natural language feedback according to emotion recognition results and driving situations to generate structured prompt words; and a voice synthesis and output module for adjusting voice parameters and outputting feedback through a vehicle-mounted multi-channel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and in particular to a real-time emotion perception and voice interaction system for an intelligent cockpit. BACKGROUND

[0002] Under the background of the development of intelligent vehicles and human-computer interaction, the intelligent cockpit, as an important carrier of information fusion and driving experience improvement, is evolving from a traditional passive display control system to an advanced system with active perception and emotional interaction capabilities. Real-time recognition of the emotional state of the driver and providing accurate feedback have become one of the core goals of improving driving safety, personalized experience, and human-vehicle empathy. How to integrate multi-source heterogeneous data in the face of complex vehicle-mounted environments and achieve high-robust recognition and adaptive feedback of the driver's emotional state is a hot and difficult point in current research in this field.

[0003] Taking an actual driving scene as an example, a vehicle continuously collects multi-modal data such as in-vehicle video, voice, and brain electricity during operation, and these information constitutes a dynamic and complex driver behavior emotion flow. Precise analysis of these data can effectively identify different emotions of the driver, timely detect and assist driving, and intervene in the driver's emotions. Most traditional methods have exposed significant defects in actual application, which are embodied in limited fusion ability of the system, single feedback mechanism, and insufficient real-time deployment. The traditional scheme exposes the following typical problems: first, in the actual in-vehicle environment, there are synchronization errors and noise interference in each modal data, which leads to a decrease in emotion recognition accuracy; second, the existing feedback mechanism lacks situational awareness and personalized expression, making it difficult to meet the user's personalized interaction needs; third, in the vehicle-level hardware environment, the deployment of complex algorithms is limited by computing power and energy consumption, and the system is difficult to respond in real time. Therefore, in order to break through the bottleneck of existing methods, it is of great theoretical exploration value and engineering application prospect to build an intelligent cockpit emotion recognition and interaction system with high robustness, low delay, and strong feedback. Existing emotion recognition technologies can be roughly divided into two categories: single-modal recognition-based methods and multi-modal fusion-based methods. However, in the intelligent cockpit scene, the system needs to collect and process data from multiple modalities such as vision, audio, and text. These data have different sampling frequencies, heterogeneous formats, and inconsistent time sequences, which leads to problems such as information redundancy, alignment difficulty, and modal mismatch in the fusion modeling process. If this problem is not solved, it will directly affect the accuracy and stability of emotion recognition. Most current mainstream emotion recognition models are based on deep neural networks or Transformer structures, which have large computational overhead and slow inference speed, making it difficult to meet the actual requirements of intelligent cockpit for low-delay feedback. Especially when running on embedded devices such as Jetson Nano and other vehicle-level hardware, the delay is likely to exceed 200 ms, which is not conducive to forming a real-time closed-loop feedback. The existing voice feedback generated based on templates or large language models often ignores the differences in driving situations and individual emotional states, which can easily lead to language redundancy, "hallucination" feedback, or lack of personalized expression. In actual use, such feedback cannot effectively help the driver adjust his emotions, and may even cause resentment or misunderstanding. Existing systems often use general models for unified recognition of all users, ignoring the differences in individual emotional responses and expression methods, resulting in a lack of personalized recognition results, single feedback strategies, and poor user experience. SUMMARY

[0004] The purpose of the present application is to provide a real-time emotion perception and voice interaction system for intelligent cockpit to solve the problems in the background art.

[0005] Technical scheme: The real-time emotion perception and voice interaction system for intelligent cockpit provided by the application comprises a multi-modal data acquisition module for synchronously acquiring a facial image sequence, a voice signal and a text input of a driver, a visual feature enhancement unit for repairing a low-quality image and extracting an emotion distribution by using a generative facial prior network GFP-GAN and a channel segmentation spatial feature transformation CS-SFT technology, an audio noise reduction and feature extraction unit for extracting a voice emotion feature based on a non-negative matrix factorization NNMF and a HuBERT model, a text emotion encoding unit for fusing relative position coding and context semantic information by using a DeBERTa structure, a cross-modal fusion module for dynamically integrating visual, audio and text features by using a cross-modal Transformer and outputting an emotion classification result, a personalized emotion database for storing historical emotion data of a user and performing emotion trend prediction and early warning judgment based on an LSTM, a large language model feedback module for generating a structured prompt word according to an emotion recognition result and a driving situation and calling a DeepSeek model to generate a natural language feedback, and a voice synthesis and output module for dynamically adjusting voice parameters based on an emotion state and feeding back and outputting through a vehicle-mounted multi-channel.

[0006] Further, the visual feature enhancement unit further comprises an emotion sharpness screening mechanism, and only image frames with a sharpness higher than a set threshold are reserved for subsequent modeling.

[0007] Further, the audio noise reduction unit adopts a time domain and frequency domain dual noise reduction strategy, and reconstructs a denoising voice signal through short-time Fourier transform and inverse transform.

[0008] Further, the cross-modal fusion module adopts a multi-layer Transformer structure, respectively models intra-modal interaction of visual, audio and text features, and then fuses features through a cross-modal attention mechanism.

[0009] Further, the personalized emotion database records multi-modal features, emotion categories and sharpness indexes of each identification, and predicts future emotion states based on an LSTM network.

[0010] Further, the prompt word constructed by the large language model feedback module comprises a current emotion state, a confidence, a vehicle speed, a driving environment and a recommended voice style.

[0011] Further, the voice synthesis module supports dynamically adjusting a speech rate, a pitch and a tone parameter according to an emotion type, and feeds back through a vehicle-mounted loudspeaker, a display screen and an atmosphere lamp in a multi-modal manner.

[0012] The real-time emotion perception and voice interaction method provided by the application is applied to an intelligent cockpit system and comprises the following steps:

[0013] (1) Collect and preprocess multi-modal data; extract visual, audio and text emotion features and perform enhancement processing;

[0014] (2) Fuse multi-modal features through cross-modal Transformer and perform emotion classification; record emotion recognition results and update personalized emotion database;

[0015] (3) Generate situational voice feedback based on historical data and current state; synthesize and output personified voice, and link to vehicle environment adjustment module.

[0016] Further, step (3) further includes compliance verification of the generated content according to driving safety regulations.

[0017] Advantages: Compared with the prior art, the technical scheme proposed by the present application breaks the path dependence of traditional emotion recognition and feedback in theory, and first realizes the "emotion recognition-semantic understanding-voice feedback" closed-loop interactive process by creatively combining the cross-modal Transformer feature modeling mechanism and the natural language generation capability of the large language model. The system forms an end-to-end data processing link in structure and supports deployment and operation on embedded platforms such as Jetson Nano, balancing recognition accuracy, response delay and resource consumption, and having good edge deployment adaptability. In actual use, the system can continuously track the trend of the driver's emotional changes and timely feedback adjustment, enhancing the emotional expression and safety assistance function of human-vehicle interaction. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 Fig. 1 is a schematic diagram of the face prior (GFP-GAN) and channel segmentation space feature transformation (CS-SFT) of the present application;

[0019] Figure 2 Fig. 3 is a schematic diagram of multi-modal feature processing of the present application;

[0020] Figure 3 Fig. 4 is a schematic diagram of data collection of the present application. DETAILED DESCRIPTION

[0021] The technical scheme of the present application will be further described below in conjunction with the drawings.

[0022] As shown in Figure 1 , the embodiment of the present application provides a real-time emotion perception and voice interaction system for intelligent cockpit, comprising: a multi-modal data acquisition module for synchronously acquiring a driver's face image sequence, voice signal and text input; as shown in Table 1.

[0023] Table 1: Collected multi-modal data

[0024] ;

[0025] The visual feature enhancement unit adopts a generative face prior network GFP-GAN and a channel segmentation spatial feature transformation CS-SFT technology to repair low-quality images and extract emotion distributions; the specific processing process is as follows:

[0026] Face sequence extraction: the system first locates the real speaker in the current sentence through the speaker detection module, and intercepts the face frame sequence of the speaker, and uniformly crops it to 224x224 pixel resolution.

[0027] Image repair enhancement: for low-quality blurred frames, the GFP-GAN network is used for image reconstruction to restore face details. The recovery process uses U-Net to extract spatial features and latent features :

[0028] ;

[0029] Balancing authenticity and fidelity: use channel segmentation spatial feature transformation (CS-SFT) to spatially modulate part of the features, combine global structure information with local details, and generate high-quality recovery results:

[0030] ;

[0031] wherein and are split features in the channel dimension.

[0032] Emotion distribution generation: each frame of image after repair uses Swin-Transformer to extract frame-level feature vector , and converts it into a smoother emotion distribution by Softmax with added noise and temperature coefficient.

[0033] ;

[0034] Clarity screening: introduce an emotion clarity index , only keep the image frames with clarity greater than the threshold value to enter the subsequent modeling process.

[0035] Final visual representation: encode the image of the screened frame and the emotion distribution to form an enhanced visual feature:

[0036] ;

[0037] ​Audio denoising and feature extraction unit, based on non-negative matrix factorization (NNMF) and HuBERT model to extract speech emotion features; text emotion encoding unit, using DeBERTa structure to fuse relative position encoding and context semantic information; as follows:

[0038] Dual-domain denoising processing: the audio signal is first subjected to time-domain and frequency-domain dual denoising. The time-domain denoising processing is as follows:

[0039] ;

[0040] wherein is the time-domain enhanced speech spectrum, is the amplitude spectrum of the noisy speech signal, is the amplitude spectrum of the noise, is a scaling factor of spatial transformation.

[0041] Frequency domain enhancement and reconstruction: using short-time Fourier transform (STFT) to extract amplitude spectrum and phase spectrum , decomposed by non-negative matrix factorization (NNMF) and inverse short-time Fourier transform (ISTFT) to obtain denoised audio signal :

[0042] ; wherein, is Hadamard product, indicating the multiplication of corresponding elements of two matrices of the same order;

[0043] HuBERT representation generation: input the denoised audio into the HuBERT model to learn the structure of the speech signal from multiple angles and extract semantic-related audio vectors for subsequent emotion fusion.

[0044] Text encoder: using DeBERTa structure, embedding the input speech recognition transcription text into word vectors, adding relative position information from label i to label j , and generating content vector , position vector :

[0045] ;

[0046] Absolute position enhancement: adding absolute position information after all Transformer layers and embedding enhancement before Softmax to ensure that both context order and key position information are considered in the emotion semantic modeling process.

[0047] Final representation output: extracting the hidden state of the first token in the sequence as the full sentence emotion representation , wherein represents the size of the text feature.

[0048] The cross-modal fusion module dynamically integrates visual, audio and text features through a cross-modal Transformer, and outputs the emotion classification result; the specific process is as follows:

[0049] Cross-modal information fusion: input the audio features and visual features into two separate self-attention Transformer layers to simulate intra-modal interaction within each modality:

[0050] ;

[0051] ;

[0052] Wherein represents the weight matrix of the i-th layer of the Transformer for audio and visual modalities.

[0053] The respective features of the text and audio modalities are input into the CMT to fuse them, thereby obtaining a fusion representation that integrates language and acoustic information, and then further integrated with the visual modality through an additional CMT layer to generate a speech-level text-audio-visual fusion representation :

[0054] ;

[0055] ;

[0056] Emotion classification and output: the fused multi-modal features are input into a Softmax classifier to predict the emotion category of the current input . The specific operation is as follows:

[0057] ;

[0058] Wherein is a learnable linear transformation matrix, is a bias term, is the predicted probability distribution of the emotion category.

[0059] Personalized emotion database for storing user historical emotion data and predicting and warning emotion trends based on LSTM; the specific processing process is as follows:

[0060] Emotion record structure: the system generates a structured record unit every time it receives an emotion recognition result:

[0061] ;

[0062] where, is the timestamp, is the feature extracted from visual, audio, text modal encoders, is the output emotion category, is the emotion clarity indicator of the image frame or speech segment.

[0063] Temporal modeling and trend prediction: use the time series modeling network LSTM to analyze the continuous T emotion record sequence:

[0064]

[0065] where is the fusion feature representation of the current moment, and the output is the emotion category prediction sequence of the future k time steps , which is used to evaluate the emotion evolution trend.

[0066] Emotion warning mechanism: the system judges whether to trigger intervention feedback according to historical trends and current input: if there are and , it is considered that the emotional state is in a potential risk area, and the feedback module is automatically started.

[0067] Large language model feedback module, according to the emotion recognition result and the driving situation, generate structured prompt words, call DeepSeek model to generate natural language feedback; the specific processing process is as follows:

[0068] Emotion state prompt word construction: after the multi-modal emotion recognition module outputs the emotion classification result, the system organizes the current emotion state, multi-modal feature summary and driving situation information into a structured prompt format to form the context semantic prompt word input into the large model. The prompt word structure used is shown in Table 2.

[0069] Table 2 Definition and example of multi-modal emotion state prompt word field

[0070] ;

[0071] The above structure will be serialized into a prompt template and input as an input text into the large language model DeepSeek.

[0072] Natural language feedback content generation: use the large language model DeepSeek with multi-round dialogue and emotion regulation capabilities to generate feedback sentences, and the response generation logic is as follows:

[0073] ;

[0074] where, is the multi-modal prompt word constructed by the tthinput, represents a natural language feedback sentence generated based on the prompt, represents the first word in the generated feedback, the entire sentence is a word sequence with a length of

[0075] The generated language content requires the following characteristics: matching the emotional state (such as anxiety corresponding to soothing suggestions); containing behavior guidance meaning; avoiding driving interference content; the output length is controlled within 1-3 sentences. For example, when detecting that the driver is in a high-intensity anxiety state, the model output content may be: "It is detected that you may currently be in an anxiety state, it is recommended that you relax your body and take deep breaths. If necessary, you can temporarily pull over and rest."

[0076] Speech synthesis and interactive feedback execution: the present application uses a text-to-speech (TTS) engine to convert the generated natural language feedback into personified speech and play it through the vehicle-mounted speaker, which includes the following processing procedures: using the Baidu TTS model for speech synthesis, supporting emotion parameter control; matching the voice tone type according to the currently detected emotional state. As shown in Table 3.

[0077] Table 3 Speech feedback parameter comparison table under different emotional states

[0078]

[0079] The output path includes: vehicle-mounted speaker main channel, instrument screen prompt, atmosphere lamp synchronization; at the same time, the feedback content is written back to the user emotion database as a subsequent personalized emotion model training sample.

[0080] Speech synthesis and output module, dynamically adjust the speech parameters based on the emotional state, and output the feedback through the vehicle-mounted multi-channel.

[0081] Experimental verification:

[0082] Obtaining experimental data

[0083] The data required for this experiment comes from the public emotion recognition dialogue dataset MELD (Multimodal EmotionLines Dataset), which provides information in three modalities: video modality (contains face image sequence), audio modality (voice tone and acoustic features) and text modality (dialogue text). There are more than 1400 dialogues and 13000 utterances. All samples are labeled with seven emotional labels, including: anger (Anger), disgust (Disgust), fear (Fear), joy (Joy), sadness (Sadness), surprise (Surprise) and calm (Neutral). As shown in Table 4.

[0084] ​​Table 4 MELD dataset distribution profile

[0085] ;

[0086] Total number of dialogues: 1433 (training 1039, validation 114, test 280), total number of sentences: 13708 (training 9989, validation 1109, test 2610), average number of sentences per dialogue: 9.6 (training) / 9.7 (validation) / 9.3 (test), average number of emotions per dialogue: 3.30 (training) / 3.35 (validation) / 3.24 (test), number of participants per dialogue: 3 or more

[0087] Verification of anti-interference ability of multi-modal emotion recognition:

[0088] Comparison scheme: In order to comprehensively evaluate the performance of the FLRME multi-modal emotion recognition system proposed in the present application, the current mainstream multi-modal conversation emotion recognition model is selected as the comparison scheme, which mainly includes the following schemes:

[0089] DialogueRNN: a method based on a recurrent neural network structure, which uses speaker state modeling to model the evolution of emotions in a dialogue;

[0090] ConGCN: a method based on graph structure construction, which maps the sentence nodes in the dialogue for processing, realizing context dependence modeling;

[0091] MM-DFN: introduces a dynamic fusion mechanism, dynamically allocates modality weights according to multi-modal context;

[0092] GA2MIF: based on a multi-head graph attention mechanism to model intra-modal context and inter-modal complementary information;

[0093] MultiEMO: uses sample weighted focal contrastive loss (SWFC) to optimize the recognition effect of minority classes and similar classes of emotions;

[0094] EACL: uses emotion label encoding as an anchor point to guide the model to learn more discriminative sentence representations;

[0095] Evaluation index: the following three evaluation indexes are used to compare the anomaly detection performance of different algorithms:

[0096] Precision: the proportion of samples that actually belong to a certain class among all samples predicted to belong to that class.

[0097] Recall: the proportion of samples that are correctly identified by the model among all samples that actually belong to a certain class.

[0098] F1 Score (F1-Score): A weighted harmonic mean of precision and recall, used to evaluate the performance of an algorithm comprehensively.

[0099] ;

[0100] Parameter settings: Input image size: 160x160; Maximum text length: 512; Batch size: 1; Learning rate: 7e-6; Dropout rate: 0.1; Transformer hidden layer dimension: 768; Swin-Transformer uses tiny version; CMT and Transformer layer attention head number is 12. To ensure the reproducibility and consistency of the experiment, all comparison models maintain consistent training configurations, only the input structure is different.

[0101] Experimental results: The experimental results of the FLRME model and each baseline model are shown in Table 5.

[0102] Table 5 Comparison results of each model on the MELD dataset

[0103] ;

[0104] The blank entry indicates that the corresponding method does not evaluate the specific emotion, and the best result is the maximum value. The experimental results show that FLRME outperforms the previous SOTA method (EACL) by 0.3% in MELD. Detailed comparison of the performance of various methods for each emotion category in MELD shows that FLRME provides better results on most emotion labels. And for the recognition of the emotion of sadness, FLRME shows significant improvement compared to the baseline model, highlighting its ability to capture subtle emotional cues.

[0105] Verification of emotion-aware personalized interaction regulation strategy:

[0106] Experimental scenario and recognition results

[0107] Selecting multi-modal input data collected in a typical driving process, including visual modality (driver face sequence image), audio modality (speech segment) and text modality (speech recognition text), the system completes the emotion recognition process and outputs the following emotion probability distribution results (normalized by Softmax): Anger: 6.53% Disgust: 4.35% Fear: 23.03% Happiness: 1.85% Neutral: 11.60% Sadness: 23.97% Surprise: 28.66%;

[0108] The system extracts the main negative emotions (fear, sadness) and high activation state (surprise) signals through analysis of the emotion distribution vector, inferring that the current user is in a complex emotional state of anxiety / overload.

[0109] Multi-modal prompt word construction

[0110] Combined with the current recognition result, driving state, and user database historical summary, the system constructs the prompt word structure as follows (simplified example):

[0111] {

[0112] "Current emotional state": "Sadness / Surprise Complex",

[0113] "Confidence": 0.2866,

[0114] "Driving environment": "Urban road, night",

[0115] "Historical emotional trend": ["Sadness", "Fear", "Calm"],

[0116] "Interaction style": "Warm and caring"

[0117] }

[0118] DeepSeek feedback generation result: call the DeepSeek large model to generate feedback language, and the system controls the output style according to the prompt word. The generated sentence is as follows:

[0119] Warm and caring feedback:

[0120] "Hello, it seems that you may be a little worried. Please take a few deep breaths and relax. I will adjust the car environment to help you relieve your mood."

[0121] Professional guidance type feedback:

[0122] "Your mood seems to be a little low. I suggest you can first close your eyes, take three deep breaths, and then slowly open your eyes. Next, I will adjust the car environment to improve comfort."

[0123] Light and humorous feedback:

[0124] "Hey, it seems like today hasn't been going well! Don't worry, let's do a little magic - I'll make the car warm and comfortable."

[0125] Linkage environment regulation response: the system automatically links the vehicle-mounted environment regulation module according to the semantic keywords in the feedback language and the emotion regulation mapping table, and outputs as shown in Table 6.

[0126] Table 6 Vehicle-mounted environment regulation parameter configuration table based on emotion recognition results

[0127] ;

[0128] According to the above parameters, the in-vehicle environment is adjusted to achieve the purpose of mood regulation.

[0129] Part of the parameters and their meaning table as shown in Table 7.

[0130] Table 7 Part of the parameters and their meaning table

[0131] .

Claims

1. A real-time emotion perception and voice interaction system for smart cockpits, characterized in that, include: The multimodal data acquisition module is used to simultaneously acquire the driver's facial image sequence, voice signal, and text input; The visual feature enhancement unit employs the generative face prior network GFP-GAN and channel segmentation spatial feature transform CS-SFT to repair low-quality images and extract emotion distribution. The audio denoising and feature extraction unit extracts speech emotion features based on nonnegative matrix factorization (NNMF) and the HuBERT model. The text emotion encoding unit uses a DeBERTa structure to fuse relative position encoding and contextual semantic information. The cross-modal fusion module dynamically integrates visual, audio, and text features through a cross-modal Transformer to output emotion classification results. The personalized emotion database stores users' historical emotion data and uses LSTM for emotion trend prediction and early warning. The large language model feedback module generates structured prompts based on emotion recognition results and driving context, and calls the DeepSeek model to generate natural language feedback. The speech synthesis and output module dynamically adjusts speech parameters based on emotional state and outputs feedback through in-vehicle multi-channels. The visual feature enhancement unit includes an emotion clarity filtering mechanism, retaining only image frames with clarity above a set threshold for subsequent modeling. The audio denoising unit employs a dual denoising strategy in the time and frequency domains, reconstructing the denoised speech signal through short-time Fourier transform and inverse transform. The cross-modal fusion module uses a multi-layer Transformer structure to perform intra-modal interaction modeling for visual, audio, and text features, and then fuses features through a cross-modal attention mechanism. The personalized emotion database records the multimodal features, emotion categories, and clarity indices of each recognition, and predicts future emotional states based on an LSTM network. The audio denoising and feature extraction unit extracts speech emotion features based on non-negative matrix factorization (NNMF) and HuBERT models. The text emotion encoding unit uses a DeBERTa structure to fuse relative positional encoding and contextual semantic information. Details are as follows: Dual-domain noise reduction processing: The audio signal first undergoes dual noise reduction in both the time and frequency domains; the time-domain noise reduction processing is as follows: ; in It is a time-domain enhanced speech spectrum. It is the amplitude spectrum of a noisy speech signal. The amplitude spectrum of the noise. It is the scaling factor for spatial transformation; Frequency Domain Enhancement and Reconstruction: Extracting Amplitude Spectrum Using Short-Time Fourier Transform Phase spectrum The denoised frequency signal is obtained by nonnegative matrix decomposition and reconstruction using inverse short-time Fourier transform. : ;in, The Hadamard product represents the element-wise multiplication of two matrices of the same order. The denoised audio signal is input into the Hubert model to learn the structure of the speech signal from multiple perspectives and extract semantically relevant audio vectors. This is used for subsequent emotional integration; Using the DeBERTa structure, the input speech recognition transcribed text is embedded as word vectors, incorporating the relative position information from marker i to marker j. and generate content vectors. Position vector : ; Absolute position enhancement: Absolute position information is added after all Transformer layers and embedded enhancement is performed before Softmax to ensure that the context order and key position information are taken into account during the sentiment semantic modeling process; the hidden state of the first token of the sequence is extracted as the sentiment representation of the whole sentence. ,in Indicates the size of the text feature.

2. The real-time emotion perception and voice interaction system for intelligent cockpits according to claim 1, characterized in that, The prompts constructed by the large language model feedback module include the current emotional state, confidence level, vehicle speed, driving environment, and recommended speech style.

3. The real-time emotion perception and voice interaction system for intelligent cockpits according to claim 1, characterized in that, The speech synthesis module supports dynamic adjustment of speech rate, pitch, and intonation parameters based on emotion type, and provides multimodal feedback through in-vehicle speakers, displays, and ambient lighting.

4. A real-time emotion perception and voice interaction method, characterized in that, The system implementation according to any one of claims 1-3 includes the following steps: (1) Collect and preprocess multimodal data; extract visual, audio and text emotion features respectively and perform enhancement processing; (2) Fuse multimodal features through cross-modal Transformer and perform emotion classification; record emotion recognition results and update personalized emotion database; (3) Generate contextualized voice feedback based on historical data and current status; synthesize and output human-like voice, and link the vehicle environment adjustment module.

5. The real-time emotion perception and voice interaction method according to claim 4, characterized in that, The voice feedback generation process also includes compliance verification of the generated content according to driving safety regulations.

Images