Multi-modal emotion recognition method for driver
By integrating the driver's voice, text, and video data and using deep learning networks for multimodal emotion recognition, the problem of single-modal data being susceptible to interference is solved, achieving accurate and real-time recognition of the driver's emotions and improving the robustness and reliability of the in-vehicle system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-29
- Publication Date
- 2026-03-27
AI Technical Summary
Existing driver emotion recognition technologies rely on single-modal data, which is easily affected by driving environment interference, resulting in inaccurate recognition results and poor reliability, making it difficult to meet the real-time and reliability requirements of in-vehicle systems.
A multimodal emotion recognition method is adopted. By simultaneously collecting and processing the driver's voice, text and video data, the hybrid MFCC, FastText and HOG feature extraction techniques are used, combined with a deep learning network for emotion inference, and the final emotion state is output through a decision-level fusion strategy.
It achieves accurate and real-time recognition of driver emotions, overcomes the limitations and susceptibility to interference of single-modal data, improves the robustness and reliability of emotion judgment, and supports in-vehicle personalized services and proactive safety warnings.
Smart Images
Figure CN121743929A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent driving technology, and in particular to a multimodal emotion recognition method for drivers. Background Technology
[0002] In the field of smart cockpits, accurate driver emotion recognition is crucial for improving human-machine interaction and active safety. Existing driver emotion recognition solutions largely rely on single-modal data, such as analyzing facial expressions or voice tone alone. These methods have significant limitations: in the complex real-world driving environment, a single data source is susceptible to various interferences. For example, facial recognition may fail in poor lighting conditions, when the driver is wearing sunglasses, or when there are obstructions; voice recognition cannot function due to in-vehicle noise or when the driver is silent; and pure text analysis struggles to capture the emotional nuances in speech.
[0003] Furthermore, these methods typically lack the time-series synchronization and effective fusion of multimodal information, resulting in biased and inaccurate emotion judgments that fail to meet the stringent reliability and real-time requirements of in-vehicle systems. Therefore, there is an urgent need for an emotion recognition technology that can comprehensively utilize multi-source information and possesses strong anti-interference capabilities. Summary of the Invention
[0004] The purpose of this invention is to provide a multimodal emotion recognition method for drivers, in order to solve the technical problems of inaccurate and unreliable recognition results caused by the one-sided data source and susceptibility to interference from the driving environment when relying on single-modal data for driver emotion recognition in the prior art.
[0005] This invention provides the following solution:
[0006] A multimodal emotion recognition method for drivers includes:
[0007] Simultaneously collect and preprocess the driver's driving data to obtain standard voice signals, standard text, and standard video data;
[0008] A hybrid MFCC method was used to extract speech spectral features;
[0009] The FastText method is used to extract semantic features from standard text;
[0010] Standard video data is processed frame by frame to extract HOG features of the driver's face;
[0011] Based on speech spectrum features, semantic features of standard text, and HOG features of the driver's face, three independent sub-networks are used to perform preliminary emotion inference. The driver's emotions are then classified through a decision-level fusion strategy, and the driver's emotional state is finally output.
[0012] Furthermore, the energy of each frame of standard speech signal passing through the Mel filter bank is calculated, and the first 13-dimensional static MFCC coefficients are obtained through discrete cosine transform. The first-order difference and second-order difference of these 13-dimensional static coefficients are calculated to obtain 13-dimensional dynamic coefficients. The static, first-order and second-order coefficients are concatenated to form a 39-dimensional hybrid MFCC feature vector, which serves as the final speech spectrum feature of that frame.
[0013] Furthermore, the pre-trained FastText model is used to transform the preprocessed text data into low-dimensional, dense word vector representations. The specific process is as follows:
[0014] The standard text is input into the FastText model, which maps each word in the standard text to a 300-dimensional word vector. The arithmetic mean of the word vectors of all words in the standard text is calculated to obtain a 300-dimensional global text feature vector, which serves as the semantic feature of the standard text.
[0015] Furthermore, the Viola-Jones algorithm is used to quickly locate the driver's face region in the video frame, and the active appearance model is used to locate and align 68 key points of the face. If face detection fails in a single frame, the result of successful detection in the previous frame is used. If detection fails for 10 consecutive frames, the time period is marked as invalid and a system alarm is triggered.
[0016] Furthermore, the aligned face regions are uniformly scaled to 64×64 pixels and converted to grayscale images;
[0017] Calculate its oriented gradient histogram (HOG) features with the following parameters: 8×8 pixels for cell units, 2×2 cells for block size, and 9 orientation bins.
[0018] Finally, a 1764-dimensional HOG feature vector is extracted from each frame of the image as the HOG feature of the driver's face.
[0019] Furthermore, the speech spectral features are input into a one-dimensional convolutional neural network combined with a gated recurrent unit model for processing. This model first extracts local temporal patterns using a two-layer 1D-CNN, then captures long-term dependencies using a Bi-GRU network with two layers and 128 hidden units. Finally, the output of the last time step of the Bi-GRU is transformed using a fully connected layer, and a Softmax classifier is used to output the emotion probability distribution Pa of the speech modality.
[0020] Furthermore, the semantic features of the standard text are input into a two-layer fully connected network. The first fully connected layer has a dimension of 256 and uses the ReLU activation function; the second fully connected layer has a dimension of 128 and also uses the ReLU activation function. Finally, a Softmax classifier is used to output the sentiment probability distribution Pt of the text modality.
[0021] Furthermore, the HOG feature sequence of the driver's face is reduced to 512 dimensions through a fully connected layer, and then input into a Bi-LSTM network with 2 layers and 256 hidden units. The output of the Bi-LSTM at all time steps is globally averaged and pooled, then transformed through a fully connected layer, and finally passed through a Softmax classifier to output the emotion probability distribution Pv of the visual modality.
[0022] Furthermore, a comprehensive probability distribution Pf is constructed based on the emotion probability distribution Pa of the speech modality, the emotion probability distribution Pt of the text modality, and the emotion probability distribution Pv of the visual modality, and the following settings are made:
[0023] Pf = w1 × Pa + w2 × Pt + w3 × Pv;
[0024] Where w1, w2, and w3 are the weight coefficients of each mode, and w1 + w2 + w3 = 1;
[0025] In Pf, the preset emotion category with the highest probability value is selected as the driver's current final emotional state determination result. The preset emotion categories include: neutral, happy, sad, angry, surprised, disgusted, and tired.
[0026] Furthermore, in the output probability distribution, the preset emotion category with the highest probability value is selected as the final emotion state judgment result of the driver.
[0027] The system sends the emotional state determination result to the vehicle control module and the human-machine interaction system in real time via the vehicle data bus. The human-machine interaction system has pre-stored feedback strategies corresponding to each emotional state.
[0028] The above solution achieves the following beneficial technical effects:
[0029] This invention achieves more accurate and real-time recognition of drivers' emotional states by integrating three types of non-physiological data: voice, text, and video. Its core advantage lies in effectively overcoming the inherent limitations of single-modal data—its incompleteness and susceptibility to environmental interference—by significantly improving the robustness and reliability of emotion determination through the complementarity and collaborative verification of multi-source information. The system can adapt to complex changes in the driving environment, and even when the quality of data in one modality deteriorates, it can still make robust inferences through other modalities. Ultimately, this efficient and stable emotion recognition capability provides high-quality decision-making basis for in-vehicle personalized services and active safety warning systems, enabling more timely intervention in drivers' negative emotional states, improving driving safety and the cockpit interaction experience, and possessing significant practical application value. Attached Figure Description
[0030] Figure 1 This is a flowchart of a multimodal emotion recognition method for drivers provided by one or more embodiments of the present invention.
[0031] Figure 2 This is a flowchart of the method for extracting HOG features of the driver's face according to the present invention.
[0032] Figure 3 This is a flowchart of the driver emotion classification method of the present invention. Detailed Implementation
[0033] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0034] Specifically, this invention is applied to intelligent cockpit environments, aiming to achieve accurate and real-time identification of the driver's emotional state by integrating three types of non-physiological data: voice, text, and video. This method provides a reliable technical means to address issues such as incompleteness and susceptibility to interference with single-modal data, and can provide decision-making support for systems such as in-vehicle personalized services and proactive safety warnings.
[0035] Please see Figure 1 As shown, the multimodal emotion recognition method for drivers includes:
[0036] Step S101: Synchronously collect the driver's driving data and preprocess it to obtain standard voice signals, standard text and standard video data. The driving data includes voice signals, text data and video data.
[0037] Specifically, the speech signal is acquired through a high signal-to-noise ratio vehicle microphone array, with a sampling frequency set to 16kHz and a recording format of PCM. The acquired raw speech is pre-emphasized, framed with a frame length of 25ms and a frame shift of 10ms, and processed with Hamming windows to obtain a standard speech signal.
[0038] The text data is transcribed into text in real time by an in-vehicle speech recognition (ASR) engine (which synchronously acquires the speech signal used for speech feature extraction), or by directly acquiring the human-computer interaction text record. The transcribed text is then processed by word segmentation, removal of stop words and punctuation marks to obtain standard text.
[0039] The video data is obtained by capturing the driver's facial video stream through a near-infrared camera deployed in front of the driver's seat, with a resolution set to 640×480 and a frame rate of 30fps to obtain standard video data.
[0040] The synchronous acquisition refers to the hardware-triggered synchronization of audio and video data timestamps, and the generation of text data by the ASR engine and association with the timestamps of the corresponding audio segments. The system slices and aligns the multimodal data in a fixed time window (e.g., 2 seconds) to ensure temporal consistency in subsequent fusion analysis.
[0041] Specifically, by simultaneously collecting voice, text, and video data and performing standardized preprocessing, the consistency and high quality of multimodal data over time are ensured. This lays a reliable foundation for subsequent feature extraction and fusion analysis, effectively reduces data noise and inconsistency, and thus improves the accuracy and stability of the emotion recognition system, enabling it to adapt to complex in-vehicle environments.
[0042] Please continue reading. Figure 1 As shown, the multimodal emotion recognition method for drivers also includes:
[0043] Step S102: Extract speech spectral features using the hybrid MFCC method.
[0044] Specifically, the process of extracting hybrid MFCC features from standard speech signals is as follows:
[0045] For each frame of standard speech signal, the energy passing through the Mel filter bank is calculated, and the first 13-dimensional static MFCC coefficients are obtained through discrete cosine transform. The first-order difference and second-order difference of these 13-dimensional static coefficients are calculated to obtain 13-dimensional dynamic coefficients. The static, first-order and second-order coefficients are concatenated to form a 39-dimensional hybrid MFCC feature vector, which serves as the final speech spectrum feature of that frame.
[0046] Specifically, by extracting speech spectral features using the hybrid MFCC method, the static and dynamic acoustic modes in the speech signal can be fully captured. This method enhances the expressiveness of speech modalities in emotion recognition, enabling the system to more accurately infer emotional changes from the driver's speech, and maintains robustness, especially in noisy environments, providing key acoustic information for multimodal fusion.
[0047] Please continue reading. Figure 1 As shown, the multimodal emotion recognition method for drivers also includes:
[0048] Step S103: Extract semantic features of standard text using the FastText method.
[0049] Specifically, the pre-trained FastText model is used to transform the preprocessed text data into low-dimensional, dense word vector representations. The specific process is as follows:
[0050] The standard text is input into the FastText model, which maps each word in the standard text to a 300-dimensional word vector. The arithmetic mean of the word vectors of all words in the standard text is calculated to obtain a 300-dimensional global text feature vector, which serves as the semantic feature of the standard text.
[0051] For example, the pre-trained FastText model can be a 300-dimensional word vector model pre-trained on the Common Crawl or Wikipedia corpus.
[0052] Specifically, if the preprocessed text data is empty, the semantic features of the modality are set as a 300-dimensional zero vector.
[0053] Specifically, the FastText method is used to extract semantic features from the text and transform the text into a low-dimensional dense vector representation, thereby capturing the deep semantic information of the language. This enables the system to understand emotional content from the driver's speech transcription or interactive text, making up for the shortcomings of pure acoustic or visual analysis and improving the semantic understanding ability and overall coverage of emotion recognition.
[0054] Please continue reading. Figure 1 As shown, the multimodal emotion recognition method for drivers also includes:
[0055] Step S104: Process the standard video data frame by frame to extract the HOG features of the driver's face.
[0056] Specifically, by processing video data frame by frame and extracting facial HOG features, the system can robustly capture the gradient direction information of the driver's facial expressions. This method can still provide consistent feature representations under conditions of changing lighting or partial occlusion, enhancing the sensitivity of the visual modality to emotional changes and providing a reliable visual basis for multimodal emotion recognition.
[0057] Please see Figure 2 As shown, the method for extracting the HOG features of the driver's face includes:
[0058] Step S201: Process the standard video data frame by frame to perform face detection and alignment.
[0059] Specifically, the Viola-Jones algorithm is used to quickly locate the driver's face region in the video frame, and the active appearance model is used to locate and align 68 key points of the face. If face detection fails in a single frame, the result of successful detection in the previous frame is used. If detection fails for 10 consecutive frames, the time period is marked as invalid and a system alarm is triggered.
[0060] Specifically, by employing the Viola-Jones algorithm combined with an active appearance model to quickly detect and accurately align faces in video frames, the accuracy and stability of facial region localization are effectively improved. This strategy can not only efficiently extract facial key points under normal lighting and pose conditions, but also maintain continuity by utilizing historical frame information when there is brief occlusion or detection failure, thus avoiding interruption of feature extraction. At the same time, an alarm mechanism is set up for the case of multiple consecutive frames of detection failure, which enhances the reliability and fault tolerance of the system under extreme driving scenarios, and provides a high-quality and structurally consistent input foundation for subsequent facial feature extraction.
[0061] Please continue reading. Figure 2 As shown, the method for extracting HOG features of the driver's face also includes:
[0062] Step S202: Extract the HOG features of the driver's face based on the aligned face region.
[0063] Specifically, the aligned face areas are uniformly scaled to 64×64 pixels and converted to grayscale images;
[0064] Calculate its oriented gradient histogram (HOG) features with the following parameters: 8×8 pixels for cell units, 2×2 cells for block size, and 9 orientation bins.
[0065] Finally, a 1764-dimensional HOG feature vector is extracted from each frame of the image as the HOG feature of the driver's face.
[0066] Specifically, by extracting histogram of oriented gradients (HOG) features from aligned face images of uniform size and grayscale, this method can robustly capture local edge and texture changes in facial expressions. It is highly robust to illumination fluctuations, slight pose shifts, and partial occlusion. The generated high-dimensional gradient feature vectors fully characterize the subtle differences in the driver's facial muscle movements, providing stable and discriminative visual cues for emotion recognition and significantly enhancing the ability of visual modalities to perceive emotions in complex in-vehicle environments.
[0067] Please continue reading. Figure 1 As shown, the multimodal emotion recognition method for drivers also includes:
[0068] Step S105: Based on speech spectrum features, semantic features of standard text, and HOG features of the driver's face, three independent sub-networks are used to perform preliminary emotion inference, and the driver's emotions are classified through a decision layer fusion strategy, and the driver's emotional state is finally output.
[0069] Specifically, by performing preliminary emotion inference through three independent sub-networks and combining them with a decision-level fusion strategy, the system can comprehensively utilize the complementary advantages of voice, text, and visual modalities. This fusion method effectively overcomes the limitations and susceptibility to interference of a single modality, improves the overall robustness and accuracy of emotion recognition, and ultimately outputs a reliable emotion state, providing real-time decision support for in-vehicle personalized services and proactive safety warnings.
[0070] Please see Figure 3 As shown, the driver emotion classification method includes:
[0071] Step S301: Based on speech spectrum features, semantic features of standard text, and HOG features of the driver's face, three independent sub-networks are used to perform preliminary emotion inference, and the driver's emotions are classified through a decision layer fusion strategy.
[0072] Specifically, the speech spectral features are input into a one-dimensional convolutional neural network combined with gated recurrent units (GRUs) for processing. This model first extracts local temporal patterns using a two-layer 1D-CNN, then captures long-term dependencies using a two-layer Bi-GRU network with 128 hidden units. Finally, the output of the last time step of the Bi-GRU is transformed using a fully connected layer, and a Softmax classifier is used to output the emotion probability distribution Pa of the speech modality. The two 1D-CNN convolutional layers have kernel sizes of 3 and 5, and the number of kernels is 32 and 64, respectively. The fully connected layer has a dimension of 128 and uses the ReLU activation function.
[0073] The semantic features of standard text are input into a two-layer fully connected network. The first fully connected layer has a dimension of 256 and uses the ReLU activation function; the second fully connected layer has a dimension of 128 and also uses the ReLU activation function. Finally, a Softmax classifier is used to output the sentiment probability distribution Pt of the text modality.
[0074] The HOG feature sequence of the driver's face is reduced to 512 dimensions through a fully connected layer, and then input into a Bi-LSTM network with 2 layers and 256 hidden units. Global average pooling is performed on the output of the Bi-LSTM at all time steps, and then it is transformed through a fully connected layer. Finally, it is passed through a Softmax classifier to output the emotion probability distribution Pv of the visual modality.
[0075] A comprehensive probability distribution Pf is constructed based on the emotion probability distribution Pa of the speech modality, the emotion probability distribution Pt of the text modality, and the emotion probability distribution Pv of the visual modality. The following settings are defined:
[0076] Pf = w1 × Pa + w2 × Pt + w3 × Pv;
[0077] Where w1, w2, and w3 are the weight coefficients of each mode, and w1 + w2 + w3 = 1;
[0078] In Pf, the preset emotion category with the highest probability value is selected as the driver's current final emotional state determination result. The preset emotion categories include: neutral, happy, sad, angry, surprised, disgusted, and tired.
[0079] For example, in this embodiment, w1 is 0.4, w2 is 0.3, and w3 is 0.3.
[0080] Specifically, by designing deep sub-networks adapted to the data characteristics of speech, text, and visual modalities respectively, and introducing a learnable decision-level weighted fusion mechanism, the system can fully leverage the complementary advantages of multimodal information. The speech sub-network combines convolutional and bidirectional gated recurrent structures to effectively model the temporal dynamics of acoustic signals; the text sub-network captures the emotional connotations of language through dense semantic vectors; and the visual sub-network uses temporal modeling to process facial feature sequences. The three sub-networks work together and are fused through adaptive weights, which not only improves the overall accuracy of emotion classification but also enhances the robustness of the model under non-ideal conditions such as noise interference and modality loss, achieving a more comprehensive and reliable inference of emotional states.
[0081] Please continue reading. Figure 3 As shown, the driver emotion classification method further includes:
[0082] Step S302: Output the driver's emotional state and perform human-computer interaction.
[0083] Specifically, in the output probability distribution, the preset emotion category with the highest probability value is selected as the driver's current final emotion state determination result;
[0084] The system sends the emotional state determination result to the vehicle control module and the human-machine interaction system in real time via the vehicle data bus. The human-machine interaction system has pre-stored feedback strategies corresponding to each emotional state.
[0085] For example, the feedback strategy may be: when the judgment result is "anger", the ambient light is controlled to change to a cool tone and the brightness is reduced, while the voice assistant plays soothing music;
[0086] When the assessment result is "fatigue", the seat will vibrate and a voice prompt will say "We suggest you go to the nearest service area to rest".
[0087] Specifically, the three sub-networks (speech, text, and vision) and the adaptive weighting module are trained end-to-end as a complete system. The training process is as follows:
[0088] A dataset containing 100 hours of multimodal data (synchronized voice, video, and transcribed text) from real-world driving scenarios and their corresponding emotion labels was used. The emotion labels covered seven categories: neutral, happy, sad, angry, surprised, disgusted, and tired.
[0089] Cross-entropy loss is used as the loss function for model training, and the Adam optimizer is used for optimization. The initial learning rate is set to 1e-4, and a learning rate decay strategy is adopted. During training, the three sub-networks are first pre-trained independently, and then the parameters of their feature extraction layers are fixed. The entire system is then jointly fine-tuned to optimize the final emotion classification performance.
[0090] Specifically, to adapt to the performance of the in-vehicle computing unit, the three sub-networks can undergo lightweight operations such as model pruning and quantization after training, or be converted to a format supported by a high-performance inference engine (such as TensorRT) to ensure that the emotion recognition task can meet the real-time requirements of driving scenarios.
[0091] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.
Claims
1. A multimodal emotion recognition method for drivers, characterized in that, include: Simultaneously collect and preprocess the driver's driving data to obtain standard voice signals, standard text, and standard video data; A hybrid MFCC method was used to extract speech spectral features; The FastText method is used to extract semantic features from standard text; Standard video data is processed frame by frame to extract HOG features of the driver's face; Based on speech spectrum features, semantic features of standard text, and HOG features of the driver's face, three independent sub-networks are used to perform preliminary emotion inference. The driver's emotions are then classified through a decision-level fusion strategy, and the driver's emotional state is finally output.
2. The multimodal emotion recognition method for drivers according to claim 1, characterized in that, For each frame of standard speech signal, the energy passing through the Mel filter bank is calculated, and the first 13-dimensional static MFCC coefficients are obtained through discrete cosine transform. The first-order difference and second-order difference of these 13-dimensional static coefficients are calculated to obtain 13-dimensional dynamic coefficients. The static, first-order and second-order coefficients are concatenated to form a 39-dimensional hybrid MFCC feature vector, which serves as the final speech spectrum feature of that frame.
3. The multimodal emotion recognition method for drivers according to claim 2, characterized in that, The pre-trained FastText model is used to transform the preprocessed text data into low-dimensional, dense word vector representations. The specific process is as follows: The standard text is input into the FastText model, which maps each word in the standard text to a 300-dimensional word vector. The arithmetic mean of the word vectors of all words in the standard text is calculated to obtain a 300-dimensional global text feature vector, which serves as the semantic feature of the standard text.
4. The multimodal emotion recognition method for drivers according to claim 3, characterized in that, The Viola-Jones algorithm is used to quickly locate the driver's face region in the video frame, and the active appearance model is used to locate and align 68 key points of the face. If the face detection fails in a single frame, the result of the successful detection in the previous frame is used. If 10 consecutive frames fail to be detected, the time period will be marked as invalid and a system alarm will be triggered.
5. The multimodal emotion recognition method for drivers according to claim 4, characterized in that, The aligned face areas are uniformly scaled to 64×64 pixels and converted to grayscale. The HOG (Histogram of Oriented Gradients) feature is calculated, and finally a 1764-dimensional HOG feature vector is extracted from each frame as the HOG feature of the driver's face.
6. The multimodal emotion recognition method for drivers according to claim 5, characterized in that, The speech spectral features are input into a one-dimensional convolutional neural network combined with a gated recurrent unit model for processing. The model first extracts local temporal patterns using a two-layer 1D-CNN, then captures long-term dependencies using a Bi-GRU network with two layers and 128 hidden units. Finally, the output of the last time step of the Bi-GRU is transformed using a fully connected layer and passed through a Softmax classifier to output the emotion probability distribution Pa of the speech modality.
7. The multimodal emotion recognition method for drivers according to claim 6, characterized in that, The semantic features of standard text are input into a two-layer fully connected network, and a Softmax classifier is used to output the sentiment probability distribution Pt of the text modality.
8. The multimodal emotion recognition method for drivers according to claim 7, characterized in that, The HOG feature sequence of the driver's face is reduced to 512 dimensions through a fully connected layer, then input into a Bi-LSTM network. The output of the Bi-LSTM at all time steps is globally averaged and pooled, then transformed through a fully connected layer, and finally passed through a Softmax classifier to output the emotion probability distribution Pv of the visual modality.
9. The multimodal emotion recognition method for drivers according to claim 8, characterized in that, A comprehensive probability distribution Pf is constructed based on the emotion probability distribution Pa of the speech modality, the emotion probability distribution Pt of the text modality, and the emotion probability distribution Pv of the visual modality. The following settings are defined: Pf = w1 × Pa + w2 × Pt + w3 × Pv; Where w1, w2, and w3 are the weight coefficients of each mode, and w1 + w2 + w3 = 1; In Pf, the preset emotion category with the highest probability value is selected as the driver's current final emotional state determination result. The preset emotion categories include: neutral, happy, sad, angry, surprised, disgusted, and tired.
10. The multimodal emotion recognition method for drivers according to claim 9, characterized in that, In the output probability distribution, the preset emotion category with the highest probability value is selected as the driver's current final emotion state determination result; The system sends the emotional state determination result to the vehicle control module and the human-machine interaction system in real time via the vehicle data bus. The human-machine interaction system has pre-stored feedback strategies corresponding to each emotional state.