Multi-mode driver emotion recognition method and system in real vehicle environment

Abstract

The invention discloses a multi-modal driver emotion recognition method and system in a real vehicle environment, and belongs to the technical field of intelligent driving. Prediction labels corresponding to physiological and image modalities are determined, and when the prediction labels are inconsistent, prediction probabilities of different emotion categories corresponding to the current physiological and image modalities are determined; respectively acquiring confidence coefficients of physiological and image modalities; and when the modal conflict degree is not less than a preset conflict threshold or the minimum value does not exceed a preset confidence threshold, marking the prediction tag as a conflict state. And when the difference value of the confidence coefficients of the physiological and image modalities in the conflict state is not greater than the confidence coefficient difference tolerance, performing historical state smooth correction on the prediction probability of the current physiological and image modalities. According to the method, the overall misjudgment risk caused by single-mode recognition deviation can be remarkably reduced, and the interpretability of a multi-mode emotion recognition system in a real driving scene is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and more specifically to a multimodal driver emotion recognition method and system in a real vehicle environment. Background Technology

[0002] Drivers are a major contributing factor to traffic accidents, with over 90% of accidents directly related to driver behavior, including overtaking, fatigued driving, and emotional driving. A driver's emotional state has a significant impact on driving safety: negative emotions can easily lead to dangerous behaviors such as speeding and road rage; while positive emotions contribute to safe driving, excessive excitement can also cause accidents. Therefore, maintaining driver emotional stability is crucial for preventing traffic accidents.

[0003] With the rapid development of intelligent driving technology, driver emotion monitoring has become an important research direction for improving driving safety. However, collecting emotion data in real driving environments faces many challenges: time-consuming processes and complex environments limit data acquisition efficiency. Therefore, researchers generally use emotion-inducing methods to obtain driving data. Current mainstream research is mostly conducted in laboratory environments, using driving simulators or video scenario playback systems to induce emotions through images, text, audio, or video, and combining this with commonly used physiological signals such as electrocardiograms (ECG), electrical activity of the skin (EDA), and electroencephalograms (EEG) for identification and modeling. However, these methods have significant limitations: simulated environments cannot accurately reproduce the situational stress and dynamic road conditions of real driving, leading to unstable and poorly repeatable induced emotions; drivers' concentration can easily suppress existing emotions, lacking an effective maintenance mechanism; large differences in multimodal data sampling rates and missing timestamps affect model robustness; emotion labels often rely on subjective ratings or single models, lacking a reliable verification mechanism.

[0004] Current emotion induction methods primarily rely on single-modal stimulation to elicit target emotions. This one-channel approach struggles to create a multimodal synergistic effect, resulting in insufficient intensity and short duration of induced emotions. This is especially problematic during driving, where the driver's heightened attention quickly dilutes the induced emotional state. Furthermore, existing methods rely solely on the SAM scale for subjective assessment of emotional states, leading to significant limitations in the reliability of their evaluation results.

[0005] Current emotion recognition research mainly relies on multimodal public datasets such as SEED. These datasets are collected from experimenters in static laboratory environments and lack participation from real-world task scenarios. Even specialized research on driver emotions mostly uses public datasets such as DEEE or collects experimental data in simulated driving environments. Because these simulated environments do not present real driving risks, the driver's psychological state differs significantly from real road conditions and cannot reflect the true emotional state during actual driving.

[0006] In actual driving, the recognition results of facial expressions and physiological signals may conflict significantly. For example, when the driver is in a brightly lit environment or wearing a mask, facial expression signals may not effectively reflect true emotions due to obstruction; while physiological signals may also deviate due to electrode detachment, motion artifacts, etc. When the emotion categories output by the two modalities in the system are inconsistent, or when the confidence levels are similar but the predicted labels are different, it is considered a modal conflict state.

[0007] In multimodal emotion recognition systems, facial images and physiological signals are often analyzed using independent models to output their respective emotion predictions. However, current fusion methods mostly employ static weighted averaging or voting strategies, ignoring conflicts between different modal recognition results. This leads to misjudgments when modal conflicts occur, failing to reflect the true emotional state during actual driving. Summary of the Invention

[0008] To address the problems existing in the above-mentioned fields, this invention proposes a multimodal driver emotion recognition method and system in a real vehicle environment. By recognizing the physiological signals and facial expression image signals of the driver collected in real time, and performing modal conflict recognition and recognition result correction on the recognition results, this method can significantly reduce the overall misjudgment risk caused by single-modal recognition deviation, and improve the interpretability, safety and engineering deployment value of the multimodal emotion recognition system in real driving scenarios.

[0009] To address the aforementioned technical problems, this invention discloses a multimodal driver emotion recognition method in a real-vehicle environment, comprising the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. The confidence levels of the physiological and image modalities are obtained by determining the information entropy of each modality. The cosine dissimilarity of the predicted probability vectors of the image and physiological modalities is used as the modality conflict degree. The minimum value of the confidence level of the physiological and image modalities is determined. When the modality conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the predicted label is marked as a conflict state. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

[0010] Preferably, the step of outputting the predicted probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determining the predicted label corresponding to each modality through the predicted probability vectors, specifically includes: The SVM model is used to predict the physiological signals collected in real time, obtain the predicted probability of different emotion categories corresponding to the current physiological modality, and form the physiological modality prediction probability vector by combining the predicted probabilities of different emotion categories. The InceptionV3 network based on transfer learning predicts image signals, obtains the prediction probability of different emotion categories corresponding to the current image modality, and combines the prediction probabilities of different emotion categories into an image modality prediction probability vector. The random forest algorithm was used to classify the physiological modality prediction probability vector and the image modality prediction probability vector, respectively, to determine the accuracy of the prediction by the SVM model and the InceptionV3 network based on transfer learning. The weights of the two models are determined by the ratio of the accuracy of the SVM model and the InceptionV3 network obtained through transfer learning to the sum of the accuracies of the two models. The physiological modality prediction probability vector and the image modality prediction probability vector are weighted and summed according to the weights of the two models. The emotion category with the highest summation probability is used as the prediction label for each modality after decision fusion.

[0011] Preferably, the step of obtaining the confidence levels of the physiological and image modalities by determining the information entropy of each modality specifically involves: The different emotion categories include positive, calm, and negative; The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the image modalities obtained through identification are as follows: The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the physiological modality are as follows: Based on the predicted probability vectors of positive, calm, and negative emotions corresponding to the image and physiological modality, respectively, the information entropy corresponding to the image and physiological modality is obtained as follows: ; Based on the information entropy corresponding to the image and physiological modality, and by normalizing the information entropy to [0,1], the confidence level corresponding to each modality is defined as follows: .

[0012] Preferably, marking the prediction label as conflicted when the modal conflict degree is not less than a preset conflict threshold or the minimum value does not exceed a preset confidence threshold specifically includes: Define modal conflict degree The cosine dissimilarity of the predicted probability vectors for image modality and physiological modality: Set conflict threshold Confidence threshold If a state satisfies one of the following conditions, it is marked as a conflict state: When a conflict occurs, it is marked as 1; otherwise, it is marked as 0.

[0013] Preferably, the step of performing historical state smoothing correction on the predicted probabilities of the current physiological and image modalities to obtain the recognition probabilities of the current physiological and image modalities specifically includes: The emotional prediction label for a state of conflict is defined as: in, Indicates the tolerance for confidence gaps; The rejection state corresponds to insufficient output confidence / excessive conflict. This is corrected by smoothing through historical states. A sliding window is used to smooth the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities, resulting in the current recognition probability: in, It is the current recognition probability; It is a smoothing weight; It is the length of the historical window.

[0014] Preferably, it also includes a multimodal driver emotion recognition system in a real-vehicle environment, comprising: The emotion signal acquisition and recognition module is used to acquire the driver's physiological signals and corresponding facial expression image signals in real time while driving the vehicle, and to perform emotion recognition. It outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. The modal conflict recognition module is used to determine the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities based on the predicted probability vectors when the predicted labels are inconsistent. By determining the information entropy of each modality, the confidence of the physiological and image modalities is obtained respectively. The cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality is used as the modal conflict degree. The minimum value of the confidence of the physiological and image modalities is determined. When the modal conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the predicted label is marked as a conflict state. The identification correction module is used to perform historical state smoothing correction on the predicted probabilities of the current physiological and image modalities when the difference in confidence scores between the physiological and image modalities under conflict states is not greater than the confidence gap tolerance, so as to obtain the identification probabilities of the current physiological and image modalities.

[0015] Preferably, a computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. By determining the information entropy of each modality, the confidence levels of the physiological and image modalities are obtained respectively. The cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality is used as the modality conflict degree; the minimum confidence of the physiological and image modalities is determined, and the predicted label is marked as conflict state when the modality conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

[0016] Preferably, a computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. By determining the information entropy of each modality, the confidence levels of the physiological and image modalities are obtained respectively. The cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality is used as the modality conflict degree; the minimum confidence of the physiological and image modalities is determined, and the predicted label is marked as conflict state when the modality conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

[0017] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes a multimodal driver emotion recognition method in a real-world driving environment. Through emotion recognition, it outputs predicted probability vectors for different emotion categories corresponding to the current physiological and image modalities, and determines the predicted labels for each modality. By calculating the confidence and modal conflict degree of each modality, dynamic consistency analysis is performed on the recognition results to determine whether conflicts exist. When conflicts exist, the current recognition result is corrected through historical state smoothing correction to obtain the recognition probabilities for the current physiological and image modalities. Compared with traditional methods, this method significantly improves the accuracy of emotion recognition results through dynamic consistency analysis, thereby significantly reducing the overall misjudgment risk caused by single-modal recognition bias. This enhances the interpretability, safety, and engineering deployment value of the multimodal emotion recognition system in real-world driving scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart of the multimodal driver emotion recognition method in a real vehicle environment proposed in this invention; Figure 2 This is a flowchart illustrating the emotion induction and maintenance process of the present invention. Figure 3 This is a flowchart of the preliminary experiments for this invention; Figure 4 This is a flowchart of the emotion induction and data collection process of the present invention; Figure 5 The results of a grid search 3D scatter plot provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the transfer learning process provided in an embodiment of the present invention; Figure 7 This invention provides a physiological signal feature-level fusion emotion recognition model framework. Figure 8This invention provides a framework for a multimodal alignment and decision-level fusion emotion recognition model. Figure 9 A comparison of various emotion recognition models provided in the embodiments of the present invention; Figure 10 The confusion matrix of the All-RF+CNN classification model with decision-level fusion provided in the embodiments of the present invention. Detailed Implementation

[0019] The following will refer to the appendices in the embodiments of the present invention. Figures 1-10 The technical solutions in the embodiments of the present invention will be clearly and completely described. It should be understood that the terminology used in the present invention is only for describing particular implementation methods and is not intended to limit the present invention.

[0020] Example like Figure 1 As shown, this invention proposes a multimodal driver emotion recognition method in a real-vehicle environment, comprising the following steps: S1: Real-time acquisition of the driver's physiological signals and corresponding facial expression image signals while driving the vehicle, and emotion recognition, outputting the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determining the prediction label corresponding to each modality through the prediction probability vectors; S2: When the predicted labels are inconsistent, determine the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities based on the predicted probability vectors, and obtain the confidence scores of the physiological and image modalities by determining the information entropy of each modality. S3: Use the cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality as the modality conflict degree; determine the minimum confidence value of the physiological and image modalities; when the modality conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, mark the predicted label as a conflict state. S4: When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

[0021] Specifically, in step S1, the emotion induction and maintenance phase... Induction Phase: During the emotional induction phase, the driver's vehicle was stopped within a safe area, and specific videos were played for the driver. These videos were collected and downloaded from Bilibili (Bilibili is a Chinese video-sharing website) based on comments, targeting three categories: positive, calm, and negative emotions. For the negative emotion-inducing videos, multiple pauses were intentionally added using video editing software to further induce negative emotions in the driver. After each emotion induction experiment and the collection of physiological data, the driver was given 5 minutes to calm down. After watching the video, the SAM (Self-Awareness and Mood Scale) was completed, and the experimenters determined whether the emotion induction was effective. If effective, the driver was instructed to immediately start the vehicle and drive normally.

[0022] Maintenance Phase: Specific audio is used during driving to prolong the target emotional state and prevent its dilution due to high-attention driving in actual driving. For example, "Lotus Pond in the Moonlight" can maintain positive emotions in test drivers, while "Spring Garden" can maintain calm emotions. A self-made noise collection, randomly mixed with sounds such as renovation noise, construction noise, car horns, aircraft engine sounds, and the sound of a metal ball falling, can maintain negative emotions in test drivers, thus obtaining diverse emotional induction data under safe driving conditions. To support the effectiveness of the induction experiment, a pre-experiment for emotional induction is conducted before inducing emotions in the test drivers, such as... Figure 2 As shown. Based on the emotional state of the test drivers, additional emotional induction methods were used, such as asking the test drivers mathematical problems of addition, subtraction, multiplication, and division within 100 every 5 seconds during the driving process, requiring them to answer or describe something that made them angry or happy within 0-3 seconds, in order to achieve highly efficient induction and maintenance of emotions.

[0023] The preliminary experiment for emotion induction specifically includes: selecting all participants in the preliminary experiment who are physically healthy and free from mental disorders or hearing impairments; A preliminary experiment was conducted in a closed indoor experimental environment. Prepared induction videos and music were played to the participants, and the results of each participant's SAM emotional scale for each material were counted. After each participant experienced all the materials in a cycle, the average score of all participants for each material was calculated to evaluate the effectiveness of the prepared induction materials. The SAM emotional scale was used to calculate the pleasure and arousal levels of all participants after they experienced each experimental material. The corresponding induced emotions for each induced material were determined by calculating the mean and standard deviation of the self-rated scores.

[0024] Throughout the experiment, the test drivers drove for approximately 5 minutes to allow them to become familiar with the vehicle and calm their emotions. During this time, the relevant procedures and precautions were explained to the test drivers. Under safe driving conditions, different audio recordings that maintained a stable state were played to induce different emotions in the test drivers, making their emotional transitions during the experiment more natural.

[0025] A multi-channel emotion induction and maintenance mechanism is applicable to the "emotion-task" coupled scenarios in real-world vehicle environments: multi-channel emotion induction produces a stronger and more stable emotion induction effect, compensating for the shortcomings of a single channel and adapting to the sensory preferences of different individuals. The maintenance mechanism can continuously and dynamically maintain a certain state for a period of time, better matching task requirements. In a real-world driving environment, dynamic disturbances such as changes in lighting and traffic pressure continuously affect emotional states. Compared to simulated driving environments, the realism and intensity of emotional responses are higher, resulting in higher data ecological validity.

[0026] like Figure 3 As shown, after the emotions were induced through audio, video, and verbal interaction, the SAM scale was used to assess the subjective self-rating of the emotions of the participants. Based on the participants' self-ratings of their emotions, the experimenters judged the effectiveness of the emotion induction.

[0027] To facilitate rapid assessment of the drivers' emotions, the experiment used two dimensions of the SAM scale: pleasure and arousal. Pleasure reflects the positive or negative nature of an individual's emotional state, i.e., its degree of activity (positive or negative); the more pleasant the mood, the higher the score, and the closer the score is to 1; the more unpleasant the mood, the closer the score is to 9. Arousal represents the degree to which emotions are aroused, primarily measuring the intensity of the emotion; the higher the score, the closer the score is to 1, such as excitement, agitation, and tension; the less excited the mood, the closer the score is to 9, such as feeling relaxed, calm, and boredom. After the driving session ended and data collection was completed, the drivers were asked to recount their driving experience to determine whether they successfully maintained their emotions during the driving process.

[0028] like Figure 4As shown, to verify the effectiveness of the prepared videos and audio, a pre-experiment was first conducted. This example recruited 10 participants (5 males and 5 females) with an average age of 21. All participants were in good physical health and had no mental or hearing impairments. Before the experiment began, each participant filled out a personal information form and carefully read and signed the informed consent form. The pre-experiment was conducted in a closed indoor environment. Prepared induction videos and music were played to the participants, and the SAM scale results for each participant on each piece of material were statistically analyzed. After each participant cycled through all the materials, the mean of the self-rating scores for each material was calculated to assess the effectiveness of the prepared induction materials. The SAM scale scores of pleasure and arousal were statistically analyzed after all participants experienced each piece of material, and the mean and standard deviation of the self-rating scores were calculated to conclude that each induction material could effectively induce the corresponding emotions.

[0029] This invention collects physiological and facial expression signals from test drivers operating real vehicles under real-world road traffic conditions. The test drivers include drivers of different ages and genders, and the experimental scenarios include different daytime and nighttime conditions. During the real-vehicle experiments, the test drivers sit in the driver's seat and operate the vehicle, while the experimenters sit in the passenger seat to induce the driver's emotions and use a laptop to record data using HRT software.

[0030] The physiological signal acquisition equipment, namely electrocardiogram (ECG) and electrodermal signal acquisition (EDS) signals, uses the Physiolab physiological signal acquisition device manufactured by Winful Instruments Technology (Shanghai) Co., Ltd., in conjunction with its accompanying HRT software. The Physiolab physiological signal acquisition device supports up to 8 signal input channels. During use, one end of the sensor wire is inserted into the device's input channel, and the other end is attached to the body using electrode patches. Before attaching the electrode patches, the skin should be cleaned with alcohol; if necessary, tape can be used to secure them. The Physiolab physiological signal acquisition device connects to the computer via Bluetooth. The HRT software can analyze the received signals in real time and visualize and record them. The instrument's sampling frequency is 1000 Hz.

[0031] The facial expression signal acquisition device uses a vehicle-mounted dashcam. The device includes three cameras: left, right, and front. The device is placed on the windshield in front of the driver without obstructing their view. The left camera can capture a frontal facial video image of the driver. The video acquisition supports a maximum resolution of 1080P and a sampling frequency of 25Hz. The acquired videos from the left, right, and front angles are uniformly timestamped with watermarks to facilitate the alignment of the video containing facial expression signals with the acquired physiological signals.

[0032] Before the experiment began, the researchers introduced the basic requirements and road conditions to the driver participants, instructing them to drive as they normally would. They then fitted the participants with physiological signal measurement devices and adjusted the signal acquisition equipment, observing whether the signal acquisition was normal. Once the signal acquisition page was confirmed to be displaying correctly, the experiment could begin.

[0033] After preprocessing the electrocardiogram (ECG) and electrodermal signal (EDA) data, valid samples were selected from the physiological signal data of 20 driver participants. Abnormal samples were removed due to excessive driving maneuvers, signal loss, or electrode detachment during the experiment. A sliding time window was used to extract training samples. Since the physiological data acquisition frequency was 1000Hz, the time window was set to 20,000 points (20 seconds), with a sliding step of 5,000 points (5 seconds). Each sliding time window contained one sample. If the data collected in a single experiment did not reach 20,000 points (20 seconds) in the last time window, that window was discarded. Each sample was automatically labeled according to the corresponding emotional state at the time of data collection: 0 represented calm emotion, -1 represented negative emotion, and 1 represented positive emotion.

[0034] The raw facial expression signal data was collected as a video file at 25 frames per second. First, one photograph was extracted every 50 frames (2 seconds) as a data sample. After preprocessing the facial expression signals, valid samples were initially selected from the facial expression signal data of 20 test drivers, eliminating abnormal samples due to reasons such as closed eyes or excessive head turning during the experiment. Each sample data was automatically labeled according to the corresponding emotional state at the time of data collection: 0 represents calm emotion, -1 represents negative emotion, and 1 represents positive emotion.

[0035] Based on the emotional feedback records (SAM score + verification results) corresponding to the induction phase, the system automatically assigns the emotional state label to all samples within the sampling window, achieving cross-modal and highly consistent automatic emotional labeling. Each sample structure includes an ECG / EDA signal segment, an image frame sequence, timestamp information, and a unified emotional label.

[0036] The emotion state label generation mechanism is based on subjective evaluation by SAM combined with expert review: the emotion is quantified by two dimensions: pleasure and arousal. The "self-evaluation-review-optimization" process realizes a dynamic feedback loop to ensure the consistency and credibility of emotion labels.

[0037] Each sample data is labeled according to the emotional state corresponding to the time of data collection: 0 represents calm emotion, -1 represents negative emotion, and 1 represents positive emotion; and the emotional state labels corresponding to different emotion categories are assigned to all samples in the sampling window.

[0038] Based on the scoring results and the results of expert review of the scores, emotional state labels corresponding to different emotional categories are determined, and the effectiveness of emotional induction is comprehensively judged.

[0039] When effective, real-time signal acquisition is performed by playing audio and video recordings of different emotional categories that have been successfully induced in pre-trial trials to the driver, as well as audio, video, and verbal interactions used to prolong the target emotional state.

[0040] Emotion recognition specifically includes: The SVM model is used to predict the physiological signals collected in real time, obtain the predicted probability of different emotion categories corresponding to the current physiological modality, and form the physiological modality prediction probability vector by combining the predicted probabilities of different emotion categories. The InceptionV3 network based on transfer learning predicts image signals, obtains the prediction probability of different emotion categories corresponding to the current image modality, and combines the prediction probabilities of different emotion categories into an image modality prediction probability vector. The random forest algorithm was used to classify the physiological modality prediction probability vector and the image modality prediction probability vector, respectively, to determine the accuracy of the prediction by the SVM model and the InceptionV3 network based on transfer learning. The weights of the two models are determined by the ratio of the accuracy of the SVM model and the InceptionV3 network obtained through transfer learning to the sum of the accuracies of the two models. The physiological modality prediction probability vector and the image modality prediction probability vector are weighted and summed according to the weights of the two models. The emotion category with the highest summation probability is used as the prediction label for each modality after decision fusion.

[0041] In step S2, the confidence levels for the physiological and image modalities are obtained respectively, specifically including: The different emotion categories include positive, calm, and negative; The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the image modalities obtained through identification are as follows: The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the physiological modality are as follows: Based on the predicted probability vectors of positive, calm, and negative emotions corresponding to the image and physiological modality, respectively, the information entropy corresponding to the image and physiological modality is obtained as follows: The smaller the entropy value of information entropy, the more confident the mode is in the prediction result; Based on the information entropy corresponding to the image and physiological modality, and by normalizing the information entropy to [0,1], the confidence level corresponding to each modality is defined as follows: The higher the confidence level, the more confident the mode is in the prediction results.

[0042] In step S3, the predicted label is marked as conflicting, specifically including: Define modal conflict degree The cosine dissimilarity of the predicted probability vectors for image modality and physiological modality: Set conflict threshold Confidence threshold If a state satisfies one of the following conditions, it is marked as a conflict state: When a conflict occurs, it is marked as 1; otherwise, it is marked as 0.

[0043] A multimodal training sample construction method combining a sliding window mechanism and a temporal synchronization strategy is proposed: Long-term temporal data is segmented using a sliding window to extract multimodal features within local segments, avoiding computational redundancy in global modeling. Dynamic time warping is utilized to address temporal drift between different modalities, ensuring strict alignment of multimodal data within the window.

[0044] In step S4, the recognition probabilities of the current physiological modality and image modality are obtained, specifically including: The emotional prediction label for a state of conflict is defined as: in, Indicates the tolerance for confidence gaps; The rejection state corresponds to insufficient output confidence / excessive conflict. This is corrected by smoothing through historical states. A sliding window is used to smooth the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities, resulting in the current recognition probability: in, It is the current recognition probability; It is a smoothing weight; It is the length of the historical window.

[0045] The multimodal driver emotion recognition method proposed in this invention under real vehicle conditions can significantly improve the accuracy of emotion recognition results, significantly reduce the overall misjudgment risk caused by single-modal recognition bias, and enhance the interpretability, safety, and engineering deployment value of the multimodal emotion recognition system in real driving scenarios.

[0046] like Figure 6 As shown, the InceptionV3 network, employing transfer learning, is used to extract emotional features from facial images. Figure 6 Only the feature extraction layer of the model was transferred, and the classification layer was redesigned. Considering that the InceptionV3 model has strong feature extraction capabilities, an overly powerful classification layer could easily lead to overfitting. Therefore, the model's classification layer was set to two layers: the first layer had 1024 neurons with the ReLU activation function, and the second layer had 3 neurons with the softmax activation function, outputting the probability that an image belongs to each emotion category. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio. After each epoch of training, the accuracy was tested on the validation set, and finally, the model's training performance was evaluated based on the classification results on the test set.

[0047] like Figure 7 As shown, SVM or random forest is used to extract time-domain and frequency-domain features from ECG / EDA signals. When constructing an emotion recognition model based on physiological signals, the constructed dataset is randomly divided into training and test sets in an 8:2 ratio. The training set is used as input data for model training, and the test set is used as data for model performance testing. When setting SVM parameters, the core hyperparameters C, kernel, and gamma are automatically tuned during training using grid search, as shown below. Figure 5 As shown, C is in the range of 0.1, 1, 10, 100; gamma is in the range of 1, 0.1, 0.01, 0.001, scale, auto; kernel is in the range of linear, poly, rbf. The system systematically traverses various parameter combinations and determines the optimal parameters through cross-validation. Finally, the core hyperparameters of SVM are set as: rbf kernel function, C=10, gamma=1, and the other hyperparameters use the default parameters.

[0048] The extracted time-domain and frequency-domain features are concatenated and stitched with the facial image emotion features to achieve feature fusion. Then, the RF algorithm is used to classify the fused features for emotion.

[0049] Multimodal alignment and decision-level fusion emotion recognition framework, such as Figure 8As shown, decision-level fusion is achieved through a weighted voting method: First, the weighted weights of the emotion recognition model based on physiological signal feature-level fusion and the emotion recognition model based on facial expression are calculated by the ratio of the accuracy of a single model to the sum of the accuracies of all models. Next, the specific probabilities of the test sample in the three classification results of the two models are calculated. Finally, these probabilities are weighted and summed, and the emotion label corresponding to the larger sum of probabilities is taken as the final test label.

[0050] The weighted weights and the final decision level are combined to calculate The probability of a label is calculated using the following formula: In the formula, For a single model The accuracy of the model; for The weights of the model; for Model prediction The probability of the label; For final decision-level fusion calculation The probability of the label.

[0051] Support for dual-layer feature-decision structure verification of sample recognition performance: Current methods for driver emotion recognition mainly rely on facial expression analysis, speech analysis and recognition, or physiological signals. However, single-modal signal data provides relatively limited information and is easily affected by environmental interference, which can lead to the inability to collect single-modal data. For example, complex and changing driving environments (such as low light at night, extreme weather interference, and facial occlusion) can easily cause signal acquisition errors.

[0052] like Figure 9 As shown, this embodiment compares the classification performance of seven models based on different modalities and fusion strategies. It can be seen that the multimodal emotion classification model significantly outperforms the unimodal model, especially the FACE-CNN model, which achieves a maximum accuracy of 95.56%. However, this model is based on unimodal facial expression signals, and if environmental interference occurs during facial expression signal acquisition in practical applications, the model's results will be significantly compromised. The All-RF+CNN classification model, using decision-level fusion, achieves an accuracy of 95.24% on top of the 93.33% accuracy of the ECG+EDA-RF and FACE-CNN models. Because this classification model is based on multimodal fusion, it exhibits stronger stability and generalization ability, and demonstrates excellent classification performance, fully proving the effectiveness of the weighted voting decision-level fusion strategy based on multimodal fusion, which can effectively improve emotion recognition.

[0053] like Figure 10 As shown, this is the confusion matrix of the decision-level fusion All-RF+CNN model provided in this embodiment, which shows that the model achieves a good balance between precision and recall, and the model has strong comprehensive prediction ability for each category.

[0054] This invention also proposes a multimodal driver emotion recognition system in a real-vehicle environment, comprising: The emotion signal acquisition and recognition module is used to acquire the driver's physiological signals and corresponding facial expression image signals in real time while driving the vehicle, and to perform emotion recognition. It outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. The modal conflict recognition module is used to determine the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities based on the predicted probability vectors when the predicted labels are inconsistent. By determining the information entropy of each modality, the confidence of the physiological and image modalities is obtained respectively. The cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality is used as the modal conflict degree. The minimum value of the confidence of the physiological and image modalities is determined. When the modal conflict degree is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the predicted label is marked as a conflict state. The identification correction module is used to perform historical state smoothing correction on the predicted probabilities of the current physiological and image modalities when the difference in confidence scores between the physiological and image modalities under conflict states is not greater than the confidence gap tolerance, so as to obtain the identification probabilities of the current physiological and image modalities.

[0055] In summary, the driver emotion induction and recognition method and system proposed in this invention for real-world driving environments first employs a three-channel combined induction approach: visual video, music / audio, and verbal interaction. While ensuring driving safety, it guides the driver into a target emotional state through carefully designed stimuli, and uses the SAM emotion scale to provide real-time feedback on the driver's emotions, verifying the induction effect. During actual driving, the system continuously plays sustaining music or conducts interactive questioning to maintain the driver's stable emotional state. Simultaneously, ECG / EDA physiological signals are acquired using a Physiolab device, along with facial video data obtained from an onboard camera. All signals are timestamped to ensure precise temporal alignment and consistent labeling of multimodal data, thereby constructing a high-quality training sample set with standardized structure and reliable labels.

[0056] In terms of model construction, the system adopts a dual fusion strategy combining feature-level and decision-level approaches. First, the features corresponding to the extracted ECG and EDA signals are concatenated and fused to establish a physiological signal emotion recognition model based on the random forest algorithm. Then, time window alignment technology is used to address the temporal discrepancy between physiological signals and facial expressions. A weighted voting method is employed to achieve decision-level fusion of multimodal recognition results, ultimately forming a driver emotion recognition system with personalized features. This solution significantly improves the accuracy and practicality of emotion recognition in in-vehicle environments through standardized emotion induction procedures, precise multimodal data synchronous acquisition, and innovative dual fusion algorithms.

[0057] This invention introduces real-time observation of the driver's performance by an experimenter, combined with feedback after the driving session, to achieve multi-dimensional cross-validation of subjective self-evaluation and subjective peer evaluation. This ensures accurate assessment of the emotion-inducing effect and significantly improves the accuracy and reliability of emotion state determination. This improvement effectively addresses the dual shortcomings of traditional methods in terms of both the intensity of emotion induction and the reliability of state determination.

[0058] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0059] Furthermore, unless otherwise stated, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. All references to this specification are incorporated by way of citation to disclose and describe methods relating to those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

Claims

1. A multimodal driver emotion recognition method in a real-vehicle environment, characterized in that, Includes the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. By determining the information entropy of each modality, the confidence levels of the physiological and image modalities are obtained respectively. The cosine dissimilarity of the predicted probability vectors of image modality and physiological modality is used as the modality conflict degree. Determine the minimum confidence level for physiological and image modalities. When the modal conflict level is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the prediction label is marked as a conflict state. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

2. The multimodal driver emotion recognition method in a real vehicle environment according to claim 1, characterized in that, The step of outputting prediction probability vectors for different emotion categories corresponding to the current physiological and image modalities, and determining the prediction label for each modality using the prediction probability vectors, specifically includes: The SVM model is used to predict the physiological signals collected in real time, obtain the predicted probability of different emotion categories corresponding to the current physiological modality, and form the physiological modality prediction probability vector by combining the predicted probabilities of different emotion categories. The InceptionV3 network based on transfer learning predicts image signals, obtains the prediction probabilities of different emotion categories corresponding to the current image modality, and combines the prediction probabilities of different emotion categories into an image modality prediction probability vector. The random forest algorithm was used to classify the physiological modality prediction probability vector and the image modality prediction probability vector, respectively, to determine the accuracy of the prediction by the SVM model and the InceptionV3 network based on transfer learning. The weights of the two models are determined by the ratio of the accuracy of the SVM model and the InceptionV3 network obtained through transfer learning to the sum of the accuracies of the two models. The physiological modality prediction probability vector and the image modality prediction probability vector are weighted and summed according to the weights of the two models. The emotion category with the highest summation probability is used as the prediction label for each modality after decision fusion.

3. The multimodal driver emotion recognition method in a real vehicle environment according to claim 2, characterized in that, The step of determining the information entropy of each modality and obtaining the confidence levels of the physiological and image modalities respectively includes: The different emotion categories include positive, calm, and negative; The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the image modalities obtained through identification are as follows: The predicted probability vectors for the three different emotion categories—positive, calm, and negative—corresponding to the physiological modality are as follows: Based on the predicted probability vectors of positive, calm, and negative emotions corresponding to the image and physiological modality, respectively, the information entropy corresponding to the image and physiological modality is obtained as follows: ； Based on the information entropy corresponding to the image and physiological modality, and by normalizing the information entropy to [0,1], the confidence level corresponding to each modality is defined as follows: 。 4. The multimodal driver emotion recognition method in a real vehicle environment according to claim 3, characterized in that, When the modal conflict degree is not less than a preset conflict threshold or the minimum value does not exceed a preset confidence threshold, the prediction label is marked as a conflict state, specifically including: Define modal conflict degree The cosine dissimilarity of the predicted probability vectors for image modality and physiological modality: Set conflict threshold Confidence threshold If a state satisfies one of the following conditions, it is marked as a conflict state: When a conflict occurs, it is marked as 1; otherwise, it is marked as 0.

5. The multimodal driver emotion recognition method in a real vehicle environment according to claim 4, characterized in that, The process of smoothing and correcting the predicted probabilities of the current physiological and image modalities using historical states to obtain the recognition probabilities of the current physiological and image modalities specifically includes: The emotional prediction label for a state of conflict is defined as: in, Indicates the tolerance for confidence gaps; The rejection state corresponds to insufficient output confidence / excessive conflict. This is corrected by smoothing through historical states. A sliding window is used to smooth the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities, resulting in the current recognition probability: in, It is the current recognition probability; It is a smoothing weight; It is the length of the historical window.

6. A multimodal driver emotion recognition system in a real-vehicle environment, characterized in that, include: The emotion signal acquisition and recognition module is used to acquire the driver's physiological signals and corresponding facial expression image signals in real time while driving the vehicle, and to perform emotion recognition. It outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. The modal conflict identification module is used to determine the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities based on the predicted probability vectors when the predicted labels are inconsistent. By determining the information entropy of each modality, the confidence of the physiological and image modalities is obtained respectively. The cosine dissimilarity of the predicted probability vectors of the image modality and the physiological modality is used as the modal conflict degree. Determine the minimum confidence level for physiological and image modalities. When the modal conflict level is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the prediction label is marked as a conflict state. The identification correction module is used to perform historical state smoothing correction on the predicted probabilities of the current physiological and image modalities when the difference in confidence scores between the physiological and image modalities under conflict states is not greater than the confidence score gap tolerance, so as to obtain the identification probabilities of the current physiological and image modalities.

7. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. By determining the information entropy of each modality, the confidence levels of the physiological and image modalities are obtained respectively. The cosine dissimilarity of the predicted probability vectors of image modality and physiological modality is used as the modality conflict degree. Determine the minimum confidence level for physiological and image modalities. When the modal conflict level is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the prediction label is marked as a conflict state. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the following steps: The system collects physiological signals and corresponding facial expression image signals of the driver while driving the vehicle in real time, performs emotion recognition, outputs the prediction probability vectors of different emotion categories corresponding to the current physiological and image modalities, and determines the prediction label corresponding to each modality through the prediction probability vectors. When the predicted labels are inconsistent, the predicted probabilities of different emotion categories corresponding to the current physiological and image modalities are determined based on the predicted probability vectors. By determining the information entropy of each modality, the confidence levels of the physiological and image modalities are obtained respectively. The cosine dissimilarity of the predicted probability vectors of image modality and physiological modality is used as the modality conflict degree. Determine the minimum confidence level for physiological and image modalities. When the modal conflict level is not less than the preset conflict threshold or the minimum value does not exceed the preset confidence threshold, the prediction label is marked as a conflict state. When the difference in confidence between the physiological and image modalities under conflict conditions is not greater than the confidence gap tolerance, the predicted probabilities of the current physiological and image modalities are smoothed by historical state correction to obtain the recognition probabilities of the current physiological and image modalities.

Images