Multi-mode driver emotion recognition method, device, equipment and medium

Abstract

The invention discloses a multi-mode driver emotion recognition method, device and equipment and a medium, and the method comprises the steps: collecting the multi-mode information of a driver after a vehicle is started; performing feature extraction on each piece of modal information in the multi-modal information to obtain multiple pieces of feature information; scoring the multiple pieces of feature information in multiple dimensions based on a pre-established emotion recognition model to obtain respective scores of each piece of feature information in multiple dimensions; fusing the score of each piece of feature information under the titer dimension to obtain a fused titer dimension score, and fusing the score of each piece of feature information under the wake-up dimension to obtain a fused wake-up dimension score; and based on the fused titer dimension score and the fused wake-up dimension score, identifying the emotion level of the driver. According to the method, the emotion state of the driver can be comprehensively collected, and the emotion can be accurately recognized.

Description

Technical Field

[0001] The present invention relates to the technical field of the combination of neuroscience and information technology, and in particular to a multimodal driver emotion recognition method, device, equipment and medium. Background Art

[0002] With the development of the automobile industry, traffic accidents have always been a serious problem faced by all countries. Most accidents can be effectively prevented by establishing laws and regulations, driving assistance, road warnings, etc., but there is no effective means to timely detect and stop emotional driving in accidents. The Institute of Psychology of the Chinese Academy of Sciences once randomly selected 900 drivers in multiple cities to conduct a questionnaire survey. About 35% of the drivers admitted that they belonged to the "road rage tribe." "Road rage" usually refers to anger caused by traffic jams or misunderstanding of other drivers' driving behaviors. Under such emotions, the perpetrators may make random changes in lanes, force overtaking, run yellow lights, swear, etc. In severe cases, they may even attack other vehicles or drivers, which will reduce driving safety and increase the possibility of traffic accidents. Summary of the invention

[0003] The embodiments of the present application provide a multimodal driver emotion recognition method, device, equipment and medium. The method can collect the driver's emotional state more comprehensively, and identify emotions more accurately from the perspective of the impact of the valence dimension and arousal dimension of emotions on driving during driving.

[0004] In a first aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0005] A multimodal driver emotion recognition method, comprising:

[0006] After the vehicle is started, the driver's multimodal information is collected; feature extraction is performed on each modal information in the multimodal information to obtain multiple feature information; based on a pre-established emotion recognition model, the multiple feature information is scored in multiple dimensions to obtain scores of each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between the feature information and the valence dimension score, and a correspondence between the feature information and the arousal dimension score; the score of each feature information in the valence dimension is fused to obtain a fused valence dimension score, and the score of each feature information in the arousal dimension is fused to obtain a fused arousal dimension score; based on the fused valence dimension score and the fused arousal dimension score, the driver's emotion level is identified, and auxiliary control of the vehicle is performed based on the emotion level.

[0007] Preferably, the establishment of the emotion recognition model includes: obtaining a sample data set, the sample data set including multiple feature information extracted from multiple groups of multimodal information; annotating each feature information in the sample data set with a valence dimension score and annotating the arousal dimension score to obtain a training set, the training set including a valence dimension score and an arousal dimension score corresponding to each feature information; based on the training set, training a pre-constructed initial model to obtain the emotion recognition model.

[0008] Preferably, the step of annotating each feature information in the sample data set with a valence dimension score and annotating each feature information in the sample data set with a arousal dimension score comprises: annotating each feature information in the sample data set with a valence dimension score and annotating each feature information in the sample data set with a valence dimension score according to a scoring rule of a SAM scale.

[0009] Preferably, the multimodal information includes EEG signals, facial data, eye movement data and voice data, and feature extraction is performed on each modal information in the multimodal information to obtain multiple feature information, including: performing a first feature extraction on the EEG signal to obtain a power spectrum density; performing a second feature extraction on the facial data to obtain target image data; performing a third feature extraction on the eye movement data to obtain the pupil diameter size and the pupil diameter change rate; performing a fourth feature extraction on the voice data to obtain a Mel spectrum feature.

[0010] Preferably, the emotion recognition model includes a brain electronic model, a facial expression sub-model, an eye movement sub-model and a speech sub-model, and the pre-constructed initial model is trained based on the training set to obtain the emotion recognition model, including: training the pre-constructed initial brain electronic model according to each power spectral density and the valence dimension score and the arousal dimension score corresponding to each power spectral density to obtain a trained brain electronic model; training the pre-constructed initial facial expression sub-model according to each target image data and the valence dimension score and the arousal dimension score corresponding to each target image data to obtain a trained facial expression sub-model; training the pre-constructed initial eye movement sub-model according to the valence dimension score corresponding to each pupil diameter size and the arousal dimension score corresponding to each pupil diameter change rate to obtain a trained eye movement sub-model; training the pre-constructed speech sub-model according to each Mel spectrum feature and the valence dimension score and the arousal dimension score corresponding to each Mel spectrum feature to obtain a trained speech sub-model.

[0011] Preferably, the method of identifying the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score comprises: taking the middle value of the valence dimension score and the arousal dimension score as the coordinate origin, the horizontal axis represents the valence dimension score, the valence dimension score increases from the negative direction to the positive direction of the coordinate, and the vertical axis represents the arousal dimension score, and the arousal dimension score increases from the negative direction to the positive direction of the coordinate; according to the first coordinate point corresponding to the minimum value of the valence dimension score and the minimum value of the arousal dimension score, the second coordinate point corresponding to the minimum value of the valence dimension score and the maximum value of the arousal dimension score, and the maximum value of the valence dimension score and the arousal dimension score. The third coordinate point corresponding to the minimum score, the fourth coordinate point corresponding to the maximum value of the valence dimension score and the maximum value of the arousal dimension score, the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point are connected to construct an emotional area; the emotional area is divided into multiple sub-areas, each single sub-area corresponds to an emotional level of the driver, wherein each sub-area contains four quadrants of the coordinates, and the negative direction score and the positive direction score of each sub-area are symmetrical based on the origin; based on the area where the fused valence dimension score is located and the area where the fused arousal dimension score is located, the emotional level of the driver is identified.

[0012] Preferably, the fusing the scores of each feature information under the valence dimension to obtain a fused valence dimension score, and the fusing the scores of each feature information under the arousal dimension to obtain a fused arousal dimension score, include: fusing according to the scores of each feature information under the valence dimension and a preset first weight set to obtain a fused valence dimension score; fusing according to the scores of each feature information under the arousal dimension and a preset second weight set to obtain a fused arousal dimension score.

[0013] In a second aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0014] A multimodal driver emotion recognition device, comprising:

[0015] An acquisition module, used to collect multimodal information of the driver after the vehicle is started;

[0016] An acquisition module, used for extracting features from each modal information in the multimodal information to obtain multiple feature information;

[0017] An acquisition module, configured to score the plurality of feature information in multiple dimensions based on a pre-established emotion recognition model, and obtain scores of each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between the feature information and the valence dimension score, and a correspondence between the feature information and the arousal dimension score;

[0018] An acquisition module, configured to fuse the scores of each feature information under the valence dimension to obtain a fused valence dimension score, and to fuse the scores of each feature information under the arousal dimension to obtain a fused arousal dimension score;

[0019] An acquisition module is used to identify the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score, and to perform auxiliary control of the vehicle based on the emotion level.

[0020] In a third aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0021] An electronic device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods described in the first aspect are implemented.

[0022] In a fourth aspect, the present invention provides the following technical solution through an embodiment of the present invention:

[0023] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0025] The multimodal driver emotion recognition method provided by the embodiment of the present invention collects the multimodal information of the driver, extracts the features of the multimodal information to obtain multiple feature information, and then collects more comprehensive driver behavior state information, and then scores the multiple feature information in the valence dimension and the arousal dimension respectively. The multi-dimensional scoring can more accurately reflect the driver's emotional state, and then the scores of each feature information in multiple dimensions are integrated to determine the driver's emotional level. The present application can collect and analyze the driver's emotional state more comprehensively, from the perspective of the impact of the valence dimension and the arousal dimension of emotions on driving during driving, which is conducive to more accurate identification of emotions, more comprehensive reflection of the driver's driving state, identification of the driver's emotional level, and appropriate vehicle auxiliary control based on the emotional level to improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 is a flow chart of a multimodal driver emotion recognition method in an embodiment of the present invention;

[0028] Figure 2 A result block diagram of multimodal information collection in an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of the performance characteristics of the valence dimension and the arousal dimension in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a tree-like LSTM model in an embodiment of the present invention;

[0031] Figure 5 Schematic diagram of the structure of a multi-task convolutional neural network model in an embodiment of the present invention;

[0032] Figure 6 Schematic diagram of the structure of a fully convolutional neural network model in an embodiment of the present invention;

[0033] Figure 7 Schematic diagram of the weight determination process in an embodiment of the present invention;

[0034] Figure 8 Schematic diagram of the structure of the PAD emotion model in an embodiment of the present invention;

[0035] Fig. 9 A schematic diagram of performing vehicle auxiliary control based on emotion level in an embodiment of the present invention;

[0036] Fig.10 is a structural block diagram of the emotion recognition process in an embodiment of the present invention;

[0037] Fig.11 Schematic diagram of the structure of a multi-modal driver emotion recognition device in an embodiment of the present invention;

[0038] Fig.12 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The embodiments of the present application provide a multimodal driver emotion recognition method, device, equipment and medium. The method can collect the driver's emotional state more comprehensively, and identify emotions more accurately from the perspective of the impact of the valence dimension and arousal dimension of emotions on driving during driving.

[0040] The technical solution of the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0041] A multimodal driver emotion recognition method comprises: collecting multimodal information of the driver after the vehicle is started; extracting features of each modal information in the multimodal information respectively to obtain multiple feature information; scoring the multiple feature information in multiple dimensions based on a pre-established emotion recognition model to obtain scores of each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a corresponding relationship between the feature information and the valence dimension score, and a corresponding relationship between the feature information and the arousal dimension score; fusing the score of each feature information in the valence dimension to obtain a fused valence dimension score, and fusing the score of each feature information in the arousal dimension to obtain a fused arousal dimension score; identifying the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score, and performing auxiliary control on the vehicle based on the emotion level.

[0042] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0043] In a first aspect, an embodiment of the present invention provides a multimodal driver emotion recognition method, specifically, as follows: Figure 1 As shown, the method includes the following steps S101 to S105:

[0044] Step S101, after the vehicle is started, collecting multimodal information of the driver;

[0045] Step S102: extracting features from each modal information in the multimodal information to obtain a plurality of feature information.

[0046] In a specific embodiment, the multimodal information may include EEG signals, facial data, eye movement data, and voice data. Figure 2 As shown, the collection of multimodal information may include collecting the driver's electroencephalogram signals through a brain-computer interface, collecting the driver's facial data and eye movement data through a camera, and collecting the driver's voice data through a microphone.

[0047] Specifically, the brain-computer interface uses a non-invasive wireless dry electrode EEG cap. The camera and microphone can be built-in devices in the vehicle or external devices. For example, an integrated kinect device is used, which includes an RGB camera to obtain 640×480 color images at 30 frames per second, two depth sensors to detect the driver's position, a set of four-element microphone arrays on both sides for sound source positioning and speech recognition, and a built-in motor base at the bottom to adjust the pitch angle. The driver's facial expressions, eye movement signals, voice intonation and other physiological information can be accurately collected through the kinect device. For the convenience of description, this example uses the kinect device. When the vehicle starts, the driver wears an EEG cap to collect the driver's EEG signals in real time. At the same time, the kinect device or the camera and microphone in the car collect the driver's facial image information and sound information in real time.

[0048] In order to obtain more accurate and interference-free information, before extracting features from each modal information in the multimodal information, the method may further include: preprocessing each modal information in the multimodal information to obtain preprocessed modal information.

[0049] Specifically, the signals collected by the EEG cap and the Kinect device are preprocessed respectively. The preprocessing of EEG signals includes filtering, baseline drift elimination, artifact elimination, feature extraction and other steps. There are many high-frequency noises in the data collected by the brain-computer interface, such as 60Hz power frequency interference, which are filtered out by low-pass filtering; baseline drift refers to the DC bias that changes with time, usually caused by changes in skin potential or equipment aging, and is eliminated by baseline drift to ensure consistency during the entire recording period; artifacts are usually caused by non-brain source signals, such as facial expressions, muscle activities, etc., and independent principal component analysis is used here to remove them, and finally feature extraction is performed based on emotion-specific EEG features.

[0050] In a specific embodiment, feature extraction is performed on each modal information in the multimodal information to obtain multiple feature information, which may include: performing a first feature extraction on the EEG signal to obtain a power spectral density; performing a second feature extraction on the facial data to obtain the target image data; performing a third feature extraction on the eye movement data to obtain the pupil diameter size and the pupil diameter change rate; performing a fourth feature extraction on the speech data to obtain a Mel spectrum feature.

[0051] Specifically, performing a first feature extraction on the EEG signal to obtain a power spectral density may include: extracting features through a wavelet transform method, which can be regarded as an improved method of short-time Fourier transform and can better process time-frequency signals. Since the EEG signal has power spectral densities in different frequency bands, Daubechies wavelet transform coefficients that can support multiple frequency bands are used to perform frequency domain conversion, thereby realizing feature extraction of the EEG signal.

[0052] Performing a second feature extraction on the facial data to obtain the target image data may include: performing noise reduction and purification on the facial data to obtain a grayscale image of the target size. Specifically, for a photo containing a face, the first step is to detect the 68 key points of the face, and then rotate and align the face and the key points of the face, and use the line connecting the centers of the two eyes as the horizontal line of the rotated image. The coordinates of the centers of the left and right eyes are (x1, y1) and (x2, y2), respectively. Assuming the angle to be rotated is θ, the calculation formula can be expressed as:

[0053]

[0054] Finally, the face part of the image is cropped according to the key points, and the size of the grayscale image is adjusted to 48×48×1 as the input shape of the model, so as to extract the effective part (i.e. the face part) and avoid interference of other objects in the natural environment on feature learning.

[0055] Extracting the third feature of the eye movement data to obtain the pupil diameter and the pupil diameter change rate may include: identifying the frame containing the driver's eyes in the video, capturing the picture and estimating the driver's pupil diameter, and calculating the pupil diameter change rate according to the timestamp. When a person is in a neutral emotion, the pupil diameter is the smallest, followed by positive emotions, and the pupil diameter is the largest when negative. Therefore, this application uses pupil diameter as the emotional feature of the eye movement signal.

[0056] Performing a fourth feature extraction on the speech data to obtain a Mel spectrum feature may include: extracting the time-frequency features of the source speech by short-time Fourier transform, and after obtaining the time-frequency features, converting them into Mel spectrum features by a Mel filter bank.

[0057] Step S103, scoring the multiple feature information in multiple dimensions based on a pre-established emotion recognition model to obtain scores for each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between the feature information and the valence dimension score, and a correspondence between the feature information and the arousal dimension score.

[0058] In a specific embodiment, establishing an emotion recognition model may include: obtaining a sample data set, the sample data set including multiple feature information extracted from multiple groups of multimodal information; annotating each feature information in the sample data set with a valence dimension score and annotating the arousal dimension score to obtain a training set, the training set including a valence dimension score and an arousal dimension score corresponding to each feature information; based on the training set, training a pre-constructed initial model to obtain an emotion recognition model.

[0059] In one embodiment, the source of the sample data set can be an open source database based on four signals, such as the DEAP library of brain-computer interface, the AffectNet library of facial expressions, the Chinese Eye-Movement Database, the Chinese speech database AIDATATANG_1505ZH, etc. For the convenience of data processing, the comprehensive database MAHNOB-HCI can be used. The database is displayed to 30 subjects by clips of movies and pictures, and is monitored by 6 cameras, head-mounted microphones, eye trackers, and physiological sensors that measure ECG (electrocardiogram), EEG (32-channel electroencephalogram), breathing amplitude and skin temperature. Therefore, the database contains the required physiological data such as expressions, sounds, eye movements, and EEG.

[0060] As another embodiment, obtaining a sample data set may include: collecting multimodal information of N drivers during driving to obtain N groups of multimodal information, and having multiple evaluators score the valence dimension and arousal dimension of the drivers based on the N groups of multimodal information to obtain a reservation dimension score and an arousal dimension score corresponding to each feature information of each group of multimodal information. For example, the evaluators may be composed of psychologists and drivers.

[0061] In order to obtain a cleaner database, after obtaining the sample data set, the following may be done: data cleaning of the sample data set. Specifically, the noise data in the open source database, such as incomplete images or unclear speech signals, may be cleaned and re-labeled.

[0062] In one embodiment, performing valence dimension scoring and arousal dimension scoring on each feature information in the sample data set may include: performing valence dimension scoring and arousal dimension scoring on each feature information in the sample data set according to the scoring rules of the SAM scale.

[0063] Specifically, each dimension of the SAM scale corresponds to different degrees of emotional states through a series of pictures, such as Figure 3 As shown, the first dimension is the valence dimension, also known as pleasure, which indicates the positivity or negativity of the emotional state, such as positive happiness, satisfaction, interest, excitement, etc., and negative pain, sadness, anger, fear, panic, etc. Too much positive or negative emotions will occupy too much of the brain's attention. For vehicle drivers, a medium level is most appropriate. Figure 3 In a, number 1 means very excited, number 5 means neutral, and number 9 means very sad.

[0064] The second dimension is the arousal dimension, also known as activation, which indicates the degree of physiological and psychological activation. Arousal can help the brain activate physiological functions, mobilize various organs, and respond in time when encountering danger. For example, high arousal emotions: excitement, anxiety, and uneasiness; low arousal emotions: boredom, laziness, and sleepiness. For the specific scenario of vehicle driving, the driver needs to maintain moderate arousal and be able to react normally. Too high or too low arousal is a dangerous driving state, which is prone to overreaction or failure to react in time. Figure 3 In b, the serial number 1 represents "completely controlled", 5 represents "neutral", and 9 represents "completely controlled".

[0065] For example, changes in the cartoon character's eyebrows and mouth represent the valence dimension score, while the vibration degree of the heart position and the eyes represent the arousal dimension score. When labeling, the evaluator only needs to choose the one from the cartoon characters of the corresponding dimension that best matches the current emotional state.

[0066] In other embodiments, each feature information can be annotated with a valence dimension score and annotated with an arousal dimension score based on the personal opinions of the evaluator and the SAM scale. According to the aforementioned scoring and annotation method, a training set can be obtained, that is, the correspondence between the feature information and the valence dimension score, and the correspondence between the feature information and the arousal dimension score.

[0067] In this embodiment, the emotion recognition model may include a brain electronic model, a facial expression sub-model, an eye movement sub-model and a speech sub-model. Based on the training set, a pre-constructed initial model is trained to obtain an emotion recognition model, which may include: training the pre-constructed initial brain electronic model according to each power spectral density and the valence dimension score and the arousal dimension score corresponding to each power spectral density, to obtain a trained brain electronic model; training the pre-constructed initial facial expression sub-model according to each target image data and the valence dimension score and the arousal dimension score corresponding to each target image data, to obtain a trained facial expression sub-model; training the pre-constructed initial eye movement sub-model according to the valence dimension score corresponding to each pupil diameter size and the arousal dimension score corresponding to each pupil diameter change rate, to obtain a trained eye movement sub-model; training the pre-constructed speech sub-model according to each Mel spectrum feature and the valence dimension score and the arousal dimension score corresponding to each Mel spectrum feature, to obtain a trained speech sub-model.

[0068] Optionally, the target image data is a grayscale image with a size of 48×48×1.

[0069] In a specific embodiment, a pre-constructed initial brain electronic model is trained according to each power spectral density and the valence dimension score and the arousal dimension score corresponding to each power spectral density to obtain a trained brain electronic model, which may include: according to each power spectral density and the valence dimension score and the arousal dimension score corresponding to each power spectral density, a tree-like LSTM (long-short term memory) model is trained to obtain a trained brain electronic model.

[0070] like Figure 4 As shown in the figure, the input is the EEG data feature PSD, and a tree-like LSTM model is used, in which the tree structure is composed of 4 levels, different levels represent different learning stages, and leaf nodes are at different levels, so that the shallower and deeper features can be initially fused and sent to the subsequent structure to learn deeper features. The output of the LSTM unit in the tree part is the output of the entire sequence, and the output of the subsequent LSTM unit is the output of the last hidden layer. Finally, the valence dimension score and the arousal dimension score are output through a fully connected layer.

[0071] In a specific embodiment, a pre-constructed initial facial expression sub-model is trained according to each target image data and the valence dimension score and the arousal dimension score corresponding to each target image data to obtain a trained facial expression sub-model, which may include: according to each target image data and the valence dimension score and the arousal dimension score corresponding to each target image data, a multi-task convolutional neural network model is trained to obtain a trained facial expression sub-model.

[0072] like Figure 5 As shown in the figure, the input is the preprocessed 48×48×1 image data, which is passed through a multi-task convolutional neural network model consisting of three convolutional layers and two fully connected layers to output the valence dimension score and the arousal dimension score respectively.

[0073] In a specific embodiment, according to the valence dimension score corresponding to each pupil diameter size, and the arousal dimension score corresponding to each pupil diameter change rate, the pre-constructed initial eye movement sub-model is trained to obtain the trained eye movement sub-model, which may include: according to the valence dimension score corresponding to each pupil diameter size, and the arousal dimension score corresponding to each pupil diameter change rate, an eye movement database is established, and the initial eye movement sub-model is trained based on the eye movement database to obtain the eye movement sub-model. The eye movement database includes multiple pupil diameter intervals corresponding to the valence dimension scores, and multiple pupil size change rate intervals corresponding to the arousal dimension scores.

[0074] Specifically, the size of pupil diameter directly corresponds to the valence dimension, reflecting negative, neutral, and positive emotions. The rate of change of pupil size reflects the arousal dimension. The features are matched with the dimension scores through the eye movement database, and the pupil diameter size and change rate are input into the eye movement sub-model to output the corresponding valence and arousal scores.

[0075] In a specific embodiment, a pre-constructed speech sub-model is trained according to each Mel-spectrogram feature and the valence dimension score and the arousal dimension score corresponding to each Mel-spectrogram feature to obtain a trained speech sub-model, which may include: according to each Mel-spectrogram feature and the valence dimension score and the arousal dimension score corresponding to each Mel-spectrogram feature, a fully convolutional neural network model is trained to obtain a trained speech sub-model.

[0076] like Figure 6 As shown in the figure, a lightweight fully convolutional neural network model is used. The model input is the Mel spectrum feature obtained by preprocessing the audio file. The middle part is mainly composed of a parallel convolution part, a residual structure, and a serial convolution part. The parallel convolution structure contains three parallel convolution layers, and the number of convolution kernels is set to 16, with sizes of 3×3, 13×1, and 1×10 respectively. Their outputs will be merged together and sent to the residual structure. The residual structure contains two convolution layers, each with 64 convolution kernels of size 3×3. The serial convolution structure includes four consecutive convolution layers, with kernel sizes of 3×3, and the number of kernels is 128, 160, 256, and 300 respectively. The final output is set as a multi-task regression model, using two fully connected layers to output the valence dimension score and the arousal dimension score.

[0077] Scoring the multiple feature information in multiple dimensions based on a pre-established emotion recognition model to obtain respective scores for each feature information in multiple dimensions may include: scoring each power spectrum density in the valence dimension and the arousal dimension based on the brain electronic model to obtain respective scores for each power spectrum density in the valence dimension and the arousal dimension; scoring each target image data in the valence dimension and the arousal dimension based on the facial expression sub-model to obtain respective scores for each target image data in the valence dimension and the arousal dimension; scoring each pupil diameter size in the valence dimension and scoring each pupil diameter change in the arousal dimension based on the eye movement sub-model to obtain scores for each pupil diameter size in the valence dimension and scores for each pupil diameter change rate in the arousal dimension; scoring each Mel spectrum feature in the valence dimension and the arousal dimension based on the speech sub-model to obtain respective scores for each Mel spectrum feature in the valence dimension and the arousal dimension.

[0078] Step S104, fusing the scores of each feature information in the valence dimension to obtain a fused valence dimension score, and fusing the scores of each feature information in the arousal dimension to obtain a fused arousal dimension score.

[0079] In a specific embodiment, the scores of each feature information under the valence dimension are fused to obtain a fused valence dimension score, and the scores of each feature information under the arousal dimension are fused to obtain a fused arousal dimension score, which may include: fusing the scores of each feature information under the valence dimension and a preset first weight set to obtain a fused valence dimension score; fusing the scores of each feature information under the arousal dimension and a preset second weight set to obtain a fused arousal dimension score.

[0080] like Figure 7 As shown in the figure, the weight of each modality is determined by judging the credibility of the modality. The higher the credibility, the higher the weight. The specific method includes: taking the search for the optimal weight of each modality awakening score as an example, assuming that there are n modalities corresponding to n regression models, and a total of T trials are used for prediction, the predicted awakening score A of the kth regression model in the tth trial tk , k{1,2,3,…,n}, t{1,2,3,…,T}, assuming a step size of 0.01, set the weight set ω{0.00,0.01,…,0.99,1.00}, and the initial variance RS min =10, the method for determining the weight of the awakening score mainly includes:

[0081] Step 1: Loop through the weights of n modes in ω, and set the weight of the kth mode to be ω k , when the sum of all weights is 1, go to step 2;

[0082] Step 2: Calculate the predicted arousal score obtained by multimodal fusion under the current weight distribution. Assume that the predicted arousal score of experiment t is y t ′, the calculation formula can be expressed as:

[0083]

[0084] Step 3: Calculate the root mean square error RMSE of T trials under the current weight distribution, denoted as RS cut , R.S. cut The calculation formula is as follows, where y t is the actual arousal score of trial t. By comparing RS cut and RS min The size relationship is used to determine whether the current weight distribution has better performance. cut <RS min, it is believed that the current weight distribution has better performance, so RS min Update to RS cut , save the current weight distribution. cut ≥RS min , it is considered that the current weight distribution does not show better performance, and there is no need to adjust RS min Regardless of the size relationship, the steps must be executed again to enumerate the next set of weight distributions until the loop enumeration ends and the optimal weight distribution is saved;

[0085]

[0086] The process ends and the optimal weight set ω is finally output last . The optimal weight set ω last As the second weight set, the same weight determination method as the arousal dimension is used to obtain the optimal weight set of the valence dimension as the first weight set. Through weight calculation, the fused arousal score and the fused valence score are obtained to mark the final emotion level.

[0087] Therefore, after obtaining the recognition results of multiple modalities, the results are weighted and fused to obtain more comprehensive results.

[0088] Step S105 , identifying the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score, and performing auxiliary control on the vehicle based on the emotion level.

[0089] In this embodiment, based on the fused valence dimension score and the fused arousal dimension score, identifying the driver's emotional level can include: taking the middle value of the valence dimension score and the arousal dimension score as the coordinate origin, the horizontal axis represents the valence dimension score, the valence dimension score increases from the negative direction to the positive direction of the coordinate, and the vertical axis represents the arousal dimension score, and the arousal dimension score increases from the negative direction to the positive direction of the coordinate; according to the first coordinate point corresponding to the minimum value of the valence dimension score and the minimum value of the arousal dimension score, the second coordinate point corresponding to the minimum value of the valence dimension score and the maximum value of the arousal dimension score, and the vertical axis corresponding to the maximum value of the valence dimension score and the arousal dimension score. The third coordinate point corresponding to the minimum value of the wakefulness dimension score, the fourth coordinate point corresponding to the maximum value of the valence dimension score and the maximum value of the wakefulness dimension score, the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point are connected to construct an emotional area containing the coordinate origin; the emotional area is divided into multiple sub-areas, each single sub-area corresponds to an emotional level of the driver, wherein each sub-area contains four quadrants of the coordinate, and the negative direction score and the positive direction score of each sub-area are symmetrical based on the origin; based on the area where the fused valence dimension score is located and the area where the fused wakefulness dimension score is located, the driver's emotional level is identified. That is, Fig. 9 shown.

[0090] Optionally, the emotion area can be divided into three sub-areas, including a first sub-area, a second sub-area and a third sub-area, the first sub-area corresponds to a third-level emotion level, the second sub-area corresponds to a second-level emotion level, and the third sub-area corresponds to a first-level emotion level. Of course, as other embodiments, the emotion area can also be divided into four areas or five areas, corresponding to four emotion levels and five emotion levels.

[0091] Specifically, if the fused valence dimension score is in the first scoring range or in the second scoring range, or the fused arousal dimension score is in the first scoring range or in the second scoring range, the driver's emotional level is identified as level three, the first scoring range and the second scoring range are symmetrical based on the origin, and the maximum value in the second scoring range is the maximum score.

[0092] If the fused valence dimension score is in the third scoring range or in the fourth scoring range, and the fused arousal dimension score is in the third scoring range or in the fourth scoring range, then the driver's emotion level is identified as level 2, the third scoring range and the fourth scoring range are symmetrical based on the origin, and the maximum value in the fourth scoring range is less than the minimum value in the second scoring range, and the minimum value in the third scoring range is greater than the maximum value in the first scoring range;

[0093] If the fused valence dimension score and the fused arousal dimension score are both in the fifth scoring range, it is identified that the driver's emotion level is level one, and the fifth scoring range includes the origin position.

[0094] The first scoring range and the second scoring range are located in the first sub-area of ​​the three sub-areas, the third scoring range and the fourth scoring range are located in the second sub-area of ​​the three sub-areas, and the fifth scoring range is located in the third sub-area of ​​the three sub-areas.

[0095] Valence and arousal can explain most types of emotions, so this application proposes a valence-arousal two-dimensional emotion classification model for the field of vehicle driving. Figure 8 As shown, by monitoring the driver's emotions when driving a vehicle, especially the stability of emotions, both the valence dimension and the arousal dimension are from one extreme to another. Therefore, the valence-arousal model proposed in this application is divided into regions. Taking the score of 1 to 9 as an example, it can be considered that the emotional crisis level corresponding to 1 point (extreme happiness) and 9 points (extreme sadness) is the same, such as 1 point and 9 points are both the most dangerous driving emotions. According to this standard, the evaluator scores the driver's characteristic information in the valence dimension and the arousal dimension.

[0096] like Figure 8 As shown, taking the example that the emotional area contains three sub-areas, the entire score is divided into three rectangular areas in total. In this application, the first score range can be [1, 2.5), the second score range can be [7.5, 9), the third score range can be [2.5, 4), the fourth score range can be [6, 7.5), and the fifth score range can be [4, 6). Therefore, the emotional levels are {1≤score<2.5, 7.5≤score<9|level 3}, {2.5≤score<4, 6≤score<7.5|level 2}, {4≤score<6|level 1}, and the data in the database are labeled according to this standard. In the specific labeling process, if the original data has valence and arousal scores, it will be re-labeled according to the area corresponding to the score. If there is no relevant score, the valence and arousal scores are first performed according to the SAM scale scoring principle, and then the emotional level labels are divided.

[0097] For example, if the fused valence dimension score is 8 points and the fused arousal dimension score is 6 points, indicating that the valence dimension score is level three and the arousal dimension score is level two, then the driver's emotional level is level three; if the fused valence dimension score is 1 point and the fused arousal dimension score is 3 points, indicating that the valence dimension score is level three and the arousal dimension score is level two, then the driver's emotional level is level two; if the fused valence dimension score is 4 points and the fused arousal dimension score is 5 points, indicating that the valence dimension score is level one and the arousal dimension score is level one, then the driver's emotional level is level one.

[0098] Next, auxiliary control of the vehicle is performed based on the emotion level, which may include: if the driver's emotion level is level one, no auxiliary control of the vehicle is performed; if the driver's emotion level is level two, the environmental atmosphere inside the vehicle is adjusted; if the driver's emotion level is level three, the vehicle is intelligently driven and parked nearby when the vehicle reaches a safe parking area.

[0099] In a specific embodiment, in order to avoid the influence of emotions on driving safety, a corresponding control strategy can be provided according to the detected emotion level, such as Fig. 9 As shown in the figure, when the emotional level is level one, calm, no regulation is performed; when the emotional level is level two, fluctuating, environmental atmosphere regulation is performed, including but not limited to in-car ambient lights, in-car music, air conditioning, ARHUD, seat massage, primary driving assistance, etc.; in-car ambient light regulation refers to the lighting regulation of vehicles with ambient lights, where yellow light and blue light have the effect of soothing emotions, and the ambient lights are adjusted to the yellow-blue series. In-car music regulation refers to the adjustment of the car music, and the vehicle presets light music for calming emotions in advance, which is automatically played when the emotional level is level two.

[0100] Specifically, air conditioning adjustment refers to adjusting the temperature inside the car. Generally, 20-22℃ makes people feel good and comfortable. When the emotional level is level two, first adjust the air conditioner to 20℃. After multiple feedbacks are ineffective, gradually lower the temperature to 15℃. Lower temperatures can calm emotions.

[0101] AR-HUD (augmented reality head-up display) is a common feature on smart vehicles. By adding some animations within the field of vision, it draws the driver's attention more to road conditions, reduces distracted thinking, and thus calms emotions.

[0102] Seat massage refers to turning on the driver's seat massage function. Massage can please the body and mind, relax the driver and achieve the purpose of soothing emotions.

[0103] Primary driving assistance refers to L1 and L2 intelligent driving. When the emotion level is level two, it mainly plays an auxiliary role to prevent emotions from distracting attention, thereby reducing driving safety performance.

[0104] When the emotional level reaches level three, the driver is in a dangerous driving state and the L3 intelligent driving system will semi-take over the vehicle. When the vehicle reaches a parking section, it will pull over to the nearest side of the road to give the driver a period of emotional calm.

[0105] By taking graded adjustment measures, the first-level atmosphere adjustment is to adjust the environment through atmosphere lights, music, AR HUD, air conditioning and other means to achieve the purpose of soothing emotions; the second-level assisted driving intervention, when the detected emotions are intensified to the critical value, on the basis of the first-level atmosphere adjustment, the intelligent driving actively intervenes to ensure safe driving.

[0106] Therefore, the multimodal driver emotion recognition method provided by this application is as follows: Fig.10 As shown, it mainly includes: multimodal information collection, multimodal information processing, introducing emotion recognition model for emotion recognition, emotion classification and vehicle control according to the level.

[0107] In summary, a multimodal driver emotion recognition method is provided by an embodiment of the present invention. The method is applied to multimodal input recognition during emotion recognition during vehicle driving. By combining multimodal means such as EEG, images (expressions, eye movements), and sounds, the problem of inaccurate driver emotion recognition by single visual, sound, and other emotion recognition schemes is solved. The driver's emotional state can be collected more comprehensively, and emotions can be identified more accurately from the perspective of the impact of the valence dimension and arousal dimension of emotions on driving during driving. Multimodal regulation means are used according to the emotion level to make emotion recognition more accurate and regulation faster.

[0108] In the second aspect, based on the same inventive concept, this embodiment provides a multi-modal driver emotion recognition device, such as Fig.11 As shown, including:

[0109] The acquisition module 401 is used to collect the multi-modal information of the driver after the vehicle is started;

[0110] The acquisition module 402 is used to extract features from each modal information in the multi-modal information to obtain multiple feature information;

[0111] An acquisition module 403 is used to score the multiple feature information in multiple dimensions based on a pre-established emotion recognition model to obtain scores of each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between the feature information and the valence dimension score, and a correspondence between the feature information and the arousal dimension score;

[0112] The acquisition module 404 is used to fuse the scores of each feature information in the valence dimension to obtain a fused valence dimension score, and to fuse the scores of each feature information in the arousal dimension to obtain a fused arousal dimension score;

[0113] The acquisition module 405 is used to identify the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score, and perform auxiliary control on the vehicle based on the emotion level.

[0114] As an optional embodiment, the establishment of the emotion recognition model includes: obtaining a sample data set, the sample data set including multiple feature information extracted from multiple groups of multimodal information; annotating each feature information in the sample data set with a valence dimension score and annotating the arousal dimension score to obtain a training set, the training set including a valence dimension score and an arousal dimension score corresponding to each feature information; based on the training set, training a pre-constructed initial model to obtain an emotion recognition model.

[0115] As an optional embodiment, each feature information in the sample data set is annotated with a valence dimension score and annotated with an arousal dimension score, including: annotating each feature information in the sample data set with a valence dimension score and annotating with an arousal dimension score according to the scoring rules of the SAM scale.

[0116] As an optional embodiment, the multimodal information includes EEG signals, facial data, eye movement data and voice data, and the feature extraction module is specifically used to: perform a first feature extraction on the EEG signal to obtain a power spectral density; perform a second feature extraction on the facial data to obtain target image data; perform a third feature extraction on the eye movement data to obtain the pupil diameter size and the pupil diameter change rate; perform a fourth feature extraction on the voice data to obtain a Mel spectrum feature.

[0117] As an optional embodiment, the emotion recognition model includes a brain electronic model, a facial expression sub-model, an eye movement sub-model and a speech sub-model. The method of training a pre-constructed initial model based on a training set to obtain an emotion recognition model includes: training the pre-constructed initial brain electronic model according to each power spectrum density and the valence dimension score and the arousal dimension score corresponding to each power spectrum density to obtain a trained brain electronic model; training the pre-constructed initial facial expression sub-model according to each target image data and the valence dimension score and the arousal dimension score corresponding to each target image data to obtain a trained facial expression sub-model; training the pre-constructed initial eye movement sub-model according to the valence dimension score corresponding to each pupil diameter size and the arousal dimension score corresponding to each pupil diameter change rate to obtain a trained eye movement sub-model; training the pre-constructed speech sub-model according to each Mel spectrum feature and the valence dimension score and the arousal dimension score corresponding to each Mel spectrum feature to obtain a trained speech sub-model.

[0118] As an optional embodiment, the emotion recognition module is specifically used to: take the middle value of the valence dimension score and the arousal dimension score as the coordinate origin, the horizontal axis represents the valence dimension score, the valence dimension score increases from the negative direction to the positive direction of the coordinate, and the vertical axis represents the arousal dimension score, and the arousal dimension score increases from the negative direction to the positive direction of the coordinate; according to the first coordinate point corresponding to the minimum value of the valence dimension score and the minimum value of the arousal dimension score, the second coordinate point corresponding to the minimum value of the valence dimension score and the maximum value of the arousal dimension score, the third coordinate point corresponding to the maximum value of the valence dimension score and the minimum value of the arousal dimension score, and the fourth coordinate point corresponding to the maximum value of the valence dimension score and the maximum value of the arousal dimension score, the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point are connected to construct an emotion area; divide the emotion area into multiple sub-areas, each single sub-area corresponds to an emotion level of the driver, wherein each sub-area includes four quadrants of the coordinate, and the negative direction score and the positive direction score of each sub-area are symmetrical based on the origin; based on the area where the fused valence dimension score is located and the area where the fused arousal dimension score is located, identify the driver's emotion level.

[0119] As an optional embodiment, the fusion module is specifically used to: fuse the scores of each feature information in the valence dimension and a preset first weight set to obtain a fused valence dimension score; fuse the scores of each feature information in the arousal dimension and a preset second weight set to obtain a fused arousal dimension score.

[0120] The above modules can be implemented by software codes, in which case, the above modules can be stored in the memory of the control device. The above modules can also be implemented by hardware such as integrated circuit chips.

[0121] An embodiment of the present invention provides a multimodal driver emotion recognition device, whose implementation principle and technical effects are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0122] In a third aspect, based on the same inventive concept, this embodiment provides an electronic device 500, such as Fig.12 As shown, it includes: a memory 501, a processor 502 and a computer program 503 stored in the memory and executable on the processor. When the processor 502 executes the program, the steps of the emotion recognition method described in the first aspect are implemented.

[0123] Since the electronic device introduced in this embodiment is an electronic device used to implement the emotion recognition method in the embodiment of this application, based on the emotion recognition method introduced in the embodiment of this application, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as a person skilled in the art implements the electronic device used in the emotion recognition method in the embodiment of this application, it belongs to the scope of protection of this application.

[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A module that specifies functions in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction module, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal driver emotion recognition method, characterized in that: include: After the vehicle is started, the driver's multimodal information is collected; Extracting features from each modal information in the multimodal information to obtain multiple feature information; Scoring the multiple feature information in multiple dimensions based on a pre-established emotion recognition model to obtain scores for each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between the feature information and the valence dimension score, and a correspondence between the feature information and the arousal dimension score; The scores of each feature information under the valence dimension are integrated to obtain a integrated valence dimension score, and the scores of each feature information under the arousal dimension are integrated to obtain a integrated arousal dimension score; Based on the fused valence dimension score and the fused arousal dimension score, the driver's emotion level is identified, and auxiliary control of the vehicle is performed based on the emotion level.

2. The method according to claim 1, characterized in that The establishment of the emotion recognition model includes: Acquire a sample data set, wherein the sample data set includes a plurality of feature information extracted from a plurality of sets of multimodal information; Performing valence dimension scoring and arousal dimension scoring annotation on each feature information in the sample data set to obtain a training set, wherein the training set includes a valence dimension score and an arousal dimension score corresponding to each feature information; Based on the training set, the pre-built initial model is trained to obtain the emotion recognition model.

3. The method according to claim 2, characterized in that The step of labeling each feature information in the sample data set with a valence dimension score and an arousal dimension score includes: According to the scoring rules of the SAM scale, each feature information in the sample data set is annotated with valence dimension scores and arousal dimension scores.

4. The method according to claim 2, characterized in that The multimodal information includes electroencephalogram signals, facial data, eye movement data and voice data. The feature extraction is performed on each modal information in the multimodal information to obtain multiple feature information, including: Performing a first feature extraction on the EEG signal to obtain a power spectral density; Performing a second feature extraction on the facial data to obtain target image data; Performing a third feature extraction on the eye movement data to obtain the pupil diameter size and the pupil diameter change rate; A fourth feature extraction is performed on the speech data to obtain a Mel spectrum feature.

5. The method according to claim 4, characterized in that The emotion recognition model includes a brain electronic model, a facial expression sub-model, an eye movement sub-model and a speech sub-model. The emotion recognition model is obtained by training a pre-built initial model based on the training set, including: According to each power spectral density and the valence dimension score and the arousal dimension score respectively corresponding to each power spectral density, the pre-constructed initial brain electronic model is trained to obtain a trained brain electronic model; According to each target image data and the valence dimension score and the arousal dimension score respectively corresponding to each target image data, the pre-constructed initial facial expression sub-model is trained to obtain a trained facial expression sub-model; According to the valence dimension score corresponding to each pupil diameter size and the arousal dimension score corresponding to each pupil diameter change rate, the pre-constructed initial eye movement sub-model is trained to obtain a trained eye movement sub-model; According to each Mel-spectrogram feature and the valence dimension score and the arousal dimension score respectively corresponding to each Mel-spectrogram feature, the pre-constructed speech sub-model is trained to obtain a trained speech sub-model.

6. The method according to claim 1, characterized in that The step of identifying the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score includes: The middle value of the valence dimension score and the arousal dimension score is taken as the origin of the coordinate. The horizontal axis represents the valence dimension score, and the valence dimension score increases from the negative direction to the positive direction of the coordinate. The vertical axis represents the arousal dimension score, and the arousal dimension score increases from the negative direction to the positive direction of the coordinate. According to the first coordinate point corresponding to the minimum value of the valence dimension score and the minimum value of the arousal dimension score, the second coordinate point corresponding to the minimum value of the valence dimension score and the maximum value of the arousal dimension score, the third coordinate point corresponding to the maximum value of the valence dimension score and the minimum value of the arousal dimension score, and the fourth coordinate point corresponding to the maximum value of the valence dimension score and the maximum value of the arousal dimension score, the first coordinate point, the second coordinate point, the third coordinate point and the fourth coordinate point are connected to construct an emotion area; Dividing the emotion region into a plurality of sub-regions, each sub-region corresponding to an emotion level of the driver, wherein each sub-region includes four quadrants of the coordinates, and the negative direction score and the positive direction score of each sub-region are symmetrical based on the origin; Based on the region where the fused valence dimension score is located and the region where the fused arousal dimension score is located, the emotion level of the driver is identified.

7. The method according to claim 1, characterized in that The step of fusing the scores of each feature information in the valence dimension to obtain a fused valence dimension score, and fusing the scores of each feature information in the arousal dimension to obtain a fused arousal dimension score, includes: According to the score of each feature information in the valence dimension and the preset first weight set, a fused valence dimension score is obtained; The scores of each feature information in the awakening dimension and the preset second weight set are fused to obtain a fused awakening dimension score.

8. A multi-modal driver emotion recognition device, characterized in that: include: A collection module, used to collect multi-modal information of the driver after the vehicle is started; A feature extraction module, used to extract features from each modal information in the multimodal information to obtain multiple feature information; A scoring module, configured to score the plurality of feature information in multiple dimensions based on a pre-established emotion recognition model, and obtain scores of each feature information in multiple dimensions, wherein the multiple dimensions include a valence dimension and an arousal dimension, and the emotion recognition model includes a correspondence between feature information and a score in the valence dimension, and a correspondence between feature information and a score in the arousal dimension; A fusion module, used to fuse the scores of each feature information in the valence dimension to obtain a fused valence dimension score, and to fuse the scores of each feature information in the arousal dimension to obtain a fused arousal dimension score; The emotion recognition module is used to identify the driver's emotion level based on the fused valence dimension score and the fused arousal dimension score, and to perform auxiliary control of the vehicle based on the emotion level.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images