A method for intelligent recognition of driver emotion based on a multi-modal network

Abstract

The present application relates to a kind of driver emotion intelligent identification method based on multimodal network, belong to wisdom traffic technical field.The present application relates to a kind of driver emotion intelligent identification method based on multimodal network, belong to wisdom traffic technical field.It includes multimodal emotion data acquisition, data preprocessing, builds emotion recognition network model based on multimodal network, multimodal emotion intervention.The advantage is that the proposed multimodal emotion recognition network model is identified by multiple modal information, compared with single modal emotion recognition, can improve the recognition accuracy to a certain extent;The face emotion recognition model proposed in the present application realizes the parameter decline under the condition that the recognition accuracy is guaranteed, greatly reduces the consumption of computing resources, improves the recognition speed;The driver emotion recognition method proposed in the present application can reduce the probability of traffic accidents caused by the bad mood of driver, and can contribute to the inference of driver's intention in intelligent driving cabin.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a driver emotion intelligent recognition method based on multi-modal. BACKGROUND

[0002] Currently, various over-reactions such as road rage caused by bad emotions of drivers are one of the main reasons to endanger public traffic safety. It is necessary to put the maintenance of public safety in a more prominent position and comprehensively improve the public safety guarantee capability. How to quickly and accurately identify the real-time emotions of drivers is the key to solving such problems.

[0003] Among the numerous studies on driver emotion recognition, according to the different collection methods, it can be roughly divided into four categories: emotion recognition based on voice, emotion recognition based on facial image, emotion recognition based on physiological signals, and emotion recognition based on driving behavior. Among them, the method of emotion recognition based on voice contains the recognition of driver's voice and semantics. However, the change of driver's emotion cannot be timely reflected through voice, so this method can only be used as an auxiliary means and cannot be used as the main monitoring method for emotion recognition of drivers; the physiological signals in the method of emotion recognition based on physiological signals refer to various physiological signals of drivers such as electroencephalogram, electrodermal signal, skin temperature, electrocardiogram, etc. This method has high recognition accuracy, but the main reason why it cannot be widely used is that it is difficult to wear the collection device. Not only will it interfere with the normal driving behavior of the driver, but also it will cause psychological pressure to the driver, so that this method can only stay in the laboratory and cannot be popularized in the real environment; the method of emotion recognition based on driving behavior refers to collecting the interaction between the driver and the key module of the vehicle, such as the interaction force of the steering wheel or foot pedal to determine the emotion.

[0004] The method of emotion recognition based on facial image is the most studied and the most widely used emotion recognition method. This method mainly includes two parts: feature extraction and emotion classification. In traditional image processing, the feature extraction part generally extracts facial image features manually according to statistical rules and prior knowledge, such as classifying happiness as mouth up and classifying surprise as eyes wide open. With the development of image processing technology, the current mainstream method is to use convolutional neural network to extract features of the image. However, due to the complexity and subjectivity of emotions and the variability and complexity of the vehicle environment, the current products cannot quickly and accurately identify the real-time emotions of drivers. SUMMARY

[0005] The application provides a driver emotion intelligent recognition method based on a multi-modal network, which improves the accuracy and speed of driver emotion recognition, improves the accuracy by recognizing multi-modal information, reduces the calculation consumption while ensuring the accuracy, and improves the recognition speed, and is suitable for preventing public safety accidents caused by excessive emotions of drivers on the vehicle side.

[0006] The technical scheme adopted by the application comprises the following steps:

[0007] Step 1: Driver multi-modal emotion data acquisition: including real-time image information, voice information, interaction intensity information and steering wheel speed information of the driver;

[0008] Step 2: Data preprocessing:

[0009] (1) The mean filtering method is used to preprocess the driver image information, so as to reduce the interference of complex light and environment and other factors on the recognition result and improve the image quality;

[0010] (2) The driver voice information is preprocessed by converting the voice signal into a spectrogram;

[0011] (3) The driver behavior information is preprocessed by standardization;

[0012] Step 3: Building an emotion recognition network model based on a multi-modal network:

[0013] The multi-modal emotion data of the driver in a real environment is selected as an emotion data set, the data set is divided into a training set and a test set according to an 8:2 ratio, the multi-modal network is trained, the best weight file of the training effect is selected and saved, the multi-modal network model and the weight file are integrated as an emotion recognition network model and stored in the vehicle side memory, the emotion recognition network model comprises a facial emotion recognition model, a sound emotion recognition model and a driving behavior emotion recognition model, and a decision layer fusion algorithm is used to effectively fuse the emotions of the three modalities;

[0014] Step 4: Multi-modal emotion intervention:

[0015] When the emotion recognition model detects that the driver has an abnormal emotion, a warning signal is sent in time, and the driver is helped to adjust the emotion in a multi-modal manner.

[0016] The emotion in step 1 can be marked as happy, angry, sad, fear, surprise, disgust, calm, and the specific collection method is: through the front portrait camera or infrared camera in the vehicle to shoot the portrait photo of the driver during the driving process, the real-time image information of the driver is obtained; through the microphone in the vehicle to collect the voice information during the driving process; through the force sensor arranged beside the foot pedal to collect the interaction force information of the driver, and through the rotating speed sensor arranged beside the steering wheel to obtain the rotating speed information of the steering wheel.

[0017] The mean filtering algorithm in (1) of step 2 adopts formula 1, wherein N neighbor represents the number of pixels in the current pixel neighborhood table, g(i neighbor ,j neighbor ) refers to the gray value of the pixel point around the pixel point, and the mean filtering refers to using the mean value of the gray value of the surrounding pixel point as the output of the pixel point, that is, g(i,j):

[0018]

[0019] In (2) of step 2, the speech signal needs to be first framed, the corresponding frequency spectrum is calculated through FFT transformation for each frame of speech, and then the frequency spectrum generated by multiple frames of speech is combined in a graph to generate a corresponding spectrogram;

[0020] In (3) of step 2, a standardized method is adopted, as shown in formula 2, wherein a is the original data, mu is the mean value of the current behavior data, sigma is the variance of the current behavior data, and b is the standardized data.

[0021]

[0022] The face emotion recognition model in step 3 comprises, after the preprocessed face picture U is acquired, first inputting the face image U into a spatial attention network STN, acquiring a picture V with visual attention effect, inputting the picture V into an improved convolutional neural network VGG, wherein the network is composed of 5 convolutional blocks, 1 channel attention module ECA-NET and 1 full connection layer FC1, the convolution kernel size in the network is all 3x3, and the pooling kernel size is all 2x2, wherein the convolutional block 1 comprises 2 convolutional layers and 1 maximum pooling layer, and the convolutional layer output channel number is 64; the convolutional block 2 comprises 2 convolutional layers and 1 maximum pooling layer, and the convolutional layer output channel number is 128; the convolutional block 3 comprises 3 convolutional layers and 1 maximum pooling layer, and the convolutional layer output channel number is 256; the convolutional block 4 comprises 3 convolutional layers and 1 maximum pooling layer, and the convolutional layer output channel number is 512; the convolutional block 5 comprises 3 convolutional layers and 1 maximum pooling layer, and the convolutional layer output channel number is 512, the output of the convolutional block 5 is input into the channel attention module ECA-NET, and the output channel number of the module is 512, wherein the ECA-NET module first performs global average pooling on the input feature map, and each channel feature layer is represented by a single value; then, the weight of each feature layer is generated by a one-dimensional convolution with a size of k, the relationship between each channel is obtained, and finally the generated weight is applied to the output feature map, so that each feature layer can be distinguished and the features can be processed more effectively, and finally a full connection layer FC1 is used to obtain the result score S of face emotion recognition face .

[0023] The STN network comprises 3 modules, namely a local network, a grid generator and a sampler, wherein the local network is composed of a small convolutional neural network, and the structure of the network is: an input convolutional layer conv1, a maximum pooling layer Pool1, an input convolutional layer conv2, a maximum pooling layer Pool2, a full connection layer FC1 and a full connection layer FC2, wherein the convolution kernel size of conv1 is 7x7, the convolution kernel size of conv2 is 5x5, the pooling kernel size is 2x2, the output dimension of the full connection layer FC1 is 32, and the output dimension of the full connection layer FC2 is 6, corresponding to 6 parameters θ 11 , θ 12 , θ 13 , θ 21 , θ 22 , θ 23 , i.e. A θ , the grid generator is used to obtain the parameter mapping relationship T between the input picture U and the output picture V θ , as shown in formula 4, wherein is the coordinates of each pixel in the input picture U, is the coordinates of each pixel in the output picture V, and the samplerθ acting on each pixel in the input picture U, output picture V;

[0024]

[0025]

[0026] The convolution layer of the network adopts a "cheap convolution" mode, which can reduce the model size, greatly reduce the calculation amount and improve the calculation speed on the premise of ensuring the accuracy, and the implementation mode of the "cheap convolution" is:

[0027] 1) first use the ordinary conventional convolution to compress the input feature dimension c to m;

[0028] 2) use layer-by-layer convolution to expand the m-dimensional feature to m*s=n, where s is the operation number of the cheap convolution, and the layer-by-layer convolution is the ghost feature;

[0029] 3) the features obtained by the conventional convolution and the ghost convolution are spliced as the output to obtain the final "cheap convolution" result.

[0030] The voice emotion recognition model in step 3 is analyzed and recognized by constructing an 8-layer convolutional neural network on the preprocessed spectrogram, wherein the model comprises 4 convolutional layers, 2 maximum pooling layers and 2 fully connected layers, the size of the convolution kernel is 3x3, the size of the pooling kernel is 2x2, the output dimension of the convolution layer 1 and the convolution layer 2 is 32, the output dimension of the convolution layer 3 and the convolution layer 4 is 64, and the output dimension of the two fully connected layers is 128 and 7 respectively, wherein the four convolutional layers of the model are realized by "cheap convolution", the input of the model is the spectrogram generated after preprocessing, and the output is the score S of voice emotion recognition voice .

[0031] The driving behavior emotion recognition model in step 3 is analyzed and recognized by establishing a three-layer perception machine model. The input of the model is the pedal force, steering wheel grip force and speed information spliced after preprocessing, and the output is the score S of driving behavior emotion behavior .

[0032] In step 3, the decision level fusion selects a weighted summation method to comprehensively judge the information of multiple modalities, through formula 5, wherein S decision is the comprehensive score of multi-modal emotion, k i is the weight of each mode, and ∑k i =1, i=1, 2, 3, in the training process, S decision is set to 0 or 1 according to the data set, so as to obtain the value of k i .

[0033] S decision =k1S face +k2S voice +k3S behavior (5)。

[0034] The visual adjustment in step 4 adjusts the mood of the driver by adjusting the background color of the central control screen and the page layout, the auditory adjustment adjusts the mood of the driver by using adaptive music and auditory intervention, and the tactile adjustment helps the driver adjust the mood by using vibration or temperature change.

[0035] The advantages of the present application are that the proposed multi-modal emotion recognition network model comprehensively recognizes and judges multiple modal information, which can improve the recognition accuracy to a certain extent compared with single-modal emotion recognition, the face emotion recognition model proposed in the present application realizes parameter reduction under the condition of guaranteed recognition accuracy, greatly reduces the consumption of computing resources, and improves the recognition speed, and the driver emotion recognition method proposed in the present application can reduce the probability of traffic accidents caused by bad emotions of the driver, and can contribute to inferring the intention of the driver in the intelligent driving cabin. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a schematic diagram of the overall system of the present application;

[0037] Figure 2 is a schematic diagram of the overall emotion recognition network model of the present application;

[0038] Figure 3 is a schematic diagram of the face emotion recognition model of the present application;

[0039] Figure 4 is a schematic diagram of the STN network structure of the present application;

[0040] Figure 5 is a schematic diagram of the sound emotion recognition model of the present application. DETAILED DESCRIPTION

[0041] The specific embodiments of the present application will be described below with reference to the accompanying drawings, although the drawings show exemplary embodiments of the present application, those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application, and are not intended to limit the protection scope of the present application.

[0042] As shown in the drawings, the following steps are included: Figure 1

[0043] Step 1: Driver's multi-modal emotion data acquisition:

[0044] ​The driver's real-time image information is obtained by taking a portrait photo of the driver during the driving of the vehicle through an in-vehicle front portrait camera or an infrared camera; the voice information during the driving of the vehicle is collected through an in-vehicle microphone; the interactive force information of the driver is collected through a force sensor placed beside the foot pedal and the steering wheel, and the steering wheel speed information is obtained through a speed sensor;

[0045] Wherein the emotion can be marked as seven basic emotions of happy, angry, sad, fear, surprise, disgust and calm;

[0046] Step 2: data preprocessing:

[0047] (1) The driver image information is preprocessed by using mean filtering and the like to reduce the interference of complex light and environment and the like on the recognition result and improve the image quality;

[0048] Wherein, the mean filtering algorithm formula is shown as formula 1, wherein N neighbor represents the number of pixels in the current pixel neighborhood table, g(i neighbor ,j neighbor ) refers to the gray value of the pixel point around the pixel point, and the mean filtering refers to using the mean value of the gray values of the surrounding pixel points as the output of the pixel point, that is, g(i,j):

[0049]

[0050] (2) The driver's voice information is preprocessed by converting the voice signal into a spectrogram, which first needs to frame the voice signal, calculate the corresponding frequency spectrum for each frame of voice through FFT transformation, and then combine the frequency spectrum generated by multiple frames of voice in a graph to generate a corresponding spectrogram;

[0051] (3) The driving behavior information is preprocessed by using standardization, as shown in formula 2, wherein a is the original data, μ is the mean value of the current behavior data, σ is the variance of the current behavior data, and b is the standardized data.

[0052]

[0053] Step 3: build an emotion recognition network model based on a multi-modal network:

[0054] The multi-modal emotion data of the driver in the real environment is selected as the emotion data set, the data set is divided into a training set and a test set according to a ratio of 8:2, the multi-modal network is trained, the best weight file of the training effect is selected and saved, the multi-modal network and the weight file are integrated as an emotion recognition network model and stored in a vehicle end storage, the emotion recognition network model includes a facial emotion recognition model, a sound emotion recognition model and a driving behavior emotion recognition model, a decision layer fusion algorithm is used to effectively fuse the emotions of the three modalities, and a schematic diagram of the overall structure of the emotion recognition network model is as shown in Figure 2 .

[0055] The structure of the facial emotion recognition model is as shown in Figure 3 When the preprocessed facial picture U is acquired, the facial image U is first input into the spatial attention network STN to acquire the picture V with visual attention effect, the network can detect the most relevant part in a picture for the recognition task and suppress useless information, and the network also has the function of “face alignment”, which can reduce the recognition rate reduction problem caused by the face not being in the center of the image due to camera shaking during vehicle driving.

[0056] The network structure of the STN is as shown in Figure 4 The STN network structure includes three modules, namely a local network, a grid generator and a sampler, wherein the local network is composed of a small convolutional neural network, the structure of the network is: an input convolutional layer conv1, a maximum pooling layer Pool1, an input convolutional layer conv2, a maximum pooling layer Pool2, a full connection layer FC1 and a full connection layer FC2, wherein the convolution kernel size of conv1 is 7x7, the convolution kernel size of conv2 is 5x5, the pooling kernel size is 2x2, the dimension of the output of the full connection layer FC1 is 32, and the dimension of the output of the full connection layer FC2 is 6, corresponding to the 6 parameters θ 11 , θ 12 , θ 13 , θ 21 , θ 22 , θ 23 , i.e. A θ The grid generator is used to obtain the parameter mapping relationship T θ between the input picture U and the output picture V, as shown in formula 3, wherein is the coordinate of each pixel in the input picture U, is the coordinate of each pixel in the output picture V, and the sampler applies the mapping relationship T θ to each pixel in the input picture U to output the output picture V.

[0057]

[0058]

[0059] The picture V is input into a modified convolutional neural network VGG, wherein the network is composed of 5 convolutional blocks, 1 channel attention module ECA-NET and 1 fully connected layer FC1, the convolution kernel size in the network is all 3x3, and the pooling kernel size is all 2x2, wherein the convolutional block 1 contains 2 convolutional layers and 1 maximum pooling layer, and the output channel number of the convolutional layer is 64; the convolutional block 2 contains 2 convolutional layers and 1 maximum pooling layer, and the output channel number of the convolutional layer is 128; the convolutional block 3 contains 3 convolutional layers and 1 maximum pooling layer, and the output channel number of the convolutional layer is 256; the convolutional block 4 contains 3 convolutional layers and 1 maximum pooling layer, and the output channel number of the convolutional layer is 512; the convolutional block 5 contains 3 convolutional layers and 1 maximum pooling layer, and the output channel number of the convolutional layer is 512, the output of the convolutional block 5 is input into the channel attention module ECA-Net, the output channel number of the module is 512, and finally a fully connected layer FC1 is used to obtain the result score S of the facial emotion recognition face ;

[0060] The ECA-NET module first performs global average pooling on the input feature map, and each channel feature layer is represented by a single value; then, the weight of each feature layer is generated by a one-dimensional convolution of size k to obtain the relationship between each channel; finally, the generated weight is applied to the output feature map, so that each feature layer can be distinguished and the features can be processed more effectively.

[0061] The convolutional layer of the network is realized in the form of “cheap convolution”, which can reduce the model size, greatly reduce the calculation amount and improve the calculation speed on the premise of ensuring the accuracy.

[0062] The implementation method of “cheap convolution” is as follows:

[0063] 1) First, use a normal regular convolution to compress the input feature dimension c to m;

[0064] 2) Expand the m-dimensional feature to m*s=n by layer-by-layer convolution, where s is the operation number of the cheap convolution, which is set to 2 in this embodiment, and the layer-by-layer convolution obtained is the Ghost feature;

[0065] 3) The features obtained by the regular convolution and the Ghost convolution are spliced as the output to obtain the final result of “cheap convolution”.

[0066] The speech emotion recognition model is obtained by constructing an 8-layer convolutional neural network to analyze and recognize the preprocessed spectrogram, and the structure of the network is as follows Figure 5As shown in the figure, the model includes 4 convolutional layers, 2 max-pooling layers, and 2 fully connected layers, the size of the convolution kernel is 3x3, the size of the pooling kernel is 2x2, the output dimension of convolutional layer 1 and convolutional layer 2 is 32, the output dimension of convolutional layer 3 and convolutional layer 4 is 64, and the output dimension of the two fully connected layers is 128 and 7 respectively, wherein the four convolutional layers of the model are realized in the form of "cheap convolution", the input of the model is the preprocessed spectrogram, and the output is the score S of speech emotion recognition voice ;

[0067] The driving behavior emotion recognition model is analyzed and recognized by establishing a three-layer perception machine model. The input of the model is the pedal force, steering wheel grip force and steering speed information after preprocessing and splicing, and the output is the score S of driving behavior emotion behavior ;

[0068] The decision-level fusion selects the weighted summation method to comprehensively judge the information of multiple modalities, through formula 5, wherein S decision is the comprehensive score of multi-modal emotion, k i is the weight of each modality, ∑k i =1, i=1, 2, 3, during the training process, S decision is set to 0 or 1 according to the data set, so as to obtain the value of k i ;

[0069] S decision =k1S face +k2S voice +k3S behavior (5)

[0070] Step 4: multi-modal emotion intervention:

[0071] When the emotion recognition model detects that the driver has an abnormal emotion, a warning signal is sent in time, and the driver is helped to adjust the emotion in a multi-modal way. In this example, the abnormal emotion is defined as anger, sadness, fear and surprise. Among them, the visual adjustment adjusts the driver's emotion through the background color and page layout adjustment of the central control screen, the auditory adjustment adjusts the driver's emotion through adaptive music and auditory intervention, and the tactile adjustment helps the driver adjust the emotion through vibration or temperature change.

Claims

1. A method for intelligent driver emotion recognition based on multimodal networks, characterized in that, Includes the following steps: Step 1: Driver's multimodal emotion data collection: including real-time image information, voice information, interaction force information, and steering wheel speed information of the driver; Step 2: Data Preprocessing (1) The driver image information is preprocessed by mean filtering to reduce the interference of complex lighting and environmental factors on the recognition results and improve the image quality; (2) The driver's speech information is preprocessed by converting the speech signal into a spectrogram; (3) Use standardized methods to preprocess driving behavior information; Step 3: Build an emotion recognition network model based on multimodal networks: Multimodal emotion data of drivers in real-world environments is selected as the emotion dataset. The dataset is divided into training and testing sets in an 8:2 ratio. The multimodal network is trained, and the weight file with the best training effect is selected and saved. The multimodal network model and the weight file are integrated into an emotion recognition network model and stored in the vehicle's memory. The emotion recognition network model includes a facial emotion recognition model, a voice emotion recognition model, and a driving behavior emotion recognition model. A decision-level fusion algorithm is used to effectively fuse the emotions of the three modalities. The facial emotion recognition model includes the following steps: After acquiring a preprocessed facial image U, the facial image U is first input into a spatial attention network (STN) to obtain an image V with visual attention effect. Image V is then input into an improved convolutional neural network (VGG), where the network consists of 5 convolutional blocks, 1 channel attention module (ECA-NET), and 1 fully connected layer (FC1). All convolutional kernels in the network are 3x3 in size, and all pooling kernels are 2x2 in size. Convolutional block 1 contains 2 convolutional layers and 1 max-pooling layer, with 64 output channels; convolutional block 2 contains 2 convolutional layers and 1 max-pooling layer, with 128 output channels; and convolutional block 3 contains 3 convolutional layers and 1 max-pooling layer, with 25 output channels. 6; Convolutional block 4 contains 3 convolutional layers and 1 max-pooling layer, with 512 output channels for each convolutional layer; Convolutional block 5 contains 3 convolutional layers and 1 max-pooling layer, with 512 output channels for each convolutional layer. The output of convolutional block 5 is input into the channel attention module ECA-NET, which has 512 output channels. The ECA-NET module first performs global average pooling on the input feature map, with each channel's feature layer represented by a single numerical value. Then, the weights of each feature layer are generated by a one-dimensional convolution of size k, obtaining the relationship between each channel. Finally, the generated weights are applied to the output feature map, thus distinguishing each feature layer and processing features more effectively. Finally, a fully connected layer FC1 is used to obtain the facial emotion recognition score. ; Step 4: Multimodal Emotion Intervention: When the emotion recognition model detects abnormal emotions in the driver, it issues a timely warning signal and helps the driver regulate their emotions through a multimodal approach.

2. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: In step 1, the emotions can be categorized into seven basic emotions: happiness, anger, sadness, fear, surprise, disgust, and calmness. The specific data collection method is to capture a portrait photo of the driver while the vehicle is in motion using a front-facing human face camera or an infrared camera inside the vehicle to obtain real-time image information of the driver. Voice information is collected during vehicle operation via an in-vehicle microphone; the driver's interaction force information is collected via a force sensor located next to the pedals; and the steering wheel speed information is obtained via a speed sensor located next to the steering wheel.

3. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: The mean filtering algorithm in step 2 (1) adopts formula 1, where This indicates the number of pixels in the current pixel's neighborhood table. Mean filtering refers to using the average grayscale values ​​of the surrounding pixels as the output of a given pixel. : （1）； In step 2 (2), the speech signal first needs to be divided into frames. The spectrum of each frame of speech is calculated by FFT transformation. Then, the spectra generated by multiple frames of speech are combined into a graph to generate the corresponding spectrogram. In step 2(3), the standardization method is as shown in Equation 2, where a is the original data. The mean of the current behavioral data. Let b be the variance of the current behavior data, and b be the standardized data. （2）。 4. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: The STN network comprises three modules: a local network, a mesh generator, and a sampler. The local network consists of a small convolutional neural network with the following structure: input convolutional layer conv1, max pooling layer Pool1, input convolutional layer conv2, max pooling layer Pool2, fully connected layer FC1, and fully connected layer FC2. The kernel size of conv1 is 7x7, the kernel size of conv2 is 5x5, and the pooling kernel size is 2x2. The output dimension of fully connected layer FC1 is 32, and the output dimension of fully connected layer FC2 is 6, corresponding to the 6 parameters required for the affine transformation. ,Right now As shown in Formula 3, the mesh generator is used to obtain the parameter mapping relationship between the input image U and the output image V. As shown in Formula 4, where ( , ) are the coordinates of each pixel in the input image U. , () represents the coordinates of each pixel in the output image V, and the sampler maps the coordinates... Applying a function to each pixel in the input image U, the output image V is generated. （3） （4）。 5. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: The convolutional layers of the network are implemented using a "cost-saving convolution" method. This method can reduce the model size, significantly reduce computational cost, and improve computational speed while maintaining accuracy. The implementation of "cost-saving convolution" is as follows: 1) First, use a regular convolution to compress the input feature dimension c to m; 2) Expand the m-dimensional features using layer-by-layer convolution. , where s is the number of cheap convolution operations, and the Ghost feature is obtained by convolution layer by layer; 3) The features obtained from regular convolution and Ghost convolution are concatenated as the output to obtain the final "cheap convolution" result.

6. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: In step 3, the speech emotion recognition model analyzes and recognizes the preprocessed spectrogram by constructing an 8-layer convolutional neural network. The model includes 4 convolutional layers, 2 max-pooling layers, and 2 fully connected layers. The convolutional kernels are all 3x3, and the pooling kernels are 2x2. The output dimensions of convolutional layers 1 and 2 are 32, and the output dimensions of convolutional layers 3 and 4 are 64. The output dimensions of the two fully connected layers are 128 and 7, respectively. The 4 convolutional layers are implemented using a "cheap convolution" approach. The input to the model is the preprocessed spectrogram, and the output is the speech emotion recognition score. .

7. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: In step 3, the driving behavior emotion recognition model analyzes and identifies emotions by establishing a three-layer perceptron model. The model's input consists of preprocessed and concatenated information on pedal force, steering wheel grip force, and engine speed, and its output is a score for driving behavior emotion. .

8. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: In step 3, the decision-level fusion uses a weighted summation method to comprehensively judge the information from multiple modalities, as shown in formula 5. This refers to the overall score of multimodal sentiment. Weights for each modality, During the training process, To obtain the result, the dataset is set to either 0 or 1. The value; （5）。 9. The method for intelligent driver emotion recognition based on multimodal networks according to claim 1, characterized in that: In step 4, visual adjustment regulates the driver's emotions by adjusting the background color and page layout of the central control screen; auditory adjustment regulates the driver's emotions by using adaptive music and auditory intervention; and tactile adjustment helps the driver regulate their emotions by changing vibrations or temperatures.

Images