A driver emotion recognition and uncertainty evaluation method based on deep learning

Abstract

The present application relates to a kind of driver emotion recognition and uncertainty evaluation method based on deep learning, belong to driver emotion recognition field.Image enhancement module uses adaptive histogram equalization technique, effectively solve the influence brought by uneven illumination in car, improve image quality.Visual feature extraction module uses improved Wide ResNet network, by adding spectral normalization layer and Gaussian process layer, not only can extract rich emotional features, also can perceive the distance relationship between features, enhance the processing capacity of model to complex and uncertain data.Emotion recognition module uses the Bayesian theory system, classifies the probability density distribution obtained by Gaussian process layer, and calculates the Gaussian process posterior through random feature expansion Laplace approximation, realizes efficient emotion recognition.The present application introduces uncertainty evaluation function, effectively predicts the distance between test sample and training sample, provides decision optimization for driver emotion recognition in automatic driving environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of driver emotion recognition, and relates to a driver emotion recognition and uncertainty evaluation method based on deep learning. BACKGROUND

[0002] With the rapid development of modern society and the continuous progress of computer technology, intelligent driving systems have become a research hotspot in the world. Driver emotion recognition, as an important research topic in the field of intelligent driving systems, has attracted attention from the scientific community. At the same time, with the increase in computing power and the decrease in computing cost, computer technology has made significant progress in the past few decades. This development is not only reflected in the rapid iteration of hardware, such as more powerful processors and larger capacity storage devices, but also in the innovation of software and algorithms. In particular, in the field of deep learning, significant breakthroughs have been made in recent years, and deep learning technology has demonstrated outstanding performance in image recognition, natural language processing, and other fields.

[0003] However, the emergence of life stress and mental health problems in modern society has a significant impact on the emotional state of drivers and traffic safety. The rapidly developing social environment and increasing work pressure have led to a large number of mental health problems, such as stress, anxiety, and depression. Emotions are a behavior, hormone, and autonomous response pattern that promotes survival. Poor emotions not only affect an individual's quality of life, but also directly relate to traffic safety and the emotional state of drivers, especially when experiencing strong emotional fluctuations, which can affect their judgment and reaction speed, thereby increasing the risk of traffic accidents. For example, emotional fluctuations can cause drivers to lose focus, react slowly, and even engage in risky driving behavior. Therefore, monitoring and managing the emotional state of drivers has become an important part of ensuring road traffic safety. With the progress of deep learning and artificial intelligence technology, driver emotion recognition methods based on deep neural networks have emerged, providing the possibility of real-time monitoring and evaluation of the emotional state of drivers, opening up new ways to improve traffic safety and optimize the driving experience.

[0004] In the driver emotion recognition method, Yang's high-purity feature separation method achieves effective separation of expression-related features and irrelevant features through two partial feature exchanges and the constraint of reconstruction loss, improving the accuracy of recognition. Wang achieves driver anger emotion recognition and risk assessment of drivers under this emotion, achieving good recognition accuracy. However, as of now, there are relatively few studies combining uncertainty estimation with facial expression recognition. To address the lack of uncertainty evaluation capability for expression states, uncertainty evaluation is applied to complex system reasoning in uncertain environments, such as adding uncertainty evaluation to driver emotion analysis, which will be of great benefit. SUMMARY

[0005] Therefore, the present application aims to provide a driver emotion recognition and uncertainty evaluation method based on deep learning.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] A driver emotion recognition and uncertainty evaluation method based on deep learning, the method comprising the following steps:

[0008] Adaptive histogram equalization processing is performed on the driver image to solve the problem caused by uneven illumination in the vehicle;

[0009] The improved Wide ResNet network is used to extract the emotional features in the driver image, wherein the improved Wide ResNet network comprises:

[0010] At least four convolutional layers, each followed by a spectral normalization SN layer;

[0011] The dropout layer, fully connected layer and softmax layer near the output in the original Wide ResNet network are replaced by a Gaussian process GP layer;

[0012] The probability density distribution obtained by the Gaussian process layer is classified by using the Bayesian theory system, wherein the posterior of the Gaussian process is calculated by using the random feature extended Laplace approximation.

[0013] Further, the adaptive histogram equalization processing comprises:

[0014] The image is divided into blocks of equal size;

[0015] The number of gray levels contained in each sub-block is calculated;

[0016] The number of gray levels contained in each sub-block is calculated;

[0017] The image histogram of the sub-image is calculated;

[0018] The transformation function is calculated to combine each sub-block into the entire image.

[0019] Further, the improved Wide ResNet network comprises:

[0020] The first convolutional layer has 64 3X3 convolutional kernels;

[0021] The second convolutional layer has 128 3X3 convolutional kernels;

[0022] The third convolutional layer has 256 3X3 convolutional kernels;

[0023] The fourth convolutional layer has 512 3X3 convolutional kernels;

[0024] A 2X2 pooling layer maxpool is connected between the convolutional layers, and the moving step is 16;

[0025] A residual module is introduced to ensure the gradient correlation between levels;

[0026] The output feature map is nonlinearly transformed using a PReLU activation function.

[0027] Further, the implementation method of the spectral normalization SN layer is:

[0028] Initialization: select a random initial vector;

[0029] Iterative calculation: matrix multiplication is performed using the initial vector and the parameter matrix; the result vector is normalized; the normalized result vector is multiplied by the parameter matrix; the result vector is normalized;

[0030] After K iterations, the approximate value of the maximum singular value is:

[0031] Where σ l (W) is the lth singular value of the matrix W, indicating the stretching degree of the matrix in a certain direction; W represents a matrix, and K represents the number of iterations; is the transpose of the lth left singular vector u i after the Kth iteration; is the lth right singular vector v l after the Kth iteration;

[0032] Divide the parameters on each parameter matrix by it to achieve the purpose of normalization.

[0033] Further, the implementation method of the Gaussian process GP layer is:

[0034] Training data: there are N training samples Where y i is the label of the ith training sample, and x i is the input feature of the ith training sample;

[0035] Hidden representation: each input x i is converted into a hidden representation h i = h(x i ) through a certain hidden representation function; h(x i ) is the result of mapping the input feature x i to the hidden space;

[0036] Conditional Gaussian process: Based on these hidden representations h i , define a conditional Gaussian process g, whose output at N hidden representation points is a vector g N×1 =[g(h1),...,g(h N )] T ;

[0037] Gaussian process prior with RBF kernel: Gaussian process model assumes g N×1 It follows a multivariate normal distribution with mean zero and covariance matrix σ 2 K, where the elements of the covariance matrix K are calculated using the RBF kernel function:

[0038] RBF kernel function: The RBF kernel function is defined as: K(x,x′)=exp(-γ‖xx′‖ 2 ), where x and x′ are two input vectors, ‖xx′‖ is the Euclidean distance between x and x′; γ is a parameter that controls the width of the kernel function;

[0039] Posterior distribution: Given the observed data Calculate the posterior distribution of g using the Bayesian formula: Among them, p(g) is the prior distribution and p(D|g) is the likelihood function of the observed data.

[0040] Furthermore, the dropout layer, fully connected layer, and softmax layer close to the output in the original Wide ResNet network are replaced with the Gaussian process GP layer, and the loss function formula is: Where N is the number of samples, C is the number of categories, and γ i,j represents the actual probability of labeling the i-th sample as the j-th class, μ i,j represents the average probability of labeling the i-th sample as the j-th class and the j-th class predicted by GP.

[0041] A driver emotion recognition and uncertainty assessment system based on deep learning, the system comprising:

[0042] An image input device for acquiring an image of the driver;

[0043] An image enhancement module, used to perform adaptive histogram equalization processing on the driver image;

[0044] Visual feature extraction module, which is used to extract emotional features from images using the improved Wide ResNet network;

[0045] The emotion recognition module is used to classify the probability density distribution obtained by the Gaussian process layer, where the posterior of the Gaussian process is calculated using the random feature extended Laplace approximation;

[0046] The image input device head is responsible for capturing the driver's image and inputting the image data into the image enhancement module;

[0047] The image enhancement module performs adaptive histogram equalization on the input driver image to address the problem caused by uneven lighting inside the car, and outputs the processed image;

[0048] The processed image is passed as input to the visual feature extraction module;

[0049] The visual feature extraction module uses the improved Wide ResNet network to extract emotional features from the image and passes the extracted features as input to the emotion recognition module;

[0050] The emotion recognition module uses the Bayesian theoretical system to classify the probability density distribution obtained through the Gaussian process layer and outputs the driver emotion recognition results.

[0051] Furthermore, the functions of the image enhancement module, visual feature extraction module and emotion recognition module are implemented by a deep learning model.

[0052] Furthermore, the image input device is a camera.

[0053] The beneficial effects of the present invention are:

[0054] (1) Through the improved Wide ResNet network, the present invention can effectively extract the emotional features in the driver's facial image and combine it with the Gaussian process layer for classification, thereby achieving high-precision driver emotion recognition.

[0055] (2) The present invention introduces an uncertainty assessment function, which provides a confidence assessment for the driver emotion recognition results by predicting the distance between the test sample and the training sample, helping the decision-making system to identify potential risks and take corresponding measures, such as reminding the driver to pay attention to safe driving or taking auxiliary driving measures.

[0056] (3) The introduction of the spectral normalization layer and the Gaussian process layer enhances the model's ability to process complex and uncertain data, enabling the system to maintain high recognition accuracy when facing interference factors such as lighting changes and differences in driver characteristics.

[0057] (4) The random feature extension Laplace approximation of the Gaussian process layer enables the model to maintain a high generalization ability even with less training data and adapt to different driving environments and driver characteristics.

[0058] (5) The application can monitor the emotional state of the driver in real time, combine the uncertainty evaluation result, identify potential risks in time, provide safety guarantee for the automatic driving system, and reduce the probability of traffic accidents.

[0059] Other advantages, objects, and features of the application will be set forth in part in the following specification taken in conjunction with the accompanying drawings, and in part will become apparent to those skilled in the art from a consideration of the following specification and drawings. The objects and other advantages of the application will be realized and attained by means of the instrumentalities and combinations pointed out in the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to make the objects, technical solutions and advantages of the application clearer, the preferred detailed description of the application will be combined with the drawings to describe the application, in which:

[0061] Figure 1 Wide Resnet model for the set SNGP;

[0062] Figure 2 Driver emotion recognition method and uncertainty evaluation method general block diagram. DETAILED DESCRIPTION

[0063] The embodiments of the application are described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the application from the disclosure of the specification. The application can also be implemented or applied through other different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0064] The drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation of the application; in order to better illustrate the embodiments of the application, some components in the drawings are omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some known structures and their descriptions in the drawings can be omitted.

[0065] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0066] Referring to Figure 1 and Figure 2 , the method is divided into three stages:

[0067] The first stage is to denoise the input image.

[0068] The second stage uses a WideResnet network to extract image features as input for the subsequent classifier

[0069] The third stage puts the image features into a Gaussian process layer for classification and uncertainty identification.

[0070] The present application mainly has the following three modules:

[0071] 1. Image enhancement module

[0072] The image enhancement module mainly solves the problem of uneven illumination in the driver image caused by uneven illumination in the vehicle through adaptive histogram equalization, and performs histogram equalization in the local area to realize compensation for local illumination changes. The specific steps include:

[0073] (1) divide the image into blocks of equal size

[0074] (2) calculate the number of gray levels r contained in each sub-block k

[0075] (3) count the number n of each gray level k

[0076] (4) calculate the image histogram of the sub-graph

[0077] (5) calculate the transformation function, that is:

[0078]

[0079] Merge each sub-block into the entire image.

[0080] This method can effectively deal with the problem of uneven illumination in the image while preserving more image details.

[0081] 2. Visual feature extraction module:

[0082] We use the WideResNet network to extract the emotional features in the image. The Wide ResNet model extracts image features, and Wide ResNet is a variant of ResNet (Residual Network) that improves performance by adjusting the width (i.e., the number of channels or filters per layer) and depth (i.e., the number of network layers) of the network. Wide ResNet improves accuracy by increasing the width of the network layers rather than the depth.

[0083] The uncertainty assessment function is implemented using a spectral-normalized neural Gaussian process. Spectral-normalized neural Gaussian process (SNGP) is a method that improves the distance perception ability of modern neural networks by adding a weight normalization step during training and replacing the output layer with a Gaussian process. It aims to improve the ability of neural networks to handle uncertainty, especially in predicting novel or out-of-distribution (OOD) data, significantly enhancing the model's performance in handling complex and uncertain data

[0084] The improved Wide ResNet has 4 convolutional layers: the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer. The first convolutional layer has 64 3X3 convolutional kernels, the second convolutional layer has 128 3X3 convolutional kernels, and the third convolutional layer has 256 3X3 convolutional kernels. The fourth layer has 512 3X3 convolutional kernels. The pooling layer between each two layers is a 2X2 pooling layer (maxpool) with a moving step of 16. A residual module is introduced to ensure the gradient correlation between levels. Then, the output feature map will be nonlinearly transformed using an activation function. The activation function uses the PReLU activation function, which has better performance in the classification field. Its formula is as follows:

[0085]

[0086] Uncertainty-aware Wide ResNet: Based on the attractive properties of SN (spectral normalization) and GP (Gaussian process), an uncertainty-aware Wide ResNet, SNGP-G, is developed, which has two key differences compared to the original WideResNet:

[0087] 1) Each convolutional layer in the original architecture is followed by an SN layer. These convolutional layers include: the convolutional layers in the main network, such as Conv Layer 1 and Conv Layer 2, to enable the residual layer to have distance perception capabilities.

[0088] 2) Replace the original dropout layer, fully connected layer, and softmax layer of the proximity output with a GP layer to enable distance calculation capabilities.

[0089] The implementation of spectral normalization is achieved by solving singular values, and the spectral norm of each parameter matrix is approximated by the "power iteration method". The iteration process is as follows:

[0090] (1) Initialization: Choose a random initial vector Usually a Gaussian vector randomly generated from a standard normal distribution.

[0091] (2) Calculate

[0092] (3) Normalization

[0093] (4) Calculate

[0094] (5) Normalization

[0095] (6) Convergence: After K iterations, the approximate value of the largest singular value is:

[0096]

[0097] After obtaining the spectral norm, the parameters on each parameter matrix are divided by it to achieve the purpose of normalization, and to make the convolution part satisfy the Lipschitz continuous condition. 1-Lipschitz refers to a function f: R n →R is called a 1-Lipschitz function if for all x, y ∈ R n , it satisfies the following inequality:

[0098] |f(x)-f(y)|≤|x-y|

[0099] |f(x)-f(y)|≤|x-y|

[0100] This means that the rate of change of function f will not exceed the rate of change of input, i.e. the increment of function f will not exceed the absolute value of the increment of input points. This type of function is very important in optimization, machine learning and numerical analysis, as it guarantees the stability and controllability of the output.

[0101] 3. Emotion recognition module:

[0102] In the emotion recognition module, the features obtained by WideResNet are taken as input features, and the probability density distribution obtained by the Gaussian process layer is classified in the Gaussian process layer by means of the knowledge of latent class conditional probability density function and maximum likelihood estimation based on the Bayesian theory system.

[0103] In order to ensure adaptation to more environments, the present discovery uses a random feature extension Laplace approximation of Gaussian process to realize the Gaussian process posterior, which has higher scalability with relatively small changes to the training model, and makes effective calculation of the uncertainty of the input picture under the premise of ensuring the fast speed of the model. And in the Gaussian process layer, for a certain test input x * Its posterior variance is proportional to its distance L2 in the hidden space to the training data point. Specifically:

[0104] (1) Training data: suppose we have N training samples where y i is the corresponding label, and x i is the input feature.

[0105] (2) Hidden representation: each input x i is converted into a hidden representation h i through a hidden representation function h i = h(x i ).

[0106] (3) Conditional Gaussian process: based on these hidden representations h i , define a conditional Gaussian process g, whose output on N hidden representation points is a vector g N×1 = [g(h1),..., g(h N )] T .

[0107] (4) Gaussian process prior with RBF kernel: this Gaussian process model assumes that g N×1 obeys a multivariate normal distribution, whose mean is zero and covariance matrix is σ 2 K, where the elements of the covariance matrix K are calculated by the RBF kernel function:

[0108]

[0109] (5) RBF kernel function: the RBF kernel function is defined as:

[0110]

[0111] Here, h i, h j h i and h j between them.

[0112] (6) Posterior distribution: Given the observed data We calculate the posterior distribution of g by Bayes' theorem:

[0113]

[0114] where p(g) is the prior distribution described above, and p(D|g) is the likelihood function of the observed data.

[0115] After replacing the last three layers in the original Wide ResNet with Gaussian process layers, the loss function formula in it is as follows:

[0116]

[0117] where N represents the number of samples, C represents the number of classes, γ i,j represents the actual probability of marking the ith sample as the jth class, μ i,j represents the average probability of GP prediction of the jth class of marking the ith sample as the jth class.

[0118] Embodiments

[0119] 1. Evaluation index:

[0120] This work uses recall rate, precision rate, and F1-Score as performance evaluation indexes. Recall rate refers to the proportion of correctly identified positive samples among all actual positive samples. It reflects the coverage ability of the model to actual positive samples. Precision rate refers to the proportion of actual positive samples among all samples predicted as positive. It reflects the accuracy of the model's prediction of positive samples. F1 score is the harmonic mean of precision rate and recall rate. It considers both precision rate and recall rate and is a balanced index between the two.

[0121] 2. Experimental data:

[0122] The dataset used in this work is MLI-DER (Multiple Light Intensities Driver Emotion Recognition). The MLI-DER dataset is a high-quality dataset specifically designed for the study of driver emotion recognition. The dataset aims to overcome the interference of different driver characteristics (such as gender, skin color, age, appearance) and changes in the light conditions inside the car on emotion recognition. The dataset content contains 37 subjects (12 females and 25 males) with an age range of 21 to 37 years old. A total of 442 video samples were obtained, with a video frame rate of 20 fps and a resolution of 640x480 pixels. The key frames extracted from the videos formed 4451 image samples.

[0123] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for driver emotion recognition and uncertainty assessment based on deep learning, characterized by: The method comprises the following steps: Adaptive histogram equalization is performed on the driver image to address the problem of uneven lighting inside the car; The improved Wide ResNet network is used to extract emotional features from driver images. The improved Wide ResNet network includes: At least four convolutional layers, each followed by a spectral normalization (SN) layer; Replace the dropout layer, fully connected layer, and softmax layer close to the output in the original Wide ResNet network with a Gaussian process GP layer; The Bayesian theoretical system is used to classify the probability density distribution obtained through the Gaussian process layer, where the posterior of the Gaussian process is calculated using the random feature extended Laplace approximation; The improved Wide ResNet network includes: The first convolutional layer has 64 3X3 convolution kernels; The second convolutional layer has 128 3X3 convolution kernels; The third convolutional layer has 256 3X3 convolution kernels; The fourth convolutional layer has 512 3X3 convolution kernels; The convolutional layers are connected to a 2X2 pooling layer maxpool with a moving step of 16; Introducing residual modules to ensure gradient correlation between levels; The output feature map is nonlinearly transformed using the PReLU activation function; The implementation method of the spectrum normalization SN layer is: Initialization: Select a random initial vector; Iterative calculation: perform matrix multiplication using the initial vector and the parameter matrix; normalize the result vector; perform matrix multiplication between the normalized result vector and the parameter matrix; normalize the result vector; After K iterations, the approximate value of the maximum singular value is: where σ l (W) is the lth singular value of the matrix W, which indicates the degree of expansion and contraction of the matrix in a certain direction; W represents a matrix, and K represents the number of iterations; is the lth left singular vector u obtained after the Kth iteration i The transpose of is the lth right singular vector v after the Kth iteration l ; Divide the parameters on each parameter matrix by it for normalization purposes; The implementation method of the Gaussian process GP layer is: Training data: There are N training samples where y i is the label of the i-th training sample, x i is the input feature of the i-th training sample; Hidden representation: For each input x i Transformed into a hidden representation h through a hidden representation function i =h(x i );h(x i ) is to input feature x i The result after mapping to the latent space; Conditional Gaussian process: Based on these hidden representations h i , define a conditional Gaussian process g, whose output at N hidden representation points is a vector g N×1 =[g(h1),...,g(h N )] T ; Gaussian process prior with RBF kernel: Gaussian process model assumes g N×1 It follows a multivariate normal distribution with mean zero and covariance matrix σ 2 K, where the elements of the covariance matrix K are calculated using the RBF kernel function: RBF kernel function: The RBF kernel function is defined as: K(x,x′)=exp(-γ‖xx′‖ 2 ), where x and x′ are two input vectors, ‖xx′‖ is the Euclidean distance between x and x′; γ is a parameter that controls the width of the kernel function; Posterior distribution: Given the observed data Calculate the posterior distribution of g using the Bayesian formula: Among them, p(g) is the prior distribution, p(D|g) is the likelihood function of the observed data; The dropout layer, fully connected layer, and softmax layer close to the output in the original Wide ResNet network are replaced with the Gaussian process GP layer. The loss function formula is: Where N is the number of samples, C is the number of categories, and γ i,j represents the actual probability of labeling the i-th sample as the j-th class, μ i,j represents the average probability of labeling the i-th sample as the j-th class and the j-th class predicted by GP.

2. The method for driver emotion recognition and uncertainty assessment based on deep learning according to claim 1, characterized in that: The adaptive histogram equalization process includes: Split the image into blocks of equal size; Calculate the gray level contained in each sub-block; Count the number of gray levels contained in each gray level; Calculate the image histogram of the sub-image; Calculate the transformation function and merge the sub-blocks into the entire image.

3. A deep learning-based driver emotion recognition and uncertainty assessment system, characterized by: The system includes: An image input device for acquiring an image of the driver; An image enhancement module, used to perform adaptive histogram equalization processing on the driver image; Visual feature extraction module, which uses the improved Wide ResNet network to extract emotional features from images; The emotion recognition module is used to classify the probability density distribution obtained by the Gaussian process layer, where the posterior of the Gaussian process is calculated using the random feature extended Laplace approximation; The image input device head is responsible for capturing the driver's image and inputting the image data into the image enhancement module; The image enhancement module performs adaptive histogram equalization on the input driver image to address the problem caused by uneven lighting inside the car, and outputs the processed image; The processed image is passed as input to the visual feature extraction module; The visual feature extraction module uses the improved Wide ResNet network to extract emotional features from the image and passes the extracted features as input to the emotion recognition module; The emotion recognition module uses the Bayesian theoretical system to classify the probability density distribution obtained through the Gaussian process layer and outputs the driver emotion recognition results.

4. The deep learning-based driver emotion recognition and uncertainty assessment system according to claim 3, characterized in that: The functions of the image enhancement module, visual feature extraction module and emotion recognition module are implemented by a deep learning model.

5. The driver emotion recognition and uncertainty assessment system based on deep learning according to claim 3, characterized in that: The image input device is a camera.

Images