Driver emotion recognition method based on ensemble learning

Abstract

The invention relates to a driver emotion recognition method based on ensemble learning, and belongs to the field of computer vision and artificial intelligence. The method comprises two stages of image preprocessing and deep learning architecture integration. In the image preprocessing stage, size adjustment, normalization and data enhancement are carried out on an input image so as to eliminate noise and improve the generalization ability of a model. In the stage of integrating the deep learning architecture, a wide residual network, a VGG network and a DenseNet network are used for extracting image features, and dimensionality reduction is carried out on the features through principal component analysis (PCA). And the dimensionality-reduced features are input into a random forest for classification prediction, so that the recognition accuracy is improved. According to the driver emotion recognition method, characteristics of multiple deep learning networks are integrated, the advantages of the random forest are combined, the accuracy and robustness of driver emotion recognition are effectively improved, and technical support is provided for application scenes such as an advanced driving assistance system.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and artificial intelligence, and relates to a driver emotion recognition method based on ensemble learning. Background Art

[0002] In recent years, facial expression recognition has been widely studied and can be applied to a variety of usage scenarios such as human-computer interaction, medical treatment, and advanced driver assistance systems.

[0003] Facial expression recognition is crucial for advanced driver assistance systems in improving road safety. Since facial expressions can reflect the driver's fatigue and emotional state, facial expression recognition is very useful for advanced driver assistance systems to identify the driver's state, thereby providing timely alerts to the driver and neighboring drivers.

[0004] The driver's emotional state is considered to be one of the factors that affect driving performance. In particular, negative emotions such as anger, fear, and sadness can even lead to traffic accidents. Studies have also shown that negative emotions are more likely to be triggered when encountering stimulating events while driving, and these negative emotions often lead to reckless driving behaviors, ultimately endangering the driver and others. Therefore, the key to reducing traffic accidents may be to effectively control the driver's emotional state, and the upstream task of controlling the driver's emotional state, effectively identifying the driver's emotional state, is very important.

[0005] Ensemble learning is a machine learning technique that combines multiple individual learners (such as classifiers and regressors) to improve the overall performance and robustness of the model. Ensemble learning methods can reduce the bias and variance of the model, thereby improving the accuracy and stability of predictions. Commonly used integration strategies in classification research include Boosting, Bagging, and Stacking. Bagging is to generate multiple different sub-datasets by sampling the original data set multiple times with replacement, and then train independent models on each sub-dataset. The final prediction result is determined by averaging the prediction results of each model (for regression tasks) or voting (for classification tasks). The prediction combination of various classifiers trained on various subsets of the training data improves the overall performance.

[0006] Currently, many researchers use ResNet networks to extract deep features, and some researchers use Transformer networks to identify emotions. Although these emotion recognition algorithms have achieved good results, there are still some problems, such as the large impact of noise, and the extraction of image facial features is easily affected by other unimportant features to affect the accuracy of discrimination. In this case, introducing ensemble learning into driver emotion recognition can well make up for these problems. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a driver emotion recognition method based on ensemble learning.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A driver emotion recognition method based on ensemble learning comprises the following steps:

[0010] Step 1: Preprocess the input image, including:

[0011] Resize images to a consistent size;

[0012] Normalize the image;

[0013] Perform data augmentation on images, including rotating, flipping, or changing colors;

[0014] Step 2: Use an integrated learning network to extract image features and put them into a random forest for classification prediction, where:

[0015] The integrated learning network includes at least two deep learning networks among the wide residual network, the VGG network and the DenseNet network;

[0016] After extracting features from each deep learning network, principal component analysis (PCA) is used to reduce the dimensionality of the features;

[0017] The reduced features are input into random forest for classification prediction.

[0018] Furthermore, in step 2, the deep learning network used is:

[0019] Wide residual network, including at least four convolutional layers, each with 64, 128, 256 and 512 convolution kernels, and a residual layer;

[0020] VGG network, including multiple 3x3 convolution kernels and ReLU activation function, and using 2x2 max pooling layer after every two convolution layers;

[0021] DenseNet network, including densely connected convolutional layers, each convolutional layer is connected to all subsequent layers.

[0022] Furthermore, in step 2, inputting the reduced-dimensional features into the random forest for classification prediction includes the following steps:

[0023] Use sampling with replacement to create a training set for the decision tree;

[0024] In the process of building a decision tree, a subset of features is randomly selected for node splitting;

[0025] Build multiple decision trees;

[0026] Prediction of driver emotion recognition using decision trees.

[0027] Furthermore, in step 2, the reduced-dimensional features are input into the random forest for classification prediction and the accuracy is judged using evaluation indicators, and the evaluation indicators used include:

[0028] The recognition accuracy of different expressions of anger, fear, disgust, happiness, surprise, sadness and neutral;

[0029] The overall recognition accuracy.

[0030] The beneficial effects of the present invention are:

[0031] (1) Through ensemble learning technology, the features of multiple deep learning networks are integrated and combined with random forest for classification prediction, which can effectively improve the accuracy of the driver emotion recognition model.

[0032] (2) Ensemble learning technology can reduce the bias and variance of the model, thereby improving the accuracy and stability of the prediction and enhancing the robustness and noise resistance of the model.

[0033] (3) Data augmentation technology can generate more diverse training samples, helping the model learn more generalizable features and thus avoid overfitting.

[0034] Compared with the prior art, the present invention uses integrated learning technology on the basis of realizing driver emotion recognition, which can effectively improve the accuracy of the driver emotion recognition model and enhance the robustness and anti-noise ability of the driver emotion recognition model.

[0035] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0037] Figure 1 Overall block diagram of the driver emotion recognition method based on ensemble learning. DETAILED DESCRIPTION

[0038] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0039] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0040] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0041] See also Figure 1 , this method is divided into two stages:

[0042] The first stage preprocesses the input image.

[0043] The second stage uses an integrated learning network to extract image features and puts them into a random forest for classification prediction.

[0044] The present invention mainly comprises the following 2 steps:

[0045] (1) Image preprocessing

[0046] There are three main preprocessing steps:

[0047] ① Adjust image size: To ensure that all images have a consistent size, the present invention uses the OpenCV module (cv2) to adjust the image size to 75x75 pixels. This adjustment process does not destroy the key features of the image because this module only changes the size of the image without modifying the existing pixel values.

[0048] ② Image normalization: In order to ensure that the pixel values ​​in the image are within a consistent range, the present invention generally divides each pixel value by a maximum value (e.g., 255) and normalizes it to a value between 0 and 1. The purpose of normalization is to help the model converge faster during training and promote optimization.

[0049] ③Data augmentation: Data augmentation is a valuable method to generate other variants of the training dataset, aiming to improve the robustness of the model and avoid overfitting. Common augmentation techniques include rotating, flipping, and changing the color of the image. These techniques can provide the model with more diverse training samples, thereby improving its generalization ability.

[0050] (2) Integrated deep learning architecture:

[0051] We first apply WideResNet on the dataset and extract features from the pooling layer, then repeat the same process for DenseNet and VGG. After extracting features from these architectures, we use PCA to reduce their size. Then we stack them with random forest.

[0052] ①Wide residual network

[0053] WideResNet is a popular deep network that consists of 1 fully connected layer and 49 convolutional layers. This architecture is different from the original convolutional neural network (CNN), in which each layer feeds back to the next layer. The key idea behind ResNet is the use of residual blocks, which enables the network to learn quickly by staying close to the data manifold.

[0054] Define x l That is the output of the residual unit. The output equation of the residual unit (skip connection) is as follows:

[0055] x l =f(x l-1 )+x l-1

[0056] In the residual network, the present invention uses 4 convolutional layers, each of which has 64, 128, 256, and 512 convolution kernels in order, and adds a residual layer to avoid neural network degradation and alleviate the problem of gradient disappearance.

[0057] ②VGG network

[0058] The VGG (Visual Geometry Group) network is a deep convolutional neural network architecture. In the present invention, the convolution layer of the VGG network uses multiple 3x3 convolution kernels, and each convolution layer is followed by a ReLU (Rectified Linear Unit) activation function to introduce nonlinearity, so that the model can better learn complex patterns and features. In the present invention, after every two convolution layers, the network uses a 2x2 maximum pooling layer (max-pooling). It is used to reduce the size of the feature map, thereby reducing the computational complexity while retaining important features. After the convolution and pooling layers are 3 layers of fully connected layers, each fully connected layer is also followed by a ReLuc activation function, which is responsible for mapping the features extracted by the convolution layer to the final classification result.

[0059] ③DenseNet network

[0060] DenseNet (Dense Convolutional Network) is a convolutional neural network. Its uniqueness lies in that the output of each layer is connected to each subsequent layer, forming a dense connection between layers. The layers inside each dense block are connected through dense connections. Each layer receives the feature maps from all previous layers as input and passes its own feature maps to all subsequent layers. This connection method can be expressed by the following formula:

[0061] x l =H l ([x0,x1,x2,...,x l-1 ])

[0062] where x0,x1,x2,...,x l-1 is the feature map from layer 0 to layer l-1, H l Represents the transformation function of the lth layer, which includes batch normalization, ReLU activation function and convolution operation.

[0063] After extracting deep features from each network, the covariance matrix is ​​calculated for the data:

[0064]

[0065] Where n is the number of samples. standardized is the normalized data matrix.

[0066] Perform eigenvalue decomposition on the covariance matrix C to obtain eigenvalues ​​and eigenvectors:

[0067] Cv i =λ i v i

[0068] Among them, λ i is the eigenvalue, v i is the corresponding eigenvector.

[0069] Sort the eigenvalues ​​in descending order, and select the eigenvectors corresponding to the first k largest eigenvalues ​​as the principal components. Let these eigenvectors form a matrix, in which each column is an eigenvector.

[0070] Finally, the original data is projected into the selected principal component space to obtain the reduced-dimensional data matrix Z:

[0071] Z=X standardized V k

[0072] Then put the produced data matrix into the random forest for classification prediction. The specific steps are:

[0073] a) Draw samples from the data with replacement to create a training set for the decision tree.

[0074] b) For each decision tree, randomly select features as a feature subset. In the process of building the decision tree, each time a node is split, the optimal feature is selected from this feature subset for splitting.

[0075] c) Repeat step b) to build a decision tree.

[0076] Finally, the decision tree is used to predict driver emotion recognition.

[0077] Example

[0078] Evaluation indicators:

[0079] This work uses the recognition accuracy of seven different expressions (including anger, fear, disgust, happiness, surprise, sadness and neutral) and the overall recognition accuracy as evaluation indicators.

[0080]

[0081] Among them: TP represents the number of true positive classes, that is, the number of samples that are actually positive and correctly predicted as positive; TN represents the number of true negative classes, that is, the number of samples that are actually negative and correctly predicted as negative; FP represents the number of false positive classes, that is, the number of samples that are actually negative but incorrectly predicted as positive; FN represents the number of false negative classes, that is, the number of samples that are actually positive but incorrectly predicted as negative.

[0082] Experimental data:

[0083] The dataset used in this work is the CK+ dataset. The CK+ dataset (Extended Cohn-Kanade Dataset) is a widely used facial expression recognition dataset that contains a series of video sequences and images with annotated facial expressions. The CK+ dataset is mainly used to study facial expression recognition and sentiment analysis. It contains 593 video sequences that show the process of gradually changing from neutral expressions to target expressions. The dataset extracts the last few frames of each video sequence, which contains a total of 327 image sequences, each of which contains a complete expression. The number of frames in each video sequence varies, and the transition frames from neutral expressions to target expressions are usually between 10 and 60 frames. The CK+ dataset annotates 7 basic facial expressions: Angry, Disgust, Fear, Happy, Sadness, Surprise, and Neutral.

[0084] These expression categories cover a wide range of human emotions and are suitable for research fields such as sentiment analysis and human-computer interaction.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A driver emotion recognition method based on ensemble learning, characterized by: The following steps are involved: Step 1: Preprocess the input image, including: Resize images to a consistent size; Normalize the image; Perform data augmentation on images, including rotating, flipping, or changing colors; Step 2: Use an integrated learning network to extract image features and put them into a random forest for classification prediction, where: The integrated learning network includes at least two deep learning networks in the wide residual network, super-resolution test sequence (Visual Geometry Group, VGG network) and DenseNet network; After extracting features from each deep learning network, principal component analysis (PCA) is used to reduce the dimension of the features; The reduced features are input into random forest for classification prediction.

2. The driver emotion recognition method based on ensemble learning according to claim 1 is characterized in that: In step 2, the deep learning network used is: Wide residual network, including at least four convolutional layers, each with 64, 128, 256 and 512 convolution kernels, and a residual layer; VGG network, including multiple 3x3 convolution kernels and ReLU activation function, and using 2x2 max pooling layer after every two convolution layers; DenseNet network, including densely connected convolutional layers, each convolutional layer is connected to all subsequent layers.

3. The driver emotion recognition method based on ensemble learning according to claim 1 is characterized in that: In the step 2, inputting the reduced dimension features into the random forest for classification prediction includes the following steps: Use sampling with replacement to create a training set for the decision tree; In the process of building a decision tree, a subset of features is randomly selected for node splitting; Build multiple decision trees; Prediction of driver emotion recognition using decision trees.

4. The driver emotion recognition method based on ensemble learning according to claim 1 is characterized in that: In the step 2, the reduced dimension features are input into the random forest for classification prediction and the accuracy is judged using evaluation indicators, and the evaluation indicators used include: The recognition accuracy of different expressions of anger, fear, disgust, happiness, surprise, sadness and neutral; The overall recognition accuracy.

Images