Driver emotion recognition method, system, equipment and medium

Abstract

The invention provides a driver emotion recognition method, system and device and a medium, and belongs to the technical field of driver emotion monitoring, and the method comprises the following steps: obtaining historical data of facial expression features, physiological features and driving behavior features of a driver, and clustering the historical data to obtain a clustering result; dividing the driving styles of the drivers into different categories according to a clustering result; inputting the clustering result data corresponding to the different driving style categories into a neural network model, and training the neural network model to obtain emotion recognition models corresponding to the different driving style categories; judging an emotion recognition model corresponding to the driving style category of the driver needing emotion recognition, and recognizing emotion state data of the driver in different time windows according to the corresponding emotion recognition model; and setting an emotion recognition threshold, and judging the emotion state of the driver according to the emotion state data greater than the emotion recognition threshold in each time window. The accuracy of driver emotion recognition can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of driver emotion monitoring, and particularly relates to a method, system, device and medium for identifying driver emotions. Background Art

[0002] With the continuous improvement of economic development and living standards, the number of automobiles in use is also increasing day by day. However, the resulting road traffic safety problems are becoming increasingly serious.

[0003] The road traffic system is a complex system composed of human-vehicle-road environment. All levels of departments have been continuously exerting efforts on the roadside for years, by means of building intelligent highways and high-grade roads, improving road quality, road reconstruction and expansion, etc., to improve road safety conditions. At present, it has entered the stage of diminishing marginal benefits, and the comprehensive improvement effect is not significant. In this process, the role of the driver himself is ignored. Some vehicle traffic accidents are directly related to the poor state of the driver. Among them, fatigue driving, distracted driving and aggressive driving are the three major triggers. Relevant research shows that when the driver is in a poor driving state, his vehicle handling ability and control ability will become sluggish. If the driver's reaction time can be increased by 0.5 s before a traffic accident occurs, the probability of the accident will be reduced by about 60%. Therefore, monitoring and warning the driver's poor state will become an important research direction of automotive active safety technology to effectively ensure the driving safety of the driver.

[0004] Road traffic safety has been increasingly emphasized globally. Through the investigation and research on the causes of traffic accidents in a certain school, it is found that 85% of the accidents are related to the driver, and vehicle and environmental factors only account for 15%. Research shows that the emotional state of the driver is one of the important factors affecting driving behavior. Negative emotions such as anger, fatigue, and anxiety will distract the driver's attention and reduce the reaction speed, thus leading to driver operation errors and increasing the accident risk.

[0005] The existing driver emotion recognition methods are mainly based on non-physiological signals or single physiological signals, such as facial expression recognition, heart rate monitoring, etc. In facial expression recognition, algorithms based on convolutional neural networks (CNNs) and transfer learning have achieved high accuracy in static and dynamic expression recognition. Their analysis of physiological signals such as electrocardiogram (ECG), electro dermal activity (EDA) and electroencephalogram (EEG) can provide a relatively reliable assessment of the emotional state. However, there are significant differences in the physiological and behavioral characteristics of different drivers. The existing emotion recognition methods ignore individual differences, resulting in low monitoring accuracy and practicability: for example, a driver with large physiological fluctuations and aggressive driving may have significant differences in driving behavior from a driver with small physiological fluctuations and stable driving. However, the existing methods show poor adaptability when facing different drivers, resulting in poor recognition accuracy and stability. Summary of the Invention

[0006] To overcome the deficiencies of the above-mentioned prior art, the present invention provides a driver emotion recognition method, which includes the following steps: Obtain historical data of the driver's facial expression features, physiological features, and driving behavior features, perform clustering on the historical data, and divide the driver's driving style into different categories according to the clustering result data; Input the clustering result data corresponding to different driving style categories into a neural network model, train the neural network model, and respectively obtain emotion recognition models corresponding to different driving style categories; Determine the emotion recognition model corresponding to the driving style category of the driver whose emotion needs to be recognized, input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver whose emotion needs to be recognized into the corresponding emotion recognition model, and recognize the emotion state data of the driver in different time windows; determine the emotion state of the driver according to the emotion state data greater than the set emotion recognition threshold within each time window.

[0007] Preferably, the step of inputting the real-time data of the facial expression features, physiological features, and driving behavior features of the driver whose emotion needs to be recognized into the corresponding emotion recognition model and recognizing the emotion state data of the driver in different time windows includes the following steps: Collect the real-time data of the facial expression features, physiological features, and driving behavior features of the driver to be recognized, perform clustering on the real-time data of the physiological features and driving behavior features, and determine the driving style category of the driver according to the clustering result data; Input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver to be recognized into the emotion recognition model corresponding to the driving style category of the driver, and recognize the emotion state data of the driver in different time windows.

[0008] Preferably, before performing clustering on the historical data, it further includes converting the facial expression features into integer labels; using the principal component analysis method to convert the physiological features and driving behavior features into one-dimensional continuous variables.

[0009] Preferably, the physiological features include the driver's heart rate change data and blood pressure change data, and the driving behavior features include the steering wheel angle, vehicle speed, and vehicle acceleration during driving.

[0010] Preferably, the neural network model includes an input layer, a fully connected layer, and an output layer. The fully connected layer is used to perform weighted and non-linear activation processing on the input data, and the output layer uses the Softmax activation function to convert the output data of the fully connected layer into a probability distribution of different emotion categories.

[0011] The present invention also provides a driver emotion recognition system, including: A driving style category acquisition module, configured to acquire historical data of a driver's facial expression features, physiological features, and driving behavior features, perform clustering on the historical data, and classify the driver's driving style into different categories according to the clustering result data; An emotion recognition model acquisition module, configured to input the clustering result data corresponding to different driving style categories into a neural network model, train the neural network model, and respectively obtain emotion recognition models corresponding to different driving style categories; An emotion recognition module, configured to determine the emotion recognition model corresponding to the driving style category of the driver whose emotion needs to be recognized, input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver whose emotion needs to be recognized into the corresponding emotion recognition model, and recognize the emotion state data of the driver in different time windows; determine the emotion state of the driver according to the emotion state data greater than the set emotion recognition threshold in each time window.

[0012] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the driver emotion recognition method.

[0013] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor to execute the driver emotion recognition method.

[0014] The driver emotion recognition method provided by the present invention has the following beneficial effects: By performing clustering analysis on the historical data of the driver's facial expression features, physiological features, and driving behavior features, the present invention can classify drivers into different categories; by respectively training the neural network model with the clustering result data of different categories, emotion recognition models corresponding to different driving style categories can be obtained, thereby fully considering the influence of driver individual differences on the emotion recognition accuracy; by analyzing the emotion prediction results within a certain time window and combining the setting of the threshold, the credibility of the emotion prediction is evaluated. Within a certain period of time, only when the time when a certain emotion of the driver appears exceeds the set threshold, it is determined that the driver is in this type of emotion, effectively improving the stability of recognition. Description of the Drawings

[0015] To more clearly illustrate the embodiments of the present invention and their design solutions, the accompanying drawings required for this embodiment will be briefly introduced below. The accompanying drawings in the following description are only partial embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0016] Figure 1 It is a flowchart of the driver emotion recognition method for the embodiment of the present invention. Specific embodiments

[0017] In order to enable those skilled in the art to better understand the technical solutions of the present invention and be able to implement them, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0018] Embodiment The present invention provides a driver emotion recognition method, specifically as Figure 1 shown, including the following steps: Step 1: Obtain the historical data of the driver's facial expression features, physiological features, and driving behavior features, cluster the historical data, and divide the driver's driving style into different categories according to the clustered result data.

[0019] The driver's facial expression features, physiological features, and driving behavior features are specifically as follows: 1) Driver facial expression features: The present invention uses an in-vehicle camera to collect the driver's facial video, preprocesses the video data, including denoising, smoothing, etc., to improve the image quality. Then, computer vision algorithms (such as convolutional neural network CNN) are used to detect and classify facial expressions, obtain the driver's facial expression state, and convert the driver's facial expression (such as emotion categories like anger, anxiety, calm, etc.) features into integer labels for data analysis and driver emotion recognition.

[0020] 2) Driver physiological features: The present invention uses physiological sensors such as a heart rate monitor to monitor the electrophysiological signal indicators of the driver, preprocesses the signal data, including resampling, band-pass filtering, etc., to ensure the accuracy and integrity of the signal indicators. Considering the differences in the physical characteristics of different drivers, the change value of the physiological signal (such as heart rate change) within a certain period of time is used as the data reflecting physiological features. Then, the selected feature data is processed by the principal component analysis method (PCA) for dimensionality reduction, and converted into a one-dimensional continuous variable to represent the driver's comprehensive physiological features, so as to reduce noise and improve the processability of the model.

[0021] 3) Driver behavior characteristics: The present invention uses in-vehicle sensors to collect data such as the steering wheel angle, vehicle speed, and vehicle acceleration during driving, and obtains the absolute value change rate of speed, acceleration change rate, etc. through analysis, to identify the driver's control behavior and vehicle motion characteristics under different driving states. These characteristics also undergo dimensionality reduction processing and are combined into one-dimensional continuous variables to comprehensively reflect the driver's behavior characteristics.

[0022] Considering the influence of individual differences among different drivers on emotion recognition, the present invention uses the k-means clustering method to cluster drivers into four categories respectively according to physiological characteristics and driving behavior characteristics, so as to further improve the accuracy of emotion recognition. The specific classification basis is as follows:

[0023] 1) Physiological characteristic clustering: Considering that there are differences in the physiological characteristics of different drivers and the influence of emotions on physiological characteristics is also different (such as the change of heartbeat), in order to reduce the influence of physiological characteristic differences on emotion recognition, the present invention divides drivers into two categories based on the influence of emotions on the physiological fluctuations of drivers: drivers with large physiological fluctuations and drivers with small physiological fluctuations.

[0024] 2) Clustering of driving behavior characteristics: Considering that there are differences in the driving behavior characteristics of different drivers and the influence of emotions on driving behavior characteristics is different (such as acceleration, acceleration change rate, etc.), in order to reduce the influence of different driving styles on emotion recognition, the present invention divides drivers into two categories according to the driving behavior characteristics of drivers: drivers with aggressive driving behavior and drivers with stable driving behavior.

[0025] Combining the physiological and behavioral characteristics of drivers for clustering, finally, drivers are clustered into four categories as shown in Table 1 below. By clustering drivers, more accurate feature inputs can be provided for the subsequent emotion recognition model, helping the model to better learn the emotion change rules of different types of drivers.

[0026] Table 1 Driver clustering categories After completing the clustering, for the drivers who need to be recognized for emotions, the k-means algorithm is used to assign the drivers to the corresponding categories according to their physiological characteristics and driving behavior characteristics, so as to improve the accuracy of subsequent emotion recognition.

[0027] Step 2: Input the clustering result data corresponding to different driving style categories into the neural network model, train the neural network model, and respectively obtain the emotion recognition models corresponding to different driving style categories.

[0028] Based on clustering, the present invention trains corresponding neural network models for each type of driver according to the clustering results, so as to match a suitable neural network model according to the driver characteristics for emotion recognition and improve the recognition accuracy. The model takes facial expression features, physiological features, and driving behavior features as inputs and the three emotional states (positive, neutral, negative) of the driver as outputs to recognize the emotional state of the driver. The neural network architecture is as follows:

[0029] 1) Input layer: The input layer of the neural network model is jointly composed of three parts: facial expression features, physiological features, and driving behavior features.

[0030] Among them, the facial expression feature is an integer label representing the facial expression category of the driver. In order to facilitate the neural network to process and capture the potential relationships between facial expressions, this integer label is mapped into a low-dimensional continuous vector through an embedding layer.

[0031] The physiological feature is a one-dimensional continuous variable, which is obtained by reducing the dimensionality of features such as the driver's heart rate change and blood pressure change through the principal component analysis (PCA), and can reflect the comprehensive physiological characteristics of the driver.

[0032] The driving behavior feature is a one-dimensional continuous variable, which is obtained by reducing the dimensionality of the driver's behavior (such as acceleration, acceleration change rate, etc.) through PCA, and can reflect the driving style of the driver.

[0033] 2) Fully connected layer: In the fully connected layer of the neural network model, the input feature data is processed through weighted sum and non-linear activation (such as ReLU activation function), and the neural network learns the potential relationships between features to generate representations that are helpful for emotion recognition.

[0034] 3) Output layer: The output layer uses the Softmax activation function to convert the network output into the probability distributions of three emotion categories, and these emotion categories respectively represent positive, neutral, and negative emotions.

[0035] The training data of the present invention includes facial expression features, the scalar of physiological features after dimensionality reduction, the scalar of driving behavior features, and the corresponding emotion labels (positive, neutral, negative). After the data is preprocessed by standardization and normalization, it is divided into a training set and a validation set for model training and evaluation.

[0036] Since the driver cannot react and record their own emotions in real time during driving, resulting in the scarcity of training data labels, the present invention combines the semi-supervised learning (Semi-Supervised Learning) method to jointly train the model with a small amount of labeled data and a large amount of unlabeled data. In the case of scarce labeled data, the unlabeled data is fully utilized for training to improve the accuracy of emotion recognition.

[0037] During training, the cross-entropy loss function is used as the loss function, and the parameters are updated through the Adam optimizer. During the training process, the value of the loss function is gradually reduced, and the training stops when convergence is achieved.

[0038] After training, the neural network generates the probability distributions of three types of emotions. To determine the final emotion category, the present invention selects the category with the highest probability output by the model as the model recognition result.

[0039] Step 3: Determine the emotion recognition model corresponding to the driving style category of the driver whose emotion needs to be recognized. Input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver whose emotion needs to be recognized into the corresponding emotion recognition model to recognize the emotion state data of the driver in different time windows. The specific steps are as follows: Collect the real-time data of the facial expression features, physiological features, and driving behavior features of the driver to be recognized. Cluster the real-time data of the facial expression features, physiological features, and driving behavior features, and determine the driving style category of the driver according to the clustering result data.

[0040] Input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver to be recognized into the emotion recognition model corresponding to the driving style category of the driver to recognize the emotion state data of the driver in different time windows.

[0041] Step 4: Determine the emotion state of the driver according to the emotion state data greater than the set emotion recognition threshold in each time window.

[0042] To improve the accuracy and reliability of emotion prediction, the present invention introduces an emotion threshold adjustment mechanism based on time windows, and dynamically evaluates the emotion state of the driver by statistically analyzing the ratio of emotion prediction results within a time period. This method analyzes the emotion prediction results within a certain time window and combines the setting of the ratio threshold to evaluate the credibility of emotion prediction, ensuring a balance between the accuracy and real-time performance of the driver's emotion recognition. The specific process is as follows:

[0043] 1) Emotion recognition time window.

[0044] In practical applications, the emotion state has strong temporal characteristics, that is, the emotions of the same driver may remain unchanged or change relatively smoothly in a short period of time. To enhance the stability of emotion recognition, the present invention performs aggregation processing of emotion prediction results in time windows, and the length of each time window is set to t seconds, and the emotion of the driver is continuously recognized multiple times within this time period.

[0045] Set the prediction frequency of the emotion recognition model to kSecondly, within a time window t seconds, a total of records of emotion recognition results will be made. By statistically analyzing these results, the trend of the driver's emotional changes during this period can be more comprehensively reflected.

[0046] .

[0047] 2) Threshold setting and determination.

[0048] To improve the accuracy and reliability of emotion prediction, within each time window, the present invention counts the number of times the driver is recognized as having a negative emotion as , and calculates the ratio of the number of times recognized as having a negative emotion to the total number of recognition times .

[0049] .

[0050] The present invention sets the threshold as . When is higher than , it is considered that the prediction result of negative emotion within this time window has a high credibility, indicating that the driver is in a negative emotion and a warning needs to be issued; conversely, if is lower than , it is considered that the credibility of emotion prediction is low and no warning is given.

[0051] 3) Threshold adjustment and sensitivity.

[0052] Considering that different driving environments, individual differences of different drivers, and different ranges of emotional fluctuations may require different threshold settings, the present invention will dynamically adjust the threshold according to the actual application scenario . By optimizing the threshold during the experiment, the balance between prediction sensitivity and warning accuracy can be achieved.

[0053] For example, in some environments with high requirements for driver behavior (such as complex road conditions or high-intensity driving tasks), the threshold can be appropriately reduced to detect potential negative emotions earlier, so as to take measures to help the driver maintain emotional stability; while in environments with low requirements for driver behavior (such as simple sections or low-risk scenarios), the threshold can be appropriately increased to avoid unnecessary warnings.

[0054] Through this method based on time window and threshold, the present invention can effectively reduce unnecessary warnings caused by accidental emotion prediction errors, while improving the sensitivity and accuracy of recognizing negative emotion states. This method has both real-time performance and robustness, providing an efficient solution for emotion prediction in dynamic driving scenarios.

[0055] The present invention also provides a driver emotion recognition system, including: A driving style category acquisition module, configured to acquire historical data of a driver's facial expression features, physiological features, and driving behavior features, perform clustering on the historical data, and classify the driver's driving style into different categories according to the clustering result data; An emotion recognition model acquisition module, configured to input the clustering result data corresponding to different driving style categories into a neural network model, train the neural network model, and respectively obtain emotion recognition models corresponding to different driving style categories; An emotion recognition module, configured to determine the emotion recognition model corresponding to the driving style category of the driver whose emotion needs to be recognized, input the real-time data of the facial expression features, physiological features, and driving behavior features of the driver whose emotion needs to be recognized into the corresponding emotion recognition model, and recognize the emotion state data of the driver in different time windows; determine the emotion state of the driver according to the emotion state data greater than the set emotion recognition threshold within each time window.

[0056] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is configured to run the computer program in the memory to execute the driver emotion recognition method.

[0057] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor to execute the driver emotion recognition method.

[0058] The driver emotion recognition method provided by the present invention has the following beneficial effects: 1) Aiming at the problems of complex data and low accuracy in the driver emotion monitoring technology, the present invention comprehensively considers the driver's facial expression features, physiological features (electrocardiogram signals), and driving behavior features (automobile driving parameters), extracts the optimal feature subset that can reflect the driving state, constructs a real-time driver emotion monitoring model based on multi-source information fusion, realizes more accurate and real-time driver emotion recognition, and thus effectively prevents traffic accidents caused by driver emotion fluctuations.

[0059] 2) Aiming at the problem that the existing emotion recognition methods do not consider the individual differences of drivers, the present invention performs clustering analysis on the historical data of drivers, divides the drivers into four categories, and trains the corresponding model parameters for different categories of drivers, fully considering the influence of driver individual differences on the emotion recognition accuracy.

[0060] 3) The present invention also introduces an emotion threshold adjustment mechanism based on a time window, which dynamically evaluates the driver's emotional state by statistically analyzing the ratio of emotion prediction results within a time period. By analyzing the emotion prediction results within a certain time window and combining the setting of the threshold, this method evaluates the credibility of emotion prediction. Within a certain period of time, only when the time when a certain emotion of the driver appears exceeds the set threshold will it be determined that the driver is in this type of emotion, effectively improving the stability of recognition.

[0061] The above-described embodiments are only preferred specific embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of technical solutions that can be obviously obtained by those skilled in the art within the technical scope disclosed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A driver emotion recognition method, characterized in that: The steps include: Obtain historical data of the driver's facial expression features, physiological features, and driving behavior features, cluster the historical data, and classify the driver's driving style into different categories based on the clustering result data; Inputting the clustering result data corresponding to different driving style categories into a neural network model, training the neural network model, and obtaining emotion recognition models corresponding to different driving style categories; Determine the emotion recognition model corresponding to the driving style category of the driver who needs emotion recognition, input the real-time data of the facial expression characteristics, physiological characteristics and driving behavior characteristics of the driver who needs emotion recognition into the corresponding emotion recognition model, and identify the driver's emotional state data in different time windows; determine the driver's emotional state based on the emotional state data greater than the set emotion recognition threshold in each time window.

2. The driver emotion recognition method according to claim 1, characterized in that: The method of inputting the real-time data of the facial expression characteristics, physiological characteristics and driving behavior characteristics of the driver who needs to perform emotion recognition into the corresponding emotion recognition model to recognize the emotional state data of the driver in different time windows includes the following steps: Collecting real-time data of facial expression features, physiological features and driving behavior features of the driver to be identified, clustering the real-time data of facial expression features, physiological features and driving behavior features, and determining the driver's driving style category based on the clustering result data; The real-time data of the facial expression features, physiological features and driving behavior features of the driver to be identified are input into the emotion recognition model corresponding to the driver's driving style category to identify the driver's emotional state data in different time windows.

3. The driver emotion recognition method according to claim 1, characterized in that: Before clustering the historical data, the method also includes converting facial expression features into integer labels; and converting physiological features and driving behavior features into one-dimensional continuous variables using principal component analysis.

4. The driver emotion recognition method according to claim 1, characterized in that: The physiological characteristics include the driver's heart rate change data and blood pressure change data, and the driving behavior characteristics include the steering wheel angle, vehicle speed and vehicle acceleration during driving.

5. The driver emotion recognition method according to claim 1, characterized in that: The neural network model includes an input layer, a fully connected layer and an output layer. The fully connected layer is used to perform weighted and nonlinear activation processing on the input data. The output layer adopts a Softmax activation function to convert the output data of the fully connected layer into probability distributions of different emotion categories.

6. A driver emotion recognition system, characterized in that: include: A driving style category acquisition module is used to acquire historical data of the driver's facial expression features, physiological features, and driving behavior features, cluster the historical data, and classify the driver's driving style into different categories based on the clustering result data; An emotion recognition model acquisition module is used to input the clustering result data corresponding to different driving style categories into a neural network model, train the neural network model, and obtain emotion recognition models corresponding to different driving style categories; The emotion recognition module is used to determine the emotion recognition model corresponding to the driving style category of the driver who needs emotion recognition, input the real-time data of the facial expression characteristics, physiological characteristics and driving behavior characteristics of the driver who needs emotion recognition into the corresponding emotion recognition model, and identify the driver's emotional state data in different time windows; determine the driver's emotional state based on the emotional state data greater than the set emotion recognition threshold in each time window.

7. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the driver emotion recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the driver emotion recognition method according to any one of claims 1-5.

Images