A driver emotion recognition method, system, device and medium

Abstract

The application provides a driver emotion recognition method, system, device and medium, and belongs to the technical field of driver emotion monitoring. The method comprises the following steps: obtaining historical data of facial expression features, physiological features and driving behavior features of a driver, clustering the historical data, and dividing the driving style of the driver into different categories according to the clustering results; inputting the clustering result data corresponding to different driving style categories into a neural network model, training the neural network model, and obtaining emotion recognition models corresponding to different driving style categories; determining the emotion recognition model corresponding to the driving style category of the driver who needs to be recognized, recognizing the emotion state data of the driver in different time windows according to the corresponding emotion recognition model; setting an emotion recognition threshold, and determining the emotion state of the driver according to the emotion state data greater than the emotion recognition threshold in each time window. The application can improve the accuracy of driver emotion recognition.

Description

Technical Field

[0001] This invention belongs to the field of driver emotion monitoring technology, specifically relating to a driver emotion recognition method, system, device, and medium. Background Technology

[0002] With economic development and the continuous improvement of living standards, the number of cars on the road is increasing. However, the resulting road traffic safety problems are also becoming increasingly serious.

[0003] The road traffic system is a complex system composed of people, vehicles, and the road environment. Various levels of government have consistently focused on roadside efforts, such as building smart highways and high-grade roads, improving road quality, and road reconstruction and expansion, to improve road safety. However, this has now entered a stage of diminishing marginal returns, with limited overall improvement. In this process, the role of the driver has been neglected. Many traffic accidents are directly related to poor driver performance, with fatigue, distracted driving, and aggressive driving being the three major triggers. Related research shows that when drivers are in a poor driving state, their ability to manipulate and control the vehicle becomes sluggish. If a driver's reaction time can be improved by 0.5 seconds before an accident occurs, the probability of an accident will be reduced by approximately 60%. Therefore, monitoring and providing early warnings of poor driver performance will become an important research direction for automotive active safety technology to effectively ensure driver safety.

[0004] Road traffic safety is receiving increasing attention globally. A study investigating traffic accidents at a school revealed that 85% of accidents were driver-related, while vehicle and environmental factors accounted for only 15%. Research indicates that a driver's emotional state is a significant factor influencing driving behavior; negative emotions such as anger, fatigue, and anxiety can distract drivers, slow reaction times, and lead to operational errors, increasing the risk of accidents.

[0005] Existing methods for driver emotion recognition primarily rely on non-physiological signals or single physiological signals, such as facial expression recognition and heart rate monitoring. In facial expression recognition, algorithms based on convolutional neural networks (CNNs) and transfer learning have achieved high accuracy in both static and dynamic expression recognition. Their analysis of physiological signals such as electrocardiogram (ECG), electrical skin activity (EDA), and electroencephalogram (EEG) can provide relatively reliable assessments of emotional states. However, significant differences exist in the physiological and behavioral characteristics of different drivers. Existing emotion recognition methods ignore individual differences, leading to low monitoring accuracy and practicality. For example, a driver with large physiological fluctuations and aggressive driving behavior may exhibit significant differences in driving behavior compared to a driver with small physiological fluctuations and stable driving behavior. However, existing methods show poor adaptability when faced with different drivers, resulting in poor recognition accuracy and stability. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention provides a driver emotion recognition method, comprising the following steps:

[0007] Historical data on drivers' facial expression features, physiological features, and driving behavior features are obtained, and the historical data is clustered. Based on the clustering results, drivers' driving styles are divided into different categories.

[0008] The clustering results data corresponding to different driving style categories are input into the neural network model, and the neural network model is trained to obtain the emotion recognition model corresponding to different driving style categories.

[0009] The system determines the corresponding emotion recognition model for the driving style category of the driver who needs emotion recognition. Real-time data of the driver's facial expression features, physiological features, and driving behavior features are input into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The driver's emotional state is determined based on the emotional state data exceeding the set emotion recognition threshold within each time window.

[0010] Preferably, the step of inputting real-time data of the driver's facial expression features, physiological features, and driving behavior features that require emotion recognition into the corresponding emotion recognition model to identify the driver's emotional state data at different time windows includes the following steps:

[0011] Real-time data of facial expression features, physiological features and driving behavior features of the driver to be identified are collected. The real-time data of physiological features and driving behavior features are clustered, and the driver's driving style category is determined based on the clustering results.

[0012] Real-time data of the driver's facial expression features, physiological features, and driving behavior features are input into the emotion recognition model corresponding to the driver's driving style category to identify the driver's emotional state data in different time windows.

[0013] Preferably, before clustering the historical data, the method further includes converting facial expression features into integer labels; and using principal component analysis to convert physiological features and driving behavior features into one-dimensional continuous variables.

[0014] Preferably, the physiological characteristics include driver's heart rate change data and blood pressure change data, and the driving behavior characteristics include steering wheel angle, vehicle speed and vehicle acceleration during driving.

[0015] Preferably, the neural network model includes an input layer, a fully connected layer, and an output layer. The fully connected layer is used to perform weighted and non-linear activation processing on the input data, and the output layer uses the Softmax activation function to convert the output data of the fully connected layer into probability distributions of different emotion categories.

[0016] This invention also provides a driver emotion recognition system, comprising:

[0017] The driving style category acquisition module is used to acquire historical data on the driver's facial expression features, physiological features, and driving behavior features, cluster the historical data, and classify the driver's driving style into different categories based on the clustering results.

[0018] The emotion recognition model acquisition module is used to input the clustering result data corresponding to different driving style categories into the neural network model, train the neural network model, and obtain the emotion recognition model corresponding to different driving style categories respectively.

[0019] The emotion recognition module is used to determine the emotion recognition model corresponding to the driving style category of the driver who needs to be identified. Real-time data of the driver's facial expression features, physiological features and driving behavior features are input into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The driver's emotional state is determined based on the emotional state data that exceeds the set emotion recognition threshold within each time window.

[0020] The present invention also provides a computer device including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the driver emotion recognition method.

[0021] The present invention also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the driver emotion recognition method.

[0022] The driver emotion recognition method provided by this invention has the following beneficial effects:

[0023] This invention uses cluster analysis of historical data on drivers' facial expression features, physiological characteristics, and driving behavior features to categorize drivers into different groups. By training a neural network model with clustering results from different groups, different emotion recognition models corresponding to different driving style categories can be obtained, thus fully considering the impact of individual driver differences on the accuracy of emotion recognition. This invention evaluates the reliability of emotion prediction by analyzing emotion prediction results within a certain time window and setting a threshold. Within a certain time, the driver is only identified as being in that emotion if the duration of a certain emotion exceeds the set threshold, effectively improving the stability of recognition. Attached Figure Description

[0024] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a driver emotion recognition method according to an embodiment of the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0027] Example

[0028] This invention provides a method for driver emotion recognition, specifically as follows: Figure 1 As shown, it includes the following steps:

[0029] Step 1: Obtain historical data on the driver's facial expression features, physiological features, and driving behavior features. Cluster the historical data and classify the driver's driving style into different categories based on the clustering results.

[0030] The driver's facial expression characteristics, physiological characteristics, and driving behavior characteristics are as follows:

[0031] 1) Driver facial expression features: This invention uses an in-vehicle camera to collect driver facial videos, preprocesses the video data including noise reduction and smoothing to improve image quality, and then uses computer vision algorithms (such as convolutional neural networks CNN) to detect and classify facial expressions to obtain the driver's facial expression state. The driver's facial expression features (such as anger, anxiety, calmness and other emotion categories) are converted into integer labels to facilitate data analysis and driver emotion recognition.

[0032] 2) Driver Physiological Characteristics: This invention utilizes physiological sensors such as heart rate monitors to monitor the driver's electrocardiographic physiological signals. Preprocessing of the signal data, including resampling and bandpass filtering, ensures the accuracy and completeness of the signal indicators. Considering the differences in physical characteristics among different drivers, physiological signal changes (such as heart rate changes) over a certain period are used as data reflecting physiological characteristics. The selected feature data is then subjected to dimensionality reduction using Principal Component Analysis (PCA) to transform it into one-dimensional continuous variables representing the driver's comprehensive physiological characteristics, thereby reducing noise and improving the model's processability.

[0033] 3) Driver behavior characteristics: This invention uses on-board sensors to collect data such as steering wheel angle, vehicle speed, and vehicle acceleration during the driving process, and analyzes the absolute change rate of speed and the rate of change of acceleration to identify the driver's control behavior and vehicle motion characteristics under different driving conditions. These characteristics are also dimensionality-reduced and merged into one-dimensional continuous variables to comprehensively reflect the driver's behavior characteristics.

[0034] Considering the impact of individual differences among drivers on emotion recognition, this invention uses the k-means clustering method to group drivers into four categories based on physiological characteristics and driving behavior characteristics, thereby further improving the accuracy of emotion recognition. The specific classification criteria are as follows:

[0035] 1) Physiological feature clustering: Considering the differences in physiological features among different drivers, and the varying degrees of influence of emotions on physiological features (such as changes in heart rate), in order to reduce the impact of physiological feature differences on emotion recognition, this invention divides drivers into two categories based on the influence of emotions on drivers' physiological fluctuations: drivers with larger physiological fluctuations and drivers with smaller physiological fluctuations.

[0036] 2) Clustering of driving behavior characteristics: Considering that different drivers have different driving behavior characteristics and that emotions have different effects on driving behavior characteristics (such as acceleration, rate of change of acceleration, etc.), in order to reduce the impact of different driving styles on emotion recognition, this invention divides drivers into two categories based on their driving behavior characteristics: drivers with aggressive driving behavior and drivers with stable driving behavior.

[0037] By combining the physiological and behavioral characteristics of drivers, the drivers were finally clustered into four categories, as shown in Table 1 below. By clustering the drivers, more accurate feature inputs can be provided for the subsequent emotion recognition model, helping the model to better learn the emotional change patterns of different types of drivers.

[0038] Table 1 Driver Cluster Categories

[0039]

[0040] After clustering is completed, for drivers who need emotion recognition, the k-means algorithm is used to classify them into corresponding categories based on their physiological characteristics and driving behavior characteristics, so as to improve the accuracy of subsequent emotion recognition.

[0041] Step 2: Input the clustering results data corresponding to different driving style categories into the neural network model, train the neural network model, and obtain the emotion recognition model corresponding to different driving style categories.

[0042] This invention, based on clustering, trains a corresponding neural network model for each driver category according to the clustering results. This allows for matching a suitable neural network model to driver characteristics for emotion recognition, improving recognition accuracy. The model uses facial expression features, physiological features, and driving behavior features as input, and outputs three driver emotion states (positive, neutral, and negative) to identify the driver's emotional state. The neural network architecture is as follows:

[0043] 1) Input layer: The input layer of the neural network model consists of three parts: facial expression features, physiological features, and driving behavior features.

[0044] Among them, facial expression features are integer labels representing the driver's facial expression category. In order to facilitate neural network processing and capture the potential relationship between facial expressions, the integer labels are mapped into low-dimensional continuous vectors through an embedding layer.

[0045] Physiological characteristics are one-dimensional continuous variables, obtained by principal component analysis (PCA) of the driver's heart rate changes, blood pressure changes, and other characteristics, and can reflect the driver's comprehensive physiological characteristics.

[0046] Driving behavior characteristics are one-dimensional continuous variables, obtained by PCA dimensionality reduction of driver behavior (such as acceleration, rate of change of acceleration, etc.), and can reflect the driver's driving style.

[0047] 2) Fully connected layer: In the fully connected layer of the neural network model, the input feature data is processed by weighted and non-linear activation (such as ReLU activation function). The neural network learns the potential relationship between features and generates a representation that helps with emotion recognition.

[0048] 3) Output layer: The output layer uses the Softmax activation function to convert the network output into a probability distribution of three emotion categories, which represent positive, neutral and negative emotions respectively.

[0049] The training data of this invention includes facial expression features, dimensionality-reduced physiological feature scalars, driving behavior feature scalars, and corresponding emotion labels (positive, neutral, negative). After standardization and normalization preprocessing, the data is divided into training set and validation set for model training and evaluation.

[0050] Since drivers cannot react to and record their emotions in real time while driving, training data labels are scarce. This invention combines semi-supervised learning to train the model using a small amount of labeled data and a large amount of unlabeled data. In the case of scarce labeled data, it makes full use of unlabeled data for training, thereby improving the accuracy of emotion recognition.

[0051] During training, the cross-entropy loss function is used as the loss function, and the parameters are updated using the Adam optimizer. During training, the loss function value is gradually decreased, and training stops upon convergence.

[0052] After training, the neural network generates probability distributions for three types of emotions. To determine the final emotion category, this invention selects the category with the highest probability output by the model as the model recognition result.

[0053] Step 3: Determine the emotion recognition model corresponding to the driving style category of the driver requiring emotion recognition. Input the real-time data of the driver's facial expression features, physiological features, and driving behavior features into the corresponding emotion recognition model to identify the driver's emotional state data at different time windows. This specifically includes the following steps:

[0054] Real-time data of facial expression features, physiological features, and driving behavior features of the driver to be identified are collected. The real-time data of facial expression features, physiological features, and driving behavior features are clustered, and the driver's driving style category is determined based on the clustering results.

[0055] Real-time data of the driver's facial expression features, physiological features, and driving behavior features are input into the emotion recognition model corresponding to the driver's driving style category to identify the driver's emotional state data in different time windows.

[0056] Step 4: Determine the driver's emotional state based on the emotional state data that exceed the set emotion recognition threshold within each time window.

[0057] To improve the accuracy and reliability of emotion prediction, this invention introduces an emotion threshold adjustment mechanism based on a time window. This mechanism dynamically assesses the driver's emotional state by statistically analyzing the ratio of emotion prediction results within a given time window. This method evaluates the reliability of emotion prediction by analyzing the emotion prediction results within a specific time window and combining this with a set ratio threshold, ensuring a balance between accuracy and real-time performance in driver emotion recognition. The specific process is as follows:

[0058] 1) Emotion recognition time window.

[0059] In practical applications, emotional states exhibit strong temporal characteristics; that is, the emotions of the same driver may remain unchanged or change relatively smoothly within a short period. To enhance the stability of emotion recognition, this invention aggregates the emotion prediction results into time windows, with the length of each time window set to... t The driver's emotions are identified multiple times within a given time period, which is measured in seconds.

[0060] Set the prediction frequency of the emotion recognition model to per second. k Therefore, within a time window t Within seconds, a total of By recording the results of emotion recognition and conducting statistical analysis on these results, a more comprehensive reflection of the driver's emotional changes during that time period can be obtained.

[0061] .

[0062] 2) Threshold setting and judgment.

[0063] To improve the accuracy and reliability of emotion prediction, this invention statistically identifies the number of times a driver experiences negative emotions within each time window. It also calculates the ratio of the number of times an emotion is identified as negative to the total number of identifications. .

[0064] .

[0065] In this invention, the threshold is set as ,when Higher than If the prediction of negative emotions occurs within a given time window, it is considered highly reliable, indicating that the driver is in a negative emotional state and a warning should be issued; conversely, if... Below If the prediction is not reliable, no warning will be issued.

[0066] 3) Threshold adjustment and sensitivity.

[0067] Considering that different driving environments, individual differences among drivers, and varying ranges of emotional fluctuations may require different threshold settings, this invention will dynamically adjust the threshold according to the actual application scenario. By optimizing the threshold during the experiment, a balance can be achieved between prediction sensitivity and warning accuracy.

[0068] For example, in environments that place high demands on driver behavior (such as complex road conditions or high-intensity driving tasks), the threshold can be appropriately lowered to detect potential negative emotions earlier, thereby taking measures to help the driver maintain emotional stability; while in environments that place lower demands on driver behavior (such as simple road sections or low-risk scenarios), the threshold can be appropriately raised to avoid unnecessary warnings.

[0069] By employing this time window and threshold-based approach, the present invention can effectively reduce unnecessary warnings caused by accidental emotion prediction errors, while improving the sensitivity and accuracy of negative emotion state recognition. This method combines real-time performance with robustness, providing an efficient solution for emotion prediction in dynamic driving scenarios.

[0070] This invention also provides a driver emotion recognition system, comprising:

[0071] The driving style category acquisition module is used to acquire historical data on the driver's facial expression features, physiological features, and driving behavior features, cluster the historical data, and classify the driver's driving style into different categories based on the clustering results.

[0072] The emotion recognition model acquisition module is used to input the clustering result data corresponding to different driving style categories into the neural network model, train the neural network model, and obtain the emotion recognition model corresponding to different driving style categories respectively.

[0073] The emotion recognition module is used to determine the emotion recognition model corresponding to the driving style category of the driver who needs to be identified. Real-time data of the driver's facial expression features, physiological features and driving behavior features are input into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The driver's emotional state is determined based on the emotional state data that exceeds the set emotion recognition threshold within each time window.

[0074] The present invention also provides a computer device including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform a driver emotion recognition method.

[0075] The present invention also provides a computer-readable storage medium storing a computer program adapted for loading by a processor to execute a driver emotion recognition method.

[0076] The driver emotion recognition method provided by this invention has the following beneficial effects:

[0077] 1) In response to the problems of complex data and low accuracy in driver emotion monitoring technology, this invention comprehensively considers the driver's facial expression features, physiological features (electrocardiogram signals) and driving behavior features (vehicle driving parameters), extracts the optimal feature subset that can reflect the driving state, and constructs a real-time driver emotion monitoring model based on multi-source information fusion to achieve more accurate and real-time driver emotion recognition, thereby effectively preventing traffic accidents caused by driver emotional fluctuations.

[0078] 2) In view of the fact that existing emotion recognition methods do not take into account the individual differences of drivers, this invention performs cluster analysis on the historical data of drivers, divides drivers into four categories, and trains corresponding model parameters for different categories of drivers, thus fully taking into account the impact of individual differences of drivers on the accuracy of emotion recognition.

[0079] 3) This invention also introduces a time-window-based emotion threshold adjustment mechanism, which dynamically assesses the driver's emotional state by statistically analyzing the ratio of emotion prediction results within a time period. This method evaluates the reliability of emotion prediction by analyzing the emotion prediction results within a certain time window and combining this with a threshold setting. Within a certain time period, the driver is only identified as being in that emotion if the duration of a certain emotion exceeds the set threshold, effectively improving the stability of the identification.

[0080] The above-described embodiments are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited thereto. Any simple changes or equivalent substitutions of the technical solutions that can be obviously obtained by those skilled in the art within the scope of the technology disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for driver emotion recognition, characterized in that, Includes the following steps: Historical data on drivers' facial expression features, physiological features, and driving behavior features are obtained. The historical data is used as data reflecting physiological features. Principal component analysis is used to reduce the dimensionality of the data reflecting physiological features. The historical data after dimensionality reduction is clustered. Based on the clustering results, the drivers' driving styles are divided into different categories. The clustering results data corresponding to different driving style categories are input into the neural network model, and the neural network model is trained to obtain the emotion recognition model corresponding to different driving style categories. Determine the corresponding emotion recognition model for the driving style category of the driver who needs emotion recognition. Input the real-time data of the driver's facial expression features, physiological features and driving behavior features into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The driver's emotional state is determined based on the emotional state data exceeding the set emotion recognition threshold within each time window; The emotion recognition threshold is dynamically set based on the external traffic scenario; The real-time data of the driver's facial expression features, physiological features, and driving behavior features that need to be recognized are input into the corresponding emotion recognition model. Combined with the emotion recognition threshold dynamically set based on the external traffic scene, the driver's emotional state data is identified. The process includes the following steps: collecting real-time data of the driver's facial expression features, physiological features, and driving behavior features to be recognized; clustering the real-time data of facial expression features, physiological features, and driving behavior features; and determining the driver's driving style category based on the clustering results. Real-time data of the driver's facial expression features, physiological features, and driving behavior features are input into the emotion recognition model corresponding to the driver's driving style category to identify the driver's emotional state data in different time windows.

2. The driver emotion recognition method according to claim 1, characterized in that, Before clustering the historical data, the process also includes converting facial expression features into integer labels; and using principal component analysis to convert physiological features and driving behavior features into one-dimensional continuous variables.

3. The driver emotion recognition method according to claim 1, characterized in that, The physiological characteristics include driver heart rate and blood pressure changes, and the driving behavior characteristics include steering wheel angle, vehicle speed, and vehicle acceleration during driving.

4. The driver emotion recognition method according to claim 1, characterized in that, The neural network model includes an input layer, a fully connected layer, and an output layer. The fully connected layer is used to perform weighted and non-linear activation processing on the input data. The output layer uses the Softmax activation function to convert the output data of the fully connected layer into probability distributions of different emotion categories.

5. A driver emotion recognition system, characterized in that, include: The driving style category acquisition module is used to acquire historical data of drivers' facial expression features, physiological features, and driving behavior features. The historical data is used as data reflecting physiological features, and the principal component analysis method is used to reduce the dimensionality of the data reflecting physiological features. The historical data after dimensionality reduction is clustered, and the drivers' driving styles are divided into different categories based on the clustering results. The emotion recognition model acquisition module is used to input the clustering result data corresponding to different driving style categories into the neural network model, train the neural network model, and obtain the emotion recognition model corresponding to different driving style categories respectively; the emotion recognition threshold is dynamically set based on the external traffic scene. The emotion recognition module is used to determine the emotion recognition model corresponding to the driving style category of the driver who needs to be emotionally recognized. Real-time data of the driver's facial expression features, physiological features and driving behavior features are input into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The driver's emotional state is determined based on the emotional state data exceeding the set emotion recognition threshold within each time window; The emotion recognition threshold is dynamically set based on the external traffic scenario; The real-time data of the driver's facial expression features, physiological features and driving behavior features that need to be recognized are input into the corresponding emotion recognition model to identify the driver's emotional state data in different time windows. The steps include: collecting real-time data of the driver's facial expression features, physiological features and driving behavior features to be recognized; clustering the real-time data of facial expression features, physiological features and driving behavior features; and determining the driver's driving style category based on the clustering results. Real-time data of the driver's facial expression features, physiological features, and driving behavior features are input into the emotion recognition model corresponding to the driver's driving style category to identify the driver's emotional state data in different time windows.

6. A computer device, characterized in that, It includes a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to perform the driver emotion recognition method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to execute the driver emotion recognition method according to any one of claims 1-4.

Images