Driver emotion recognition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a driver emotion recognition method and device, equipment and a medium. The method comprises the steps of obtaining working condition information of a vehicle, and determining driving characteristics of a driver based on the working condition information; acquiring real-time road information, and determining the driving behavior of the driver based on the real-time road information and the driving characteristics; acquiring a face video of the driver, and determining expression features of the driver based on the face video; and determining an emotional state of the driver based on the driving behavior and / or the expression feature, and determining an emotional pacifying strategy corresponding to the driver based on the emotional state. According to the embodiment of the invention, the emotion state of the driver can be accurately determined, and the driving risk caused by the negative emotion of the driver is avoided, so that the driver is helped to keep a better driving state.

Description

Technical Field

[0001] This application relates to the technical field of emotion recognition, and particularly to a method and device for recognizing a driver's emotion, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularization of vehicles, there are more and more vehicles on the road, and road safety issues have become increasingly prominent. Drivers are affected by emotions during driving, resulting in unstable and impulsive driving behaviors, such as intense driving behaviors like sudden acceleration and sharp turns.

[0003] In the prior art, although there are some driver behavior monitoring devices, most of them focus on monitoring driving skills and behaviors, lacking the monitoring and recognition of the driver's physiological characteristics. The driver's emotional state has a direct impact on driving behaviors, including but not limited to emotional states such as anxiety, anger, and fatigue, as well as changes in heart rate. Therefore, there is an urgent need for a method to monitor and recognize the driver's driving behaviors and emotional states to improve driving safety and reduce traffic accidents. Summary of the Invention

[0004] To solve the above technical problems, embodiments of the present application provide a method and device for recognizing a driver's emotion, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the embodiments of the present application, a method for recognizing driving emotion is provided, including: obtaining the working condition information of the vehicle, and determining the driving characteristics of the driver based on the working condition information; obtaining the real-time road information, and determining the driving behavior of the driver based on the real-time road information and the driving characteristics; obtaining the facial video of the driver, and determining the expression characteristics of the driver based on the facial video; determining the emotional state of the driver based on the driving behavior and / or the expression characteristics, and determining the corresponding emotion soothing strategy for the driver based on the emotional state.

[0006] According to one aspect of the embodiments of the present application, the working condition information includes wheel speed sensing data and inertia measurement data, and the determining the driving characteristics of the driver based on the working condition information includes: determining the real-time vehicle speed of the vehicle based on the wheel speed sensing data and the inertia measurement data; determining the real-time acceleration value of the vehicle based on the real-time vehicle speed; obtaining the historical acceleration value of the vehicle, and calculating the acceleration change rate of the vehicle based on the historical acceleration value and the real-time acceleration value; determining the driving characteristics of the driver based on the real-time vehicle speed, the real-time acceleration value, and the acceleration change rate.

[0007] According to one aspect of the embodiments of the present application, determining the driving behavior of the driver based on the real-time road information and the driving characteristics includes: determining the road conditions based on the real-time road information, where the road conditions include road type and road geometric features; determining the driving behavior of the driver based on the road type, the road geometric features, and the driving characteristics.

[0008] According to one aspect of the embodiments of the present application, the real-time road information further includes traffic data, and the method further includes: determining traffic restriction parameters in the real-time environment based on the traffic data; determining road restriction parameters in the real-time environment based on the road type and the road geometric features; determining an acceleration change rate threshold of the vehicle based on the road restriction parameters and the traffic restriction parameters; if the acceleration change rate is greater than or equal to the acceleration change rate threshold, determining that the driver is in an aggressive driving behavior, and determining that the driver is in a negative emotional state based on the aggressive driving behavior.

[0009] According to one aspect of the embodiments of the present application, determining the emotional state of the driver based on the expression features includes: obtaining the expression features corresponding to each frame of the facial video to obtain an expression feature sequence arranged in time series; performing expression semantic recognition on the expression feature sequence to obtain the expression semantics of the driver; determining the emotional state of the driver based on the expression semantics.

[0010] According to one aspect of the embodiments of the present application, determining the expression features of the driver based on the facial video includes: obtaining the dynamic facial image of the driver based on the facial video; preprocessing the dynamic facial image and inputting the processed dynamic facial image into a preset face detection model; determining the expression features of the driver based on the face feature key points output by the preset face detection model.

[0011] According to one aspect of the embodiments of the present application, the method further includes: if any one of the driving behavior and the expression features indicates that the driver is in a negative emotional state, obtaining the environmental information inside the cockpit of the vehicle; controlling at least one of the fragrance system, the seat massage system, the seat ventilation system, and the music system inside the cockpit to be turned on based on the environmental information inside the cockpit.

[0012] According to one aspect of the embodiments of the present application, a driver emotion recognition device is provided, including: an acquisition module, configured to acquire the operating condition information of a vehicle and determine the driving characteristics of the driver based on the operating condition information; a first determination module, configured to acquire real-time road information and determine the driving behavior of the driver based on the real-time road information and the driving characteristics; a second determination module, configured to acquire the facial video of the driver and determine the expression characteristics of the driver based on the facial video; a soothing module, configured to determine the emotional state of the driver based on the driving behavior and / or the expression characteristics, and determine the corresponding emotion soothing strategy for the driver based on the emotional state.

[0013] According to one aspect of the embodiments of the present application, an electronic device is provided, including: one or more processors; a storage device, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, enable the electronic device to implement the driver emotion recognition method as described above.

[0014] According to one aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor of a computer, enable the computer to execute the driver emotion recognition method as described above.

[0015] According to one aspect of the embodiments of the present application, a computer program product is further provided, including a computer program, and when the computer program is executed by a processor, implement the steps in the driver emotion recognition method as described above.

[0016] In the technical solution provided by the embodiments of the present application, by combining the operating condition information of the vehicle, the real-time road information, and the facial video of the driver, the driving environment and state of the driver can be comprehensively perceived from multiple dimensions, thereby improving the accuracy of emotion recognition, and evaluating the driving behavior of the driver by combining the road information, thereby improving the effectiveness of driving behavior evaluation. On the other hand, the expression characteristics of the driver are also determined by combining the facial expressions of the driver, and then the emotional state of the driver can be accurately determined. Then, appropriate emotion soothing strategies can relieve the negative emotions of the driver, improve the comfort and pleasure during driving, and thus help the driver maintain a better driving state.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0018] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts. In the accompanying drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment for driver emotion recognition during driving shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flowchart of a driver emotion recognition method shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0022] Figure 4 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0023] Figure 5 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0024] Figure 6 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0025] Figure 7 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0026] Figure 8 is a flowchart of a driver emotion recognition method shown in another exemplary embodiment of the present application;

[0027] Figure 9 is a schematic diagram of a brief process for driver emotion recognition during driving in an exemplary application scenario;

[0028] Figure 10 is a block diagram of a driver emotion recognition device shown in an exemplary embodiment of the present application;

[0029] Figure 11 shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. Detailed implementation manners

[0030] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are only exemplary descriptions and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0033] As used in this application, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0034] First of all, it should be noted that when a driver is in an angry or tense emotional state, the human body will secrete hormones such as adrenaline, resulting in physiological reactions such as a rapid heartbeat, elevated blood pressure, and rapid breathing. These physiological changes will affect the driver's body coordination and reaction ability, making it difficult to accurately operate the vehicle during driving. For example, an angry driver may step on the brake or accelerator too hard, causing the vehicle to stop suddenly or accelerate rapidly; a tense driver may have trembling hands and feet, affecting the operation of the steering wheel and gear lever. Positive emotions such as pleasure and excitement can improve the driver's attention and alertness, making their observation of road conditions more sensitive, judgment more accurate, and reaction more rapid.

[0035] Negative emotions such as sadness, fear, anger, etc. can reduce a driver's attention and judgment, making it easy for them to be distracted and inattentive, and turning a blind eye to potential dangers. For example, a sad driver may be immersed in their own emotions and ignore the driving conditions of surrounding vehicles; a fearful driver may be overly nervous and make wrong judgments about normal traffic conditions. Among them, anger is one of the common negative emotions during driving, which may be caused by the uncivilized behavior of other drivers, traffic congestion and other factors. An angry driver may lose control of the vehicle and is more likely to make impulsive decisions, such as dangerous behaviors like speeding, forcefully changing lanes or overtaking.

[0036] See Figure 1 , Figure 1 is a schematic diagram of the implementation environment shown in an exemplary embodiment of the present application. As Figure 1 shown, during the driving process of the vehicle, the vehicle sends the vehicle's working condition information, real-time road information, and the driver's facial video to the server side 120 through the intelligent terminal 110. Furthermore, the vehicle's working condition information can be sent to the server side 120. After that, the server side 120 determines the driver's driving characteristics based on the vehicle's working condition information, and determines the driver's driving behavior by combining the real-time road information and driving characteristics. At the same time, the server side 120 also determines the driver's expression characteristics based on the driver's facial video. Then, the server side 120 can determine the driver's emotional state through the driver's driving behavior and expression characteristics, and further determine the driver's emotion soothing strategy according to the emotional state. The server side 120 feeds back the emotion soothing strategy to the intelligent terminal 110, and the terminal implements the intervention through interactive devices such as in-vehicle audio, screen display, or vibrating seats. Thus, while realizing the monitoring of the driver's emotions during driving, it can also soothe the driver's emotions in a timely manner and improve driving safety.

[0037] Among them, Figure 1 the intelligent terminal 110 shown can be any terminal device that supports installing and collecting vehicle-related information, such as a smart phone, in-vehicle computer, tablet computer, laptop computer, or wearable device, but is not limited thereto. Figure 1 The server side 120 shown can be, for example, an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. There is no limitation here either.

[0038] In the prior art, although there are some driver behavior monitoring devices, most of them focus on monitoring driving skills and behaviors, lacking the monitoring and identification of drivers' physiological characteristics. The emotional state of a driver has a direct impact on driving behavior, including but not limited to emotional states such as anxiety, anger, fatigue, and changes in heart rate. Therefore, there is an urgent need for a method to detect a driver's driving behavior and emotional state to improve driving safety and reduce traffic accidents.

[0039] The problems pointed out above are generally applicable in general travel scenarios. It can be seen that if the emotions of drivers cannot be monitored and appeased in multiple dimensions, the driving risk will increase. To solve these problems, embodiments of the present application respectively propose a driver emotion recognition method, a driver emotion recognition device, an electronic device, a computer-readable storage medium, and a computer program product. These embodiments will be described in detail below.

[0040] Please refer to Figure 2 , Figure 2 FIG. is a flowchart of a driver emotion recognition method shown in an exemplary embodiment of the present application. This method can be applied to Figure 1 the implementation environment shown, and is specifically executed by the server 120 in this implementation environment. It should be understood that this method can also be applicable to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment applicable to this method.

[0041] As Figure 2 shown, in an exemplary embodiment, the driver emotion recognition method may include steps S210 to S240, which are introduced in detail as follows:

[0042] Step S210, obtain the working condition information of the vehicle, and determine the driving characteristics of the driver based on the working condition information.

[0043] Exemplarily, during the vehicle driving process, the working condition information of the vehicle can be obtained through corresponding devices. For example, vehicle operation information such as the vehicle speed, steering wheel angle, steering amplitude, acceleration, fuel consumption, braking force, and positioning. Then, key indicators that can reflect the driving characteristics are extracted from the working condition information. These indicators should be able to comprehensively and accurately describe the driving characteristics of the driver, such as the acceleration change rate, braking force, steering stability, etc.

[0044] Step S220, obtain the real-time road information, and determine the driving behavior of the driver based on the real-time road information and the driving characteristics.

[0045] Real-time road information of the vehicle can be obtained through devices such as on-vehicle cameras, lidar, and ultrasonic radars. Among them, the road information includes road curvature, road slope, road traffic flow, etc. Furthermore, the driving behavior of the driver can be evaluated by combining the real-time road information where the vehicle is located and the driving characteristics of the driver, and it can be determined whether the driver has dangerous driving behaviors, aggressive driving behaviors, etc.

[0046] Exemplarily, road information can be collected in real time through high-precision maps, sensors such as radars and cameras. Real-time road information sent from a traffic platform can also be received through a mobile communication network or vehicle networking technology. Then, based on the real-time road information, the current road conditions can be analyzed, such as slippery road surface, low visibility, construction areas, etc. Considering the road type (highway, urban street, rural road) and traffic flow, potential influencing factors are evaluated. Then, the driving behavior of the driver is evaluated by combining the real-time road information and the driving characteristics of the driver.

[0047] Step S230, obtain the facial video of the driver, and determine the expression characteristics of the driver based on the facial video.

[0048] On the other hand, it is also possible to obtain the facial video of the driver inside the vehicle cockpit. This facial video can be the facial video of the driver within a short time period. Then, the expression characteristics of the driver can be determined by analyzing this facial video. In some feasible embodiments, the expression changes of the driver can further be determined based on the facial video to more accurately determine the emotional state of the driver.

[0049] Exemplarily, install a high-definition camera at a suitable position inside the vehicle (such as above the dashboard or near the rearview mirror) to ensure that the driver's face can be clearly captured. Use a face detection algorithm (perform face detection on each frame of the video, determine the position and size of the face according to the detection results, and prepare for subsequent expression feature extraction. After detecting the face, use a feature point localization algorithm (such as the regression tree combination method) to locate key feature points in the face area. These feature points usually include multiple points in parts such as eyes, nose, and mouth. The key feature points are mainly used to describe the shape and texture of the face. Then, according to the located key feature points, extract feature parameters related to expressions. These parameters may include the opening degree of the eyes, the opening and closing degree of the mouth, the bending degree of the eyebrows, etc. Then, feature extraction methods such as the optical flow method, active appearance model method, or differential image method can be used to obtain these parameters. Input the extracted expression characteristics into a classifier (such as an artificial neural network, support vector machine, hidden Markov model, etc.) for classification. The classifier will match and identify the input expression characteristics according to the preset expression categories (such as happy, sad, angry, surprised, frightened, disgusted, etc.). Finally, the classifier outputs the identified expression category as the current expression characteristics of the driver.

[0050] Step S240: Determine the driver's emotional state based on the driving behavior and / or facial expression features, and determine the corresponding emotional soothing strategy for the driver based on the emotional state.

[0051] Continuing with the above embodiments, the driving behavior of the driver can be determined according to the vehicle's operating condition information and real-time road information, and the facial expression features of the driver can be determined through the driver's facial video. Then, the emotional state of the driver can be determined according to the driving behavior and facial expression features of the driver, and further, the emotional soothing strategy for the driver can be determined according to the emotional state of the driver. Among them, for a nervous or angry driver, voice prompts can be provided to advise them to stay calm, play soothing music, or adjust the in-vehicle lighting and temperature to create a relaxing atmosphere. For a fatigued driver, suggestions for refreshing drinks or food can be provided, exciting music can be played, or the vehicle driving mode can be automatically adjusted to reduce the driving difficulty. At the same time, the soothing strategy should consider the driver's personal preferences and historical data to achieve a more accurate and personalized soothing effect.

[0052] Optionally, in some realizable embodiments, if the driver's driving behavior is evaluated as an aggressive driving behavior, a dangerous driving behavior, etc. that may pose a road safety risk, for example, collect the vehicle's operating condition information (such as vehicle speed, acceleration, braking frequency, etc.) and real-time road information (such as traffic conditions, road types, etc.). Then, these operating condition information can be analyzed using an algorithm or model. If aggressive driving or dangerous driving behaviors are identified, such as speeding, hard braking, frequent lane changes, non-compliance with traffic rules and other driving behaviors, an immediate and clear prompt can be sent to the driver through the in-vehicle voice system, pointing out the danger of their driving behavior and suggesting a safer driving method, and safety driving suggestions can be displayed on the head-up display system on the in-vehicle display screen, such as reducing the vehicle speed, maintaining a safe distance, smoothly turning on the turn signal, or automatically playing soothing music to help the driver relax and reduce tension. If the driver does not respond to the soothing strategy or the driving behavior remains dangerous, the vehicle can be automatically controlled to decelerate to reduce potential risks. When necessary, the vehicle can be automatically controlled to smoothly turn to avoid collisions with obstacles or other vehicles.

[0053] Optionally, in some other implementable embodiments, if it is determined that the driver is in a negative emotional state through the driver's facial expression features. For example, if negative emotions, such as anger, sadness, fear, disgust, or fatigue, are accurately recognized from the driver's facial video. These emotions are usually accompanied by specific facial muscle movements and expression features, such as frowning, downturned corners of the mouth, and flickering eyes. Then, the in-vehicle voice system can provide immediate and gentle feedback to the driver, reminding them to pay attention to the emotional changes and suggesting relaxation measures. Or, automatically adjust the in-vehicle music, lighting, and temperature to create a comfortable and relaxing atmosphere. For example, play soothing music, reduce the in-vehicle brightness, or adjust the air conditioner temperature to help the driver relieve negative emotions.

[0054] In some embodiments of the present application, by combining the vehicle condition information, real-time road information, and the driver's facial video, the driving environment and state of the driver can be comprehensively perceived from multiple dimensions, thereby improving the accuracy of emotion recognition. And by combining the road information to evaluate the driver's driving behavior, the effectiveness of driving behavior evaluation can be improved. On the other hand, the facial expression of the driver is also combined to determine the driver's expression features, and then the emotional state of the driver can be accurately determined. Then, appropriate emotion soothing strategies can relieve the driver's negative emotions, improve the comfort and pleasure during driving, and thus help the driver maintain a better driving state.

[0055] Further, based on the above embodiments, please refer to Figure 3 , in one exemplary embodiment provided by the present application, the above vehicle condition information includes wheel speed sensing data and inertia measurement data. The specific implementation process of determining the driver's driving characteristics based on the vehicle condition information may further include steps S310 to S340, which are introduced in detail as follows:

[0056] Step S310, determine the real-time vehicle speed based on the wheel speed sensing data and the inertia measurement data.

[0057] It should be noted that a wheel speed sensor is a device that detects the rotational speed of a wheel, and its data is widely used in important functions such as vehicle speed calculation, Anti-Lock Braking System (ABS), and Electronic Stability Program (ESP). The wheel speed sensor continuously monitors and sends the rotational speed information of the wheel to the Electronic Control Unit (ECU), usually in the form of pulses, and each pulse represents a specific rotation angle of the wheel. After receiving these pulses, the ECU calculates the number of pulses per unit time, which directly reflects the rotational speed of the wheel. Since the circumference of the wheel is known (usually determined by the tire diameter, i.e., tire diameter × π), the ECU can obtain the distance traveled by the vehicle per unit time by multiplying the wheel rotational speed by the circumference, that is, the vehicle speed. In addition, considering unit conversion, the calculated vehicle speed is usually in meters per second or feet per second, and then converted to kilometers per hour or miles per hour as needed. Since the wheel speed sensor data may be affected by factors such as tire wear, air pressure changes, and wheel slippage, in order to improve the accuracy of vehicle speed and acceleration measurement, the inertia measurement data of the vehicle can be introduced. Among them, the Inertial Measurement Unit (IMU) is an inertial sensor module used to measure the acceleration, angular velocity, and (in some cases) magnetic field strength of an object.

[0058] Optionally, in the vehicle speed recognition method, the real-time vehicle speed can be calculated through the data collected by the wheel speed sensor and the inertial measurement unit to make the obtained real-time vehicle speed more accurate. Among them, since the wheel speed sensor has high accuracy at low speeds but fails when slipping or idling. And the Inertial Measurement Unit (IMU) has high accuracy in a short time, but will drift due to integration error during long-term use. Therefore, the speed can be calculated by weighting the combination of the wheel speed sensor and the Inertial Measurement Unit (IMU) to make the obtained speed value more accurate.

[0059] Exemplarily, the wheel speed sensor can calculate the speed through the following formula:

[0060] v wheel (t) = r·ω(t)

[0061] where, v wheel (t) is the speed calculated by the wheel speed sensor, r is the wheel radius, and ω(t) is the angular velocity of the wheel.

[0062] Then, the Inertial Measurement Unit (IMU) calculates the speed by integrating the acceleration

[0063] V IMU (t) = VIMU (t - 1)+a(t)·Δt

[0064] Wherein, V IMU (t) is the speed calculated by the IMU, V IMU (t - 1) is the speed calculated by the IMU at the previous moment, a(t) is the acceleration measured by the IMU, and Δt is the time interval. After combining the dynamic weighting of the wheel speed sensor and the IMU, the real-time speed calculation result is obtained.

[0065] v fused (t) = w·v wheel (t)+(1 - w)·v IMU (t)

[0066]

[0067] Where v fused (t) is the weighted real-time speed, w is the weight of the wheel speed sensor, and the value range is 0 to 1. is the error variance of the wheel speed sensor. is the error variance of the IMU.

[0068] Step S320, determining the real-time acceleration value of the vehicle based on the real-time vehicle speed;

[0069] Step S330, obtaining the historical acceleration value of the vehicle, and calculating the acceleration change rate of the vehicle based on the historical acceleration value and the real-time acceleration value;

[0070] Specifically, the real-time vehicle speed of the vehicle is obtained through the vehicle's sensor system (such as wheel speed sensor, positioning system, etc.). The vehicle speed data is usually in units of meters per second (m / s) or kilometers per hour (km / h). Acceleration is the rate of change of speed with time. Therefore, to calculate the real-time acceleration, the change in vehicle speed within a short period of time (such as Δt seconds) needs to be known. The acceleration can be approximately calculated by the difference method: a = (v2 - v1) / Δt, where v2 and v1 are the vehicle speeds before and after Δt seconds respectively, and a is the acceleration. To improve the accuracy, more complex numerical differentiation methods, such as the central difference method or the backward difference method, can be used. Obtain the acceleration values of the vehicle within a past period of time from the vehicle's sensor system or data storage. These historical acceleration values should be arranged in chronological order for subsequent analysis. The acceleration change rate is the rate of change of acceleration with time, and can be approximated by calculating the difference between the real-time acceleration and the historical acceleration.

[0071] Exemplarily, a time window can be selected (such as the most recent n acceleration data points), and the average value or weighted average value of the accelerations at these points can be calculated as a representative value of the historical acceleration. Then, the real-time acceleration value is subtracted by this representative value of the historical acceleration, and then divided by the time interval (i.e., the time length covered by n data points) to obtain an approximate value of the acceleration change rate. It should be noted that since the acceleration data may be affected by noise and errors, in practical applications, filtering techniques (such as Kalman filtering, moving average filtering, etc.) may be adopted to smooth the data and improve the accuracy of the acceleration change rate calculation.

[0072] Optionally, in some realizable embodiments, the driving behavior score of hard acceleration and hard deceleration operations within a unit time can be used as the determination basis for identifying aggressive driving behavior. It is determined that the vehicle is in the electronic stability program (ESP) on state and non-race track mode, and the detection of this acceleration change rate value is not performed when the ESP or race track mode is turned off. First, the acceleration a(t) at time t is calculated, and then the change rate j(t) of the acceleration is calculated. The calculation method is as follows:

[0073]

[0074] where, v fused (t) is the weighted real-time speed at time t, Δt is the time interval, a(t) is the acceleration at time t, and j(t) is the change rate of the acceleration at time t.

[0075] Step S340, determining the driving characteristics of the driver based on the real-time vehicle speed, real-time acceleration value, and acceleration change rate.

[0076] Specifically, the driving characteristics of the driver can be evaluated according to the real-time vehicle speed of the vehicle, the real-time acceleration value of the vehicle, and the driving change rate. For example, if the real-time vehicle speed remains within a relatively stable range with little fluctuation, it indicates that the driver has a relatively stable control of the speed and pays attention to driving safety; if the real-time vehicle speed changes rapidly and frequently, such as hard acceleration and hard deceleration, it may indicate that the driver has a more aggressive driving style and the control of the speed is not stable enough. If the real-time vehicle speed changes greatly and both the acceleration and the acceleration change rate are high, it may indicate that the driver's driving style is relatively aggressive. That is to say, when analyzing the driving characteristics of the driver, multiple parameters such as the real-time vehicle speed, real-time acceleration value, and acceleration change rate should be comprehensively considered to avoid one-sided judgment caused by a single parameter.

[0077] Optionally, following the above embodiments, after calculating the acceleration change rate j(t) of the vehicle, it is determined that the vehicle is in a hard acceleration or hard deceleration behavior. When |j(t)| > j threshold at this time, it is determined that the current moment is an aggressive driving moment, where, j thresholdis a set acceleration change rate threshold. If the acceleration change rate value is greater than the acceleration change rate threshold, it indicates that the driver has aggressive driving behavior.

[0078] In some embodiments of the present application, the wheel speed sensor can accurately measure the rotational speed of the wheel. Combining with the diameter of the wheel, the real-time vehicle speed can be calculated. And the inertia measurement data can provide additional information about the vehicle's motion state, which helps to correct the influence of factors such as tire wear, air pressure change, and wheel slip on the wheel speed sensing data, thereby improving the accuracy of vehicle speed and acceleration measurement. Furthermore, based on the real-time vehicle speed, real-time acceleration value, and acceleration change rate, the driving characteristics of the driver can be accurately determined.

[0079] Further, based on the above embodiments, please refer to Figure 4 , in an exemplary embodiment provided by the present application, the specific implementation process of determining the driving behavior of the driver based on the real-time road information and driving characteristics may further include step S410 and step S420, which are introduced in detail as follows:

[0080] Step S410, determining the road condition based on the real-time road information, where the road condition includes road type and road geometric features;

[0081] Step S420, determining the driving behavior of the driver based on the road type, road geometric features, and driving characteristics.

[0082] Specifically, the real-time road information may include the type of the road, such as urban road, urban-rural combined road, mountain road, highway, etc. Since different types of roads have different impacts on the driving behavior of the driver. For example, urban roads may be more congested, requiring the driver to frequently brake and accelerate; while mountain roads may be more rugged, requiring the driver to operate the vehicle more carefully. Among them, the road geometric features in the real-time road information include the width, curvature, slope, road surface condition, etc. These road geometric features will affect the driving stability of the vehicle and the driving difficulty of the driver. For example, a road with a larger curvature may require the driver to turn the steering wheel more frequently; a road with a larger slope may affect the braking and accelerating performance of the vehicle. After determining the road condition and driving characteristics, the driving behavior of the driver can be inferred based on this information. For example, if the road is a congested urban road and the driving characteristics of the driver are manifested as frequent braking and accelerating, it can be inferred that the driver may be experiencing congestion and taking corresponding driving behaviors. Another example, if the road is a rugged mountain road and the driving characteristics of the driver are manifested as careful operation of the vehicle, it can be inferred that the driver is adjusting the driving behavior according to the road geometric features, thereby avoiding misjudgment of the driving behavior of the driver and improving the accuracy of the evaluation of the driving behavior of the driver.

[0083] In some embodiments of the present application, by combining driving characteristics with road conditions, the driving behavior of a driver under different road conditions can be analyzed more accurately. By continuously learning and adapting to different road conditions and driving characteristics, misjudgment of the driver's driving behavior is avoided, and the accuracy of evaluating the driver's driving behavior and emotional state is improved.

[0084] Further, based on the above embodiments, please refer to Figure 5 , in one exemplary embodiment provided by the present application, the above real-time road information further includes traffic data. The specific implementation process of the above driver emotion recognition method may further include steps S510 to S540, which are introduced in detail as follows:

[0085] Step S510, determining traffic restriction parameters in the real-time environment based on traffic data;

[0086] Step S520, determining road restriction parameters in the real-time environment based on the road type and road geometric features;

[0087] Step S530, determining the acceleration change rate threshold of the vehicle based on the road restriction parameters and traffic restriction parameters;

[0088] Step S540, if the acceleration change rate is greater than or equal to the acceleration change rate threshold, determining that the driver is in an aggressive driving behavior, and determining that the driver is in a negative emotional state based on the aggressive driving behavior.

[0089] Exemplarily, the traffic restriction parameters are completed based on the traffic data in the real-time environment. Among them, the traffic data may include traffic flow, road congestion, accident reports, the number of traffic lights, etc. These data help to understand the current road traffic conditions, so as to determine some traffic-based restriction parameters. Among them, the road restriction parameters are determined based on the type of road (such as highway, urban road, rural path, etc.) and the geometric features of the road (such as curve radius, slope, road surface condition, etc.). These parameters reflect the restrictive conditions of the road itself on vehicle driving. Then, by combining the above traffic restriction parameters and road restriction parameters, the reasonable and safe acceleration change rate range of the vehicle in the current real-time road environment can be calculated, and thus a threshold is set. When the acceleration change rate of the vehicle is greater than or equal to this threshold, it is considered that the driver may be performing aggressive driving behavior, and based on this aggressive driving behavior, it can be further inferred that the driver may be in a negative emotional state. When it is detected that the acceleration change rate of the vehicle exceeds this standard, not only can the aggressive driving behavior of the driver be identified, but also the possible negative emotional state hidden behind this behavior can be further inferred.

[0090] Optionally, in some feasible embodiments, the behavior score of rapid acceleration and rapid deceleration operations per unit time is calculated. In this embodiment, when the score value is greater than the preset score threshold, it is considered that the driver has adopted aggressive driving behavior in the approaching driving section. The expression is as follows:

[0091]

[0092] where S is the aggressive driving behavior score, N is the number of detected rapid accelerations and rapid decelerations, j i is the acceleration change rate of the i-th rapid acceleration and rapid deceleration, j threshold is the set acceleration change rate threshold.

[0093] To make j threshold meet the settings of different working conditions during the driving of the vehicle, this embodiment designs an adaptive threshold expression:

[0094] j threshold = α·σ j + β·μ j

[0095] where σ j is the standard deviation of the acceleration change rate, μ j is the mean value of the acceleration change rate, and this value is actually calibrated by a preset standard driving distance.

[0096] Finally, to make the recognition of aggressive driving behavior adapt to different working conditions, this embodiment designs a method for dynamically controlling the α and β parameters. The expression is as follows:

[0097] α = α0 + k1·Δs + k2·n light + k3·Δv

[0098] β = β0 + k4·Δs + k5·n light + k6·Δv

[0099] where α0 and β0 are the basic weights; Δs is the absolute value of the slope change per unit time, given by the slope sensor; n light is the number of traffic lights recognized per unit time; Δv is a piecewise function. When the current speed exceeds the speed limit, Δv is the difference between the current speed and the speed limit. Otherwise, Δv is equal to 0; k1, k2, k3, k4, k5, k6 are adjustment coefficients used to control the influence degree of different working conditions on α and β. Among them, α is the weight parameter of the standard deviation of the acceleration change rate determined according to the road limit parameters, and β is the weight parameter of the mean value of the acceleration change rate determined according to the road limit parameters.

[0100] Substituting α and β into the basic formula, the acceleration change threshold of the vehicle under the current road can be obtained, j threshold= (α0 + k1·Δs + k2·n light + k3·Δv)·σ j + (β0 + k4·Δs + k5·n light + k6·Δv)·μ j

[0101] Furthermore, the threshold of the acceleration change rate of the vehicle can be adjusted according to the real-time traffic condition information, so that the identified emotional state of the driver is more accurate.

[0102] In this embodiment, by combining traffic data, road type and geometric features, the traffic restriction parameters and road restriction parameters in the real-time environment can be more comprehensively understood. These factors act together on the driving process of the vehicle and affect the driving behavior of the driver. Based on the above restriction parameters, the threshold of the acceleration change rate of the vehicle can be set more precisely. This threshold can reflect the acceleration change range required for the vehicle to drive safely and smoothly under different road and traffic conditions. By real-time monitoring the acceleration change rate of the vehicle and comparing it with the set threshold, it can accurately identify whether the driver is in aggressive driving behavior. This identification method is more accurate and reliable than the traditional identification method based on a single factor (such as vehicle speed, acceleration, etc.).

[0103] Further, based on the above embodiment, please refer to Figure 6 , in an exemplary embodiment provided by the present application, the specific implementation process of determining the emotional state of the driver based on the facial expression features may further include steps S610 to S630, which are introduced in detail as follows:

[0104] Step S610, obtain the facial expression features corresponding to each frame of the facial video to obtain a sequence of facial expression features arranged in time series;

[0105] Step S620, perform facial expression semantic recognition on the sequence of facial expression features to obtain the facial expression semantics of the driver;

[0106] Step S630, determine the emotional state of the driver based on the facial expression semantics.

[0107] Specifically, due to the complex lighting conditions in the cockpit environment (such as strong light, alternating light and dark when entering and exiting tunnels, and backlight), visible light camera sensors are easily interfered with, resulting in a decrease in face recognition accuracy. Infrared cameras have their unique spectral characteristics and can achieve an automatic fill-light function. Therefore, the recognition images in the cockpit environment are stable. Therefore, in this embodiment, a high-resolution infrared camera can be used for dual-modal imaging in the infrared band (Infrared Band IR) and visible light (Red, Green, Blue RGB). The visible light (RGB) band is used to monitor the environmental light intensity, and the infrared (IR) band is used to collect the user's facial video. The visible light (RGB) band image is obtained at a low frequency. Since the RGB band is only used for environmental light intensity monitoring (not real-time video requirements), the sampling frequency can be reduced to 1 frame per second (1 fps) to reduce the computational load. Here, fps (Frames Per Second) is an index used to measure the smoothness of the video; the IR band needs to maintain a high frame rate (such as 30 fps) to continuously capture facial dynamics. The image is converted into the HSV color representation format through the color space. Among them, HSV (Hue, Saturation, Value) is a color space model. Among them, V represents the brightness of the color in the HSV color model. The average value of the image on the V channel is extracted as the environmental light intensity, and the power of the infrared fill-light module is dynamically adjusted according to the environmental light intensity.

[0108] Specifically, extracting images frame by frame from the facial video usually involves decoding the video file into a series of individual image frames. Then, for each frame of the image, face recognition technology and expression recognition algorithms are used to identify facial key points and expression features. These features may include information such as the shape, position, and movement of parts such as eyes, mouth, eyebrows, and cheeks. After that, the expression features extracted from each frame are arranged in the time sequence in the video to form an expression feature sequence. This expression feature sequence reflects the change of the driver's facial expression over time. Finally, the extracted expression features can be mapped to predefined expression semantic labels, which may include basic emotions such as happiness, sadness, anger, surprise, fear, disgust, or more complex emotion combinations.

[0109] Then, a trained machine learning model (such as a convolutional neural network, support vector machine, etc.) can be used to classify the facial expression features to identify the semantic meaning of the facial expression corresponding to each frame. Considering the temporal sequence of the facial expression feature sequence, a sequence processing model (such as a recurrent neural network, long short-term memory network, etc.) may also be needed to capture the temporal dependence relationship between the facial expression features, so as to more accurately identify the semantic meaning of the entire facial expression sequence. Based on the identified semantic meaning of the facial expression, the emotional state of the driver is classified into one of the predefined emotional categories, and further evaluate whether the emotional state of the driver is positive, negative or neutral, and whether certain measures need to be taken (such as reminding the driver to rest, adjusting the driving behavior, etc.).

[0110] In some embodiments of the present application, by extracting facial expression features from each frame of the facial video, the subtle changes in the facial expressions of the driver can be captured. This fine-grained feature extraction method helps to improve the accuracy of emotional state recognition, and by performing facial expression semantic recognition on the facial expression feature sequence, the facial expression can be converted into specific emotional labels, making the judgment of the driver's emotional state more intuitive and clear.

[0111] Further, based on the above embodiments, please refer to Figure 7 , in one exemplary embodiment provided by the present application, the specific implementation process of determining the facial expression features of the driver based on the facial video may further include steps S710 to S730, which are introduced in detail as follows:[[]]

[0112] Step S710, obtaining a dynamic facial image of the driver based on the facial video;

[0113] Step S720, preprocessing the dynamic facial image and inputting the processed dynamic facial image into a preset face detection model;

[0114] Step S730, determining the facial expression features of the driver based on the key points of the facial features output by the preset face detection model.

[0115] Optionally, enable the HDR mode to enhance details when sufficient visible light is available; rely on the infrared camera in low light / backlight conditions. That is, the dynamic range of imaging can be enhanced through the High-Dynamic Range (HDR) mode. For example, a dynamic facial image of the driver can be obtained in a facial video. Among them, the HDR mode can significantly improve the dynamic range of the video / image, enabling clear presentation of details in both bright and dark areas. Then, the obtained dynamic facial image can be further preprocessed, where the preprocessing includes image grayscaling, noise removal, image enhancement, data normalization, etc. Then, the preprocessed dynamic facial image is input into a preset face detection model, and the preset face detection model can be the YOLOv5-face detection model. Among them, YOLOv5 (You Only Look Once version 5) is a popular computer vision algorithm mainly used for object detection, and YOLOv5-Face is a face detection model improved based on the YOLOv5 object detection algorithm.

[0116] Since the static features of each frame of the face were obtained during the previous frame-by-frame analysis, in order to improve the accuracy of expression recognition, it is also necessary to collect the dynamic features of each frame. Here, the dynamic features refer to information such as the position change, speed, direction, and angle of the corresponding key points of each frame of the face. YOLOv5-Face is used to perform two-stage key point detection: first, locate the face area (coarse-grained), and then accurately locate the internal (eyebrows / eyes, etc.) and contour key points (fine-grained) through a cascaded network. This algorithm parallelly uses two cascaded convolutional neural networks for key point detection, splitting the face key points into internal key points and contour key points. The internal key points include a total of 5 key points such as eyebrows, eyes, nose, and mouth, and the contour key points are 17 key points related to the face contour. After obtaining the key points of each frame, the dynamic features of the face can be extracted using the optical flow method or the trajectory extraction method, and the static facial features and dynamic facial features are fused into the final facial feature sequence. (The dynamic features here can refine the expression semantics and also include micro-expression semantics). After obtaining the facial feature sequence, a temporal analysis model is used to extract the temporal features of the facial feature sequence and recognize the semantics. Here, the Long Short-Term Memory (LSTM) is used to capture the temporal dependence, and then the attention mechanism of the Transformer neural network is combined to focus on the key frames to enhance the micro-expression recognition ability.

[0117] Exemplarily, consecutive frames are intercepted from a video containing the driver's face. These frames constitute a sequence of dynamic facial images of the driver. This process generally involves video decoding and frame extraction techniques. To improve the accuracy of subsequent face detection and feature extraction, it may be necessary to enhance the intercepted facial images, which includes adjusting the brightness, contrast, sharpness, etc. of the images to ensure that the image quality meets the analysis requirements. Then, techniques such as filters are used to remove noise in the images to reduce interference with face detection and feature extraction. Next, parameters such as the size and resolution of the images are unified to facilitate subsequent processing and analysis. Finally, a trained face detection model is selected. This model can accurately locate the face region in the image. The trained face detection model is usually a deep learning model, such as a Convolutional Neural Network (CNN), etc. The preprocessed sequence of dynamic facial images is input into the preset face detection model. The model will analyze the images frame by frame, detect and locate the face region. Once the face region is detected, the model will further extract the key points of the face features. These key points usually include the specific position information of parts such as eyes, mouth, eyebrows, nose, etc. Based on the extracted key points of the face features, the facial expression features of the driver are analyzed. This involves calculating geometric relationships such as the distances and angles between the key points, as well as their changes over time. The expression features of each frame of the image are arranged in chronological order to construct a sequence of expression features. This sequence reflects the changes in the driver's facial expressions over time and can be used for further expression recognition or emotion analysis. The selected face detection model and feature extraction algorithm need to have high accuracy to ensure the reliability of subsequent analysis. Sliding window analysis can be performed on the expression features of consecutive N frames (such as 30 frames), and the LSTM is used to capture the instantaneous changes of micro-expressions (such as the twitching of the corners of the mouth within 0.5 seconds). When processing real-time video streams, the real-time requirements of the algorithm need to be considered to ensure that face detection and feature extraction can be completed within a reasonable time. And the algorithm needs to be able to adapt to factors such as different lighting conditions, facial occlusions, and head postures to ensure accurate face detection and expression feature extraction in various environments.

[0118] In some embodiments of the present application, through the combined use of preprocessing and the face detection model, it is possible to handle facial image recognition tasks in complex environments with different lighting, angles, occlusions, etc. This enhances the robustness of face recognition and enables it to work stably in different scenarios. This helps to more comprehensively analyze the facial expression features of the driver and improve the accuracy of recognition.

[0119] Further, based on the above embodiments, please refer to Figure 8, in one exemplary embodiment provided by the present application, the specific implementation process of determining the emotion soothing strategy corresponding to the driver based on the emotion state may further include step S810 and step S820, which are introduced in detail as follows:

[0120] Step S810, if any one of the driving behavior and the facial expression features indicates that the driver is in a negative emotion state, obtain the environmental information inside the vehicle cockpit;

[0121] Step S820, based on the environmental information inside the cockpit, control at least one of the fragrance system, the seat massage system, the seat ventilation system, and the music system inside the cockpit to be turned on.

[0122] Specifically, if it is determined according to the driver's driving behavior that the driver exhibits intense driving behaviors such as sudden braking, sudden acceleration, and frequent steering wheel operations, it is considered that the driver is in a negative emotion state. Then, the environmental information inside the vehicle cockpit can be obtained to turn on at least one of the fragrance system, the seat massage system, the seat ventilation system, and the music system inside the cockpit through the environmental information, thereby soothing the driver's negative emotion state. Or, if it is determined according to the driver's facial expression semantics that the driver is in a negative emotion state (such as anger, sadness, fear, etc.), the environmental information inside the cockpit, such as temperature, humidity, air quality, light intensity, etc., can be obtained through sensors or other devices. Then, the target comfort conditions can be determined based on the driver's historical data on this vehicle. The target comfort conditions are obtained based on the user's historical usage data of the above four systems on this vehicle. Under the target comfort conditions, control at least one or more of the fragrance system, the seat massage system, the seat ventilation system, and the music system including but not limited to the above.

[0123] Exemplarily, devices such as cameras and sensors can be used to continuously monitor the driver's driving behavior (such as sudden braking, sudden acceleration, and frequent steering wheel operations) and facial expression features (such as frowning, excessive eye closure, and tense facial muscles). These data are analyzed in real time to identify whether the driver is in a negative emotion state. Then, based on the obtained environmental information and the preset comfort conditions, the system intelligently controls multiple systems inside the cockpit to improve the driver's emotion state. The controllable systems include but are not limited to: Fragrance system: Releases fragrance to improve the atmosphere inside the cockpit. Seat massage system: Provides seat massage to relieve the driver's physical fatigue. Seat ventilation system: Provides seat ventilation in hot weather to increase comfort. Music system: Plays relaxing or pleasant music to improve the driver's mood.

[0124] In this embodiment, when it is detected that the driver is in a negative emotional state, intervention measures can be immediately taken to relieve the driver's negative emotions by adjusting the environment in the cockpit. Moreover, by turning on the fragrance system, seat massage system, seat ventilation system, and music system, a comfortable experience can be provided for the driver from multiple sensory levels such as smell, touch, and hearing, which helps to quickly improve the driver's emotional state.

[0125] Figure 9 It is a schematic flowchart of the brief process for driver emotion recognition during vehicle driving in an exemplary application scenario. In the application scenario shown in FIG. 9, during vehicle driving, the working condition information of the vehicle is obtained, where the working condition information includes the real-time vehicle speed and the real-time acceleration value of the vehicle. Furthermore, the driving characteristics of the driver can be determined based on the real-time vehicle speed and the real-time acceleration value of the vehicle. Then, in combination with the real-time road information and the driving characteristics, the driving behavior of the driver is determined. If the driving behavior of the driver shows aggressive driving behavior, it can be determined that the driver is in a negative emotional state based on the aggressive driving behavior, and then a corresponding emotion soothing strategy can be generated. On the other hand, at the same time, the facial video of the driver in the vehicle cockpit is obtained, and then the expression characteristics of the driver can be determined from the facial video. The expression semantics of the expression characteristics are recognized, and the emotion state of the driver is determined based on the expression semantics. If the emotion state of the driver is characterized as being in a negative emotional state, a corresponding emotion soothing strategy can be generated. For the detailed implementation process, please refer to the descriptions in the foregoing various embodiments, and details will not be repeated here.

[0126] Figure 10 It is a block diagram of a driver emotion recognition device shown in an exemplary embodiment of the present application. This device can be applied to Figure 1 the implementation environment shown in the figure and is specifically configured in the server 120. This device can also be applicable to other exemplary implementation environments and is specifically configured in other devices. The implementation environment applicable to this device is not limited in this embodiment.

[0127] As Figure 10 shown, the exemplary driver emotion recognition device 1000 includes: an acquisition module 1010, configured to acquire the working condition information of the vehicle and determine the driving characteristics of the driver based on the working condition information; a first determination module 1020, configured to acquire the real-time road information and determine the driving behavior of the driver based on the real-time road information and the driving characteristics; a second determination module 1030, configured to acquire the facial video of the driver and determine the expression characteristics of the driver based on the facial video; and a soothing module 1040, configured to determine the emotion state of the driver based on the driving behavior and / or the expression characteristics, and determine the corresponding emotion soothing strategy for the driver based on the emotion state.

[0128] According to one aspect of the embodiments of the present application, the above-mentioned first determination module 1020 is further configured to determine the real-time vehicle speed based on the wheel speed sensing data and the inertia measurement data; determine the real-time acceleration value of the vehicle based on the real-time vehicle speed; obtain the historical acceleration value of the vehicle, and calculate the acceleration change rate of the vehicle based on the historical acceleration value and the real-time acceleration value; determine the driving characteristics of the driver based on the real-time vehicle speed, the real-time acceleration value, and the acceleration change rate.

[0129] According to one aspect of the embodiments of the present application, the above-mentioned first determination module 1020 is further configured to determine the road condition based on the real-time road information, where the road condition includes the road type and the road geometric characteristics; determine the driving behavior of the driver based on the road type, the road geometric characteristics, and the driving characteristics.

[0130] According to one aspect of the embodiments of the present application, the above-mentioned first determination module 1020 is further configured to determine the traffic restriction parameters in the real-time environment based on the traffic data; determine the road restriction parameters in the real-time environment based on the road type and the road geometric characteristics; determine the acceleration change rate threshold of the vehicle based on the road restriction parameters and the traffic restriction parameters; if the acceleration change rate is greater than or equal to the acceleration change rate threshold, it is determined that the driver is in an aggressive driving behavior, and the negative emotional state of the driver is determined based on the aggressive driving behavior.

[0131] According to one aspect of the embodiments of the present application, the above-mentioned second determination module 1030 is further configured to obtain the expression features corresponding to each frame of the facial video to obtain an expression feature sequence arranged in time series; perform expression semantic recognition on the expression feature sequence to obtain the expression semantics of the driver; determine the emotional state of the driver based on the expression semantics.

[0132] According to one aspect of the embodiments of the present application, the above-mentioned second determination module 1030 is further configured to obtain the dynamic facial image of the driver based on the facial video; preprocess the dynamic facial image and input the processed dynamic facial image into a preset face detection model; determine the expression features of the driver based on the key points of the face features output by the preset face detection model.

[0133] According to one aspect of the embodiments of the present application, the above-mentioned soothing module 1040 is further configured to, if any one of the driving behavior and the expression features indicates that the driver is in a negative emotional state, obtain the environmental information inside the vehicle cockpit; control at least one of the fragrance system, the seat massage system, the seat ventilation system, and the music system inside the vehicle cockpit to be turned on based on the environmental information inside the vehicle cockpit.

[0134] It should be noted that the driver emotion recognition device provided in the above embodiments and the driver emotion recognition method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit perform operations have been described in detail in the method embodiments, and will not be elaborated here. In practical applications, the driver emotion recognition device provided in the above embodiments can, according to needs, allocate the above functions to different functional modules, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not limited here either.

[0135] An embodiment of the present application also provides an electronic device, including: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the driver emotion recognition method provided in each of the above embodiments.

[0136] Figure 11 The structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown. It should be noted that Figure 11 The computer system 1100 of the electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0137] As Figure 11 shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1102 or the program loaded from the storage section 1108 into the random access memory (RAM) 1103, such as executing the method in the above embodiments. In the RAM 1103, various programs and data required for system operation are also stored. The CPU 1101, ROM 1102, and RAM 1103 are connected to each other through a bus 1104. The input / output (I / O) interface 1105 is also connected to the bus 1104.

[0138] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed so that a computer program read from the removable medium 1111 is installed into the storage section 1108 as needed.

[0139] Specifically, according to an embodiment of the present application, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 1109, and / or installed from the removable medium 1111. When the computer program is executed by a central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

[0140] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0142] The units involved in the embodiments described in this application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, constitute a limitation on the units themselves.

[0143] Another aspect of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the driver emotion recognition method as described above is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist alone without being assembled into the electronic device.

[0144] Another aspect of this application also provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the driver emotion recognition method provided in each of the above embodiments.

[0145] The above content is only a preferred exemplary embodiment of this application and is not used to limit the implementation of this application. Those of ordinary skill in the art can easily make corresponding adaptations or modifications according to the main concept and spirit of this application. Therefore, the protection scope of this application should be subject to the protection scope required by the claims.

Claims

1. A driver emotion recognition method, characterized in that: include: Acquiring operating condition information of the vehicle, and determining a driving characteristic of the driver based on the operating condition information; acquiring real-time road information, and determining the driving behavior of the driver based on the real-time road information and the driving characteristics; Acquiring a facial video of the driver, and determining facial features of the driver based on the facial video; The emotional state of the driver is determined based on the driving behavior and / or the facial expression characteristics, and a corresponding emotion soothing strategy for the driver is determined based on the emotional state.

2. The method according to claim 1, characterized in that The operating condition information includes wheel speed sensor data and inertia measurement data, and determining the driving characteristics of the driver based on the operating condition information includes: determining a real-time vehicle speed of the vehicle based on the wheel speed sensor data and the inertia measurement data; Determining a real-time acceleration value of the vehicle based on the real-time vehicle speed; Acquire a historical acceleration value of the vehicle, and calculate an acceleration change rate of the vehicle based on the historical acceleration value and the real-time acceleration value; The driving characteristics of the driver are determined based on the real-time vehicle speed, the real-time acceleration value, and the acceleration change rate.

3. The method according to claim 2, characterized in that The determining the driving behavior of the driver based on the real-time road information and the driving characteristics comprises: Determine a road condition based on the real-time road information, the road condition including a road type and a road geometry feature; A driving behavior of the driver is determined based on the road type, the road geometry, and the driving characteristics.

4. The method according to claim 3, characterized in that The real-time road information also includes traffic data, and the method further includes: determining traffic restriction parameters in a real-time environment based on the traffic data; Determining road restriction parameters in a real-time environment based on the road type and the road geometric characteristics; Determining an acceleration rate change threshold of the vehicle based on the road restriction parameter and the traffic restriction parameter; If the acceleration change rate is greater than or equal to the acceleration change rate threshold, it is determined that the driver is in an aggressive driving behavior, and based on the aggressive driving behavior, it is determined that the driver is in a negative emotional state.

5. The method according to claim 1, characterized in that Determining the driver's emotional state based on the facial expression features includes: Acquire the expression feature corresponding to each frame of the facial video to obtain an expression feature sequence arranged in time sequence; Performing expression semantic recognition on the expression feature sequence to obtain the expression semantics of the driver; The emotional state of the driver is determined based on the expression semantics.

6. The method according to claim 5, characterized in that The determining the facial expression features of the driver based on the facial video comprises: Acquiring a dynamic facial image of the driver based on the facial video; Preprocessing the dynamic facial image, and inputting the processed dynamic facial image into a preset face detection model; The facial expression features of the driver are determined based on the facial feature key points output by the preset face detection model.

7. The method according to claim 1, characterized in that Determining the corresponding emotional soothing strategy for the driver based on the emotional state includes: If any one of the driving behavior and the facial expression characteristic indicates that the driver is in a negative emotional state, obtaining environmental information in the cabin of the vehicle; At least one of a fragrance system, a seat massage system, a seat ventilation system and a music system in the cabin is controlled to be turned on based on the environmental information in the cabin.

8. A driver emotion recognition device, characterized in that: The device comprises: An acquisition module, used to acquire operating condition information of the vehicle and determine the driving characteristics of the driver based on the operating condition information; a first determination module, configured to obtain real-time road information, and determine the driving behavior of the driver based on the real-time road information and the driving characteristics; a second determination module, configured to obtain a facial video of the driver and determine an expression feature of the driver based on the facial video; A soothing module is used to determine the emotional state of the driver based on the driving behavior and / or the facial expression characteristics, and determine an emotional soothing strategy corresponding to the driver based on the emotional state.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the driver emotion recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the driver emotion recognition method according to any one of claims 1 to 7.

Images