Driver emotion intelligent analysis and early warning method and system

Abstract

The application discloses a driver emotion intelligent analysis and early warning method and system, and belongs to the intelligent cabin and safety technical field.The method comprises the following steps: individual biological characteristics of a driver are obtained through individual calibration; multi-modal driving data are collected in real time during driving; the data are input into a preset composite emotion model to obtain original valence and original arousal; original emotion values are calibrated based on the individual biological characteristics to obtain a comprehensive safety coefficient; and hierarchical early warning or intervention operation is performed according to the value of the coefficient.The application can accurately identify the composite emotion state in a complex driving scene through multi-modal fusion and individual baseline calibration, effectively overcomes the false alarm and missed alarm problems caused by individual differences, and realizes real-time monitoring and safety early warning of the driver state with high accuracy, high reliability and privacy protection through an offline model fixed in a vehicle-mounted system.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit and safety technology, and in particular to a method and system for driver emotional intelligence analysis and early warning. Background Technology

[0002] As the level of automotive intelligence continues to improve, intelligent cockpit systems are playing an increasingly important role in enhancing the driving experience and safety. Among these, real-time and accurate monitoring and understanding of the driver's emotions and mental state is crucial for preventing traffic accidents caused by negative states such as fatigue, distraction, and road rage.

[0003] Currently, several driver state monitoring solutions based on vision or voice exist. However, these solutions face a common core challenge in practical applications: the difficulty in implementing an emotion analysis and early warning method that can accurately understand complex and multifaceted driver emotional states and adapt to the different normal baselines of individual drivers in resource-constrained offline in-vehicle environments. Specifically, general models are inaccurate in recognizing complex emotions such as "suppressed anger" and "fake smiles due to fatigue"; and because they do not consider the differences in facial expressions and voice baselines among individual drivers in calm and focused states, using a uniform threshold can easily lead to continuous false alarms for some users, while reacting slowly to the deterioration of the state of others. This severely restricts the accuracy, reliability, and user experience of the early warning system. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the current field of intelligent cockpit and safety technology, the present invention provides a driver emotion intelligence analysis and early warning method and system, which can achieve highly accurate, highly reliable and practical driver emotion intelligence analysis and proactive safety early warning effects.

[0005] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: A method for driver emotional intelligence analysis and early warning, comprising: Personalized calibration of the current driver to obtain their individual biometrics; Multimodal driving data is collected in real time during driving; The multimodal driving data is input into a preset composite emotion model to obtain the original valence and original arousal. Based on the individual's biological characteristics, the corresponding individual feature interval is matched from the pre-stored feature interval library, and the original valence and original arousal are personalized baseline calibrated based on the emotion distribution parameters within the interval, and a comprehensive safety coefficient is calculated. Based on the comprehensive safety factor, corresponding graded early warning or intervention operations are performed.

[0006] According to one aspect of the present invention, the preset composite emotion model specifically includes: A baseline emotion recognition model is trained based on a multimodal driving dataset. This baseline emotion recognition model is used to output the raw valence and raw arousal. The baseline emotion recognition model is trained a second time based on private driver group data to construct a feature interval library containing multiple individual feature intervals, where each individual feature interval represents the emotion distribution parameters of a specific driver group under typical driving conditions. The feature interval library is embedded into the baseline emotion recognition model to generate the composite emotion model that is fixed in the vehicle system. The composite emotion model is not updated after deployment.

[0007] According to one aspect of the invention, the personalized calibration of the current driver to obtain their individual biometrics specifically includes: Guide users to complete the preset calibration process in a safe driving environment; Extract the static face embedding and neutral voiceprint embedding generated by the user in the calibration process as the individual biometrics; The calibration process includes performing smooth driving to establish an initial emotional baseline, and reading aloud preset neutral statements.

[0008] According to one aspect of the present invention, the real-time acquisition of multimodal driving data includes: Capture in-vehicle video streams and microphone audio streams; Before inputting the multimodal driving data into the composite emotion model, the method further includes denoising the multimodal driving data.

[0009] According to one aspect of the present invention, the noise reduction process includes: Spatial domain noise reduction: Based on seat position and line of sight, locate and retain only the driver's face area from the video stream; Acoustic domain noise reduction: Driver's voice is picked up directionally using beamforming technology, and environmental noise is filtered out; Feature domain noise reduction: An attention masking mechanism is introduced during feature extraction to enhance the features of facial emotion-sensitive areas.

[0010] According to one aspect of the present invention, the step of inputting the multimodal driving data into a preset composite emotion model to obtain the original valence and original arousal specifically includes: Extracting dynamic micro-expression temporal features of facial key points from denoised visual data; Extracting prosodic and emotional features of tone, speech rate, and fundamental frequency profile from the noise-reduced audio data; By fusing the temporal features of the dynamic micro-expressions with the prosodic emotion features through a cross-modal attention mechanism, a unified multimodal emotion embedding vector is generated. Based on the multi-modal emotion embedding vectors, the original valence and original arousal are inferred through the composite emotion model.

[0011] According to one aspect of the present invention, based on the individual biological characteristics, the corresponding individual characteristic interval is matched from the pre-stored characteristic interval library, and the original valence and original arousal are calibrated with personalized baselines based on the emotion distribution parameters within this interval, and a comprehensive safety factor is calculated, specifically including: Based on the individual biological characteristics, the corresponding individual characteristic interval is matched from the pre-stored characteristic interval library, and the valence mean, valence standard deviation, arousal mean, and arousal standard deviation of this interval are obtained; Based on the obtained means and standard deviations, the standardized deviation degrees of the original valence and original arousal relative to their corresponding baselines are calculated respectively; Based on the standardized deviation degrees, a negative valence risk component and an arousal anomaly risk component are constructed; The negative valence risk component, the arousal anomaly risk component, and an independent fatigue risk component are weighted and fused to obtain a total risk value; The total risk value is converted into the comprehensive safety factor with a value range between 0 and 1 through a preset mapping function.

[0012] According to one aspect of the present invention, the calculation formula for the weighted fusion is: R_total = w_v * R_v + w_a * R_a + w_f * R_f Where, R_total is the total risk value, R_v is the negative valence risk component, R_a is the arousal anomaly risk component, R_f is the fatigue risk component, w_v, w_a, w_f are the corresponding weights, and w_v + w_a + w_f = 1.0; The mapping function is an exponential decay function: S = exp(-k * R_total), where S is the comprehensive safety factor and k is the decay rate constant.

[0013] According to one aspect of the present invention, performing corresponding hierarchical early warning or intervention operations based on the comprehensive safety factor specifically includes: If the comprehensive safety factor S > 0.8, it is determined that the state is normal and no early warning or intervention is performed; If 0.4 < S ≤ 0.8, a gentle reminder operation is performed; If S ≤ 0.4, a high-risk intervention operation is performed; Among them, the high-risk intervention operation includes at least one of the following: sending a warning notice to a preset emergency contact, restricting or exiting the intelligent driving function of the vehicle.

[0014] A driver emotion intelligence analysis and early warning system, comprising: The calibration module performs personalized calibration on the current driver to obtain their individual biometrics. The data acquisition module collects multimodal driving data in real time during the driving process; The model module inputs the multimodal driving data into a preset composite emotion model to obtain the original valence and original arousal. The calculation module performs personalized baseline calibration on the original valence and original arousal based on the individual's biometrics, and calculates a comprehensive safety factor. The early warning module performs corresponding graded early warning or intervention operations based on the comprehensive safety coefficient.

[0015] The advantages of this invention are as follows: By employing a composite emotion model based on multimodal fusion and personalized baseline calibration, it can accurately identify the complex emotional states of drivers in complex driving scenarios (such as "suppressing anger"), effectively overcoming the shortcomings of inaccurate identification by traditional single-modal or simple rule models; by matching personalized emotion baselines to different drivers and performing standardized risk calculations, it significantly reduces false alarms and false negatives caused by individual differences, making the warnings more accurate and reliable; at the same time, by using a "pre-trained general model + embedded group feature interval library" approach, an offline model solidified in the vehicle system is generated, which, while ensuring high-performance analysis capabilities, perfectly meets the core requirements of the vehicle environment for offline deployment, real-time response, and data privacy, thus systematically realizing highly accurate, highly reliable, and practical driver emotion intelligent analysis and proactive safety warnings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a driver emotion intelligence analysis and early warning method according to the present invention; Figure 2 This is a flowchart illustrating a driver emotion intelligence analysis and early warning system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 like Figure 1 As shown, a driver emotional intelligence analysis and early warning method includes the following steps: Step S1: Perform personalized calibration on the current driver to obtain their individual biometrics.

[0020] Step S1 specifically includes: When the vehicle is powered on or a new driver is detected, the system determines whether the driver has completed personalized calibration. If not, the calibration process is initiated. The system guides the user to complete preset calibration tasks in a safe driving environment (e.g., when the vehicle is stationary or on a closed, straight road). The calibration tasks include: driving smoothly for a period of time (e.g., 5 minutes, without sudden acceleration, sudden braking, or lane departure) to establish the user's emotional baseline under normal conditions; and reading aloud several preset neutral statements (e.g., "The weather is sunny today").

[0021] During this process, the system simultaneously collects the driver's facial video and voice data, and extracts static face embeddings (representing the user's inherent facial structural features, such as the distance between eyebrows and eyes, and the natural curvature of the corners of the mouth) and neutral voiceprint embeddings (representing the user's inherent voice characteristics in a calm state). These two types of embedding information together constitute the driver's individual biometric profile and are stored locally. If the system detects that the driver has completed calibration, it directly retrieves the driver's stored individual biometric profile.

[0022] Step S2: Collect multimodal driving data in real time during the driving process.

[0023] Step S2 specifically involves: continuously acquiring in-vehicle video streams via the vehicle's built-in driver-facing camera, and simultaneously acquiring microphone audio streams via a microphone array. These two data streams together constitute real-time, raw multimodal driving data.

[0024] In this embodiment of the invention, the acquisition of in-vehicle video streams and microphone audio streams involves the processing of the driver's facial image and voice data. All data acquisition and processing operations strictly comply with relevant laws, regulations, and personal information protection policies. Before implementing this invention, the system will clearly inform the driver through the in-vehicle interface of the purpose (only for driver status analysis and safety warnings), method (local real-time processing, not uploaded to the cloud), and scope (limited to facial area images and voice during driving). The driver can express informed consent by actively confirming or continuing to drive. Simultaneously, the system provided by this invention performs feature desensitization processing on the acquired facial images, extracting only irreversible feature vectors such as facial key points, without saving the original image or audio data. All calculations are completed locally on the in-vehicle terminal to fully protect the driver's privacy rights.

[0025] Step S3: Input the multimodal driving data into a preset composite emotion model to obtain the original valence and original arousal.

[0026] Step S3 specifically includes: First, the raw multimodal driving data collected in step S2 is subjected to real-time noise reduction and preprocessing to obtain clean visual and audio data. Then, the processed data is input into a composite emotion model pre-installed in the in-vehicle system.

[0027] The model performs the following calculations internally: Dynamic micro-expression temporal features of facial key points (such as eyebrows and corners of the mouth) are extracted from clean visual data using a visual encoder (such as Transformer). From clean audio data, prosodic and emotional features such as intonation, speech rate, and fundamental frequency profile are extracted using acoustic models. By fusing the aforementioned visual and audio features through a cross-modal attention mechanism, a unified multimodal emotion embedding vector is generated. Ultimately, the model outputs two core, unpersonally calibrated raw emotion dimensions based on this embedded vector inference: raw valence (V_raw) and raw arousal (A_raw). The valence ranges from -1.0 to +1.0, with positive values ​​representing positive emotions and negative values ​​representing negative emotions; the arousal ranges from 0.0 to 1.0, with higher values ​​indicating higher physiological arousal levels.

[0028] Step S4: Based on the individual's biometrics, match the corresponding individual feature interval from the pre-stored feature interval library, and perform personalized baseline calibration on the original valence and original arousal based on the emotion distribution parameters within the interval, and calculate a comprehensive safety coefficient.

[0029] Step S4 specifically includes: Step S41: Match Personalized Baseline. Based on the individual biometrics (static face embedding and neutral voiceprint embedding) of the current driver obtained in Step S1, match the individual feature interval that best matches the driver's group from the feature interval library embedded in the composite emotion model, and obtain the emotion baseline parameters of the group under typical driving conditions, including: mean valence (μ_v), standard deviation of valence (σ_v), mean arousal (μ_a), and standard deviation of arousal (σ_a).

[0030] Step S42: Calculate the standardized deviation. Using the raw values ​​(V_raw, A_raw) obtained in Step S3 and the baseline parameters (μ_v, σ_v, μ_a, σ_a) obtained in Step S41, calculate the standardized deviation (Z-score) of valence and arousal relative to their personalized baselines: Valence deviation z_v=(V_raw−μ_v) / (σ_v+1e−8) Arousal deviation z_a=(A_raw−μ_a) / (σ_a+1e−8) Step S43: Constructing and fusing risk components. Construct risk items based on standardized deviations: Negative valence risk R_v=max(0,−z_v) (risk is only included when the mood is more negative than the baseline).

[0031] Arousal abnormality risk R_a=|z_a| (whether the arousal level is too high (excited) or too low (drowsy), it is included in the risk).

[0032] Introduce an independent fatigue risk component R_f (e.g., obtained via the PERCLOS eye-tracking index).

[0033] The three risk components are then weighted and combined according to preset weights to obtain the total risk value R_total: R_total=w_v*R_v+w_a*R_a+w_f*R_f Where w_v, w_a, and w_f are weights, and w_v + w_a + w_f = 1.0. A typical configuration could be w_v = 0.4, w_a = 0.35, and w_f = 0.25.

[0034] Step S44: Mapping to a safety factor. The total risk value is mapped to the [0,1] interval using an exponential decay function to obtain the final comprehensive safety factor S: S=exp(−k∗R_total) Where k is the decay rate constant (e.g., k=1.2). The closer the S value is to 1, the safer the state; the closer it is to 0, the higher the risk.

[0035] Step S5: Perform corresponding classification warnings or intervention operations according to the value of the comprehensive safety factor.

[0036] Specifically, step S5 is as follows: The system compares the real-time comprehensive safety factor S calculated in step S4 with a preset threshold value and triggers corresponding response strategies: If S > 0.8, it is determined that the driver's state is normal and no operation is performed.

[0037] If 0.4 < S ≤ 0.8, it is determined as a medium risk, and a gentle reminder operation is performed, such as emitting a soft warning sound or prompt message through cockpit voice or HUD.

[0038] If S ≤ 0.4, it is determined as a high risk, and high-risk intervention operations are performed, such as automatically lowering the in-vehicle entertainment volume, flashing the hazard warning lights, and performing further active safety measures, including but not limited to sending a warning notice to a preset emergency contact and restricting or exiting the vehicle's intelligent driving function.

[0039] Advantages of this embodiment: By establishing an emotional baseline for different drivers through personalized calibration steps, the problem of misjudgment of the unified threshold value caused by individual differences is fundamentally solved; by processing multi-modal data through a composite emotion model, the recognition ability of complex emotions such as "suppressing anger" is improved; through standardized deviation calculation and weighted risk fusion, the original and abstract valence and arousal degree values are scientifically and reasonably quantified into an intuitive and gradable response comprehensive safety factor. The entire solution process is clear, and each step is closely connected, ultimately achieving accurate, personalized, and offline-operable driver emotional state analysis and safety warning.

[0040] Embodiment 2 The difference between this embodiment and Embodiment 1 lies in the specific implementation method of personalized calibration and the construction and adaptive mechanism of the composite emotion model, aiming to provide an alternative technical solution that is insensitive to users and does not rely on specific group annotation data to further improve the usability and generalization ability of the system.

[0041] This embodiment adopts an unsupervised self-learning calibration method based on initial driving behavior. Specifically, it includes: When the system first identifies a new driver, it does not immediately start an active guidance process, but uses a period of initial and continuous driving data (for example, the first 20 - 40 minutes after the start of driving) as the calibration period. During this calibration period, the system continuously collects the driver's multi-modal data (vision and audio), and assumes that the driver is generally in a normal driving state during this stage.

[0042] The system analyzes the temporal features extracted during the calibration period using unsupervised clustering algorithms (such as K-means), automatically identifying and filtering out outlier data segments that clearly belong to extreme emotions (such as laughter or shouting). Based on the remaining data clusters representing "normal" states, the system automatically calculates and extracts the driver's average facial feature embedding vector and average acoustic feature embedding vector, using these as their individual biometrics. Simultaneously, during this calibration period, the system runs a baseline emotion model (see below) and records the output valence and arousal sequences to calculate the driver's initial emotional baseline parameters, including mean valence (μ*v), standard deviation of valence (σ*v), mean arousal (μ*a), and standard deviation of arousal (σ*a). This method requires no active cooperation from the driver, achieving "unobtrusive" personalization and a more natural calibration process.

[0043] The advantages of this embodiment are: personalized calibration is completed through unsupervised self-learning, which completely eliminates the dependence on the user's active cooperation, achieves seamless adaptation that is "ready to use as soon as you get in the car", is more in line with real-world driving scenarios, and lowers the threshold for users to use it.

[0044] Example 3 like Figure 2 As shown, a driver emotion intelligence analysis and early warning system includes: The calibration module M1 performs personalized calibration on the current driver to obtain their individual biometrics. The data acquisition module M2 collects multimodal driving data in real time during driving. Model module M3 inputs the multimodal driving data into a preset composite emotion model to obtain the original valence and original arousal. Calculation module M4 performs personalized baseline calibration of the original valence and original arousal based on the individual's biometrics, and calculates a comprehensive safety factor; The early warning module M5 executes corresponding graded early warning or intervention operations based on the value of the comprehensive safety coefficient.

[0045] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for driver emotional intelligence analysis and early warning, characterized in that, include: Personalized calibration of the current driver to obtain their individual biometrics; Multimodal driving data is collected in real time during driving; The multimodal driving data is input into a preset composite emotion model to obtain the original valence and original arousal. Based on the individual's biometrics, a personalized baseline calibration is performed on the original titer and original arousal level to calculate a comprehensive safety factor, which specifically includes: Based on the individual's biological characteristics, the corresponding individual characteristic interval is matched from the pre-stored feature interval library, and the mean valence, standard deviation of valence, mean arousal and standard deviation of arousal of the interval are obtained. Based on the obtained mean and standard deviation, the standardized deviation of the original valence and original arousal relative to their corresponding baselines is calculated respectively. Based on the standardized deviation, a negative valence risk component and an arousal abnormality risk component are constructed; The negative valence risk component, the arousal abnormality risk component, and the independent fatigue risk component are weighted and fused to obtain the total risk value. The total risk value is converted into a comprehensive safety factor with a value between 0 and 1 through a preset mapping function; Wherein, the negative valence risk is R_v=max(0,−z_v), and z_v is the valence deviation; the arousal anomaly risk component is R_a=|z_a|, and z_a is the arousal deviation. Based on the comprehensive safety factor, corresponding graded early warning or intervention operations are performed.

2. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, The pre-defined composite emotion model specifically includes: A baseline emotion recognition model is trained based on a multimodal driving dataset. This baseline emotion recognition model is used to output the raw valence and raw arousal. The baseline emotion recognition model is trained a second time based on private driver group data to construct a feature interval library containing multiple individual feature intervals, where each individual feature interval represents the emotion distribution parameters of a specific driver group under typical driving conditions. The feature interval library is embedded into the baseline emotion recognition model to generate the composite emotion model that is fixed in the vehicle system. The composite emotion model is not updated after deployment.

3. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, The process involves personalized calibration of the current driver to obtain their individual biometric characteristics. Specifically, it includes: Guide users to complete the preset calibration process in a safe driving environment; Extract the static face embedding and neutral voiceprint embedding generated by the user in the calibration process as the individual biometrics; The calibration process includes performing smooth driving to establish an initial emotional baseline, and reading aloud preset neutral statements.

4. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, The real-time acquisition of multimodal driving data includes: Capture in-vehicle video streams and microphone audio streams; Before inputting the multimodal driving data into the composite emotion model, the method further includes denoising the multimodal driving data.

5. The driver emotion intelligence analysis and early warning method according to claim 4, characterized in that, The noise reduction process includes: Spatial domain noise reduction: Based on seat position and line of sight, locate and retain only the driver's face area from the video stream; Acoustic domain noise reduction: Driver's voice is picked up directionally using beamforming technology, and environmental noise is filtered out; Feature domain noise reduction: An attention masking mechanism is introduced during feature extraction to enhance the features of facial emotion-sensitive areas.

6. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, Inputting the multi-modal driving data into a preset composite emotion model to obtain an original valence and an original arousal level specifically includes: Extracting dynamic micro-expression time series features of facial key points from the denoised visual data; Extracting prosodic emotion features of intonation, speech rate, and fundamental frequency contour from the denoised audio data; Fusing the dynamic micro-expression time series features and the prosodic emotion features through a cross-modal attention mechanism to generate a unified multi-modal emotion embedding vector; Based on the multi-modal emotion embedding vector, inferring the original valence and the original arousal level through the composite emotion model.

7. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, The calculation formula for the weighted fusion is: R_total = w_v * R_v + w_a * R_a + w_f * R_f where R_total is the total risk value, R_v is the negative valence risk component, R_a is the arousal anomaly risk component, R_f is the fatigue risk component, w_v, w_a, and w_f are the corresponding weights, and w_v + w_a + w_f = 1.0; The mapping function is an exponential decay function: S = exp(-k * R_total), where S is the comprehensive safety factor and k is the decay rate constant.

8. The driver emotion intelligence analysis and early warning method according to claim 1, characterized in that, Performing corresponding hierarchical warning or intervention operations according to the comprehensive safety factor specifically includes: If the comprehensive safety factor S > 0.8, it is determined that the state is normal and no warning or intervention is performed; If 0.4 < S ≤ 0.8, a gentle reminder operation is performed; If S ≤ 0.4, a high-risk intervention operation is performed; where the high-risk intervention operation includes at least one of the following: sending a warning notice to a preset emergency contact, restricting or exiting the intelligent driving function of the vehicle.

9. A driver emotion intelligence analysis and early warning system, characterized in that, Based on the driver emotion intelligent analysis and warning method according to any one of claims 1 to 8, including: A calibration module for performing personalized calibration on the current driver to obtain their individual biometric characteristics; An acquisition module for collecting multi-modal driving data in real time during driving; A model module for inputting the multi-modal driving data into a preset composite emotion model to obtain an original valence and an original arousal level; [[ID= ​

Images