A method and system for driver emotion detection in a cockpit environment

Abstract

The present application belongs to the technical field of intelligent auxiliary driving, and particularly relates to a driver emotion detection method and system in a cockpit environment, comprising a multi-modal sensing module, an edge computing module with a hardware synchronization function, and an execution control module integrated with a vehicle bus interface. The present application simplifies the classification of related abnormal emotions, reduces algorithm complexity, and improves algorithm efficiency and accuracy by targeting specific scenarios of vehicle driving. By comprehensively collecting and fusing multi-modal features such as facial features, sound features, and sitting posture features, the calculation steps are simplified, and the calculation speed is improved. By actively learning the emotional characteristics of individual drivers, an end-side personalized model is established, which is more in line with the characteristics of individual drivers and improves recognition accuracy. By using AIGC technology, after identifying abnormal emotions, content to calm emotions is generated and synthesized into speech for playback, effectively and timely relieving the abnormal emotions of the driver.

Description

Technical Field

[0001] This invention belongs to the field of intelligent assisted driving technology, specifically relating to a method and system for detecting driver emotions in the cockpit environment. Background Technology

[0002] With the rapid increase in car ownership, the impact of driver emotional state on driving safety is becoming increasingly prominent. Existing emotion detection technologies mainly suffer from the following shortcomings: Single-modal detection (such as CN110785765A which only uses facial recognition) is not robust enough in complex driving environments, with a false alarm rate as high as 35% in actual tests. Multimodal systems (such as US20220157321A) use a serial processing architecture, resulting in a processing latency of over 300ms, which fails to meet real-time requirements. The existing classification system does not consider the specificities of driving scenarios, including all 28 basic emotions in the detection, resulting in a waste of computational resources; it also lacks an effective personalized adaptation mechanism, leading to cross-user recognition accuracy fluctuations of ±25%. To address this, a method and system for detecting driver emotions in the cockpit environment are provided.

[0003] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for detecting driver emotions in a cockpit environment, in order to solve the problems of the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method and system for detecting driver emotions in a cockpit environment, comprising the following steps: Step S101: Establish a dedicated emotion classification system for driving scenarios, dividing abnormal emotions into two dimensions: impulsive and inhibitory. Step S102: Simultaneously acquire facial features, voice features, and body pressure features via hardware; Step S103: Use a spatiotemporal alignment algorithm to perform fusion calculation on multimodal features; Step S104: Output the sentiment classification result based on the fusion features; Step S105: Trigger a graded response strategy based on the emotion level.

[0006] Preferably, the spatiotemporal alignment algorithm in step S103 includes: Visual feature extraction and AU action unit encoding; Calculate the MFCC+ΔF0 parameters for the speech features; Analysis of body pressure characteristics reveals the trajectory of the pressure center; Multimodal feature stitching is achieved through timestamp alignment.

[0007] Preferably, the hierarchical response strategy in step S105 includes: When the emotion level is 3-4, AIGC voice reassurance is triggered; When the emotional level is 5-6, the vehicle comfort control system is activated. When the emotional level reaches 7, the emergency stop protocol is activated.

[0008] Preferably, personalized learning steps are also included: Establish a user-specific feature database; The model is fine-tuned using a triplet loss function. Incremental learning and updates are performed every 24 hours.

[0009] The present invention also provides a driver emotion detection system in a cockpit environment, the system comprising: The system includes a multimodal sensing module, an edge computing module with hardware synchronization capabilities, and an execution control module with an integrated vehicle bus interface.

[0010] Preferably, the multimodal sensing module includes: 3D vision unit: A global shutter camera with 940nm infrared illumination; Acoustic sensing unit: 4-microphone beamforming array; Body pressure detection unit: 16-point capacitive pressure sensor network.

[0011] Preferably, the edge computing module includes: Supports μs-level synchronization of multimodal data; Lightweight fusion model with less than 3MB of runtime parameters; A federated learning framework is used for model updates.

[0012] Preferably, the execution control module includes: The vehicle's comfort system is linked via the CAN bus. Integrated AIGC speech synthesis unit (response latency < 50ms).

[0013] Compared with the prior art, the present invention has the following beneficial effects: The driver emotion detection method and system in the cockpit environment of the present invention simplifies the classification of related abnormal emotions, reduces algorithm complexity, and improves algorithm efficiency and accuracy by targeting specific scenarios of vehicle driving.

[0014] By comprehensively collecting and fusing multimodal features such as facial features, voice features, and posture features, the calculation steps are simplified and the calculation speed is improved.

[0015] By actively learning the emotional characteristics of individual drivers, a personalized model can be established on the edge, which is more in line with the characteristics of individual drivers and improves the accuracy of recognition.

[0016] By using AIGC technology, after identifying abnormal emotions, content to soothe the emotions is generated and synthesized into speech for playback, effectively and promptly alleviating the driver's abnormal emotions.

[0017] By collecting vehicle driving data and road condition data, and combining this with abnormal emotional changes in drivers, a predictive algorithm model is established to predict in advance what abnormal emotions drivers may experience, and to provide preventive strategies in advance, effectively reducing the occurrence of abnormal emotions. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structural framework of the multimodal sensing module of the present invention; Figure 2 This is a schematic diagram of the structural framework of the edge computing module of the present invention; Figure 3 This is a schematic diagram of the structural framework of the vehicle interface module of the present invention; Figure 4 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0019] The technical solution of this invention patent will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0020] In the description of this invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention.

[0021] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0022] Single-modal detection (such as CN110785765A which only uses facial recognition) is not robust enough in complex driving environments, with a false alarm rate as high as 35% in actual tests. Multimodal systems (such as US20220157321A) use a serial processing architecture, resulting in a processing latency of over 300ms, which fails to meet real-time requirements. The existing classification system does not take into account the special characteristics of driving scenarios and includes all 28 basic emotions in the detection, resulting in a waste of computing resources; it also lacks an effective personalized adaptation mechanism, and the cross-user recognition accuracy fluctuates by ±25%.

[0023] See attached document Figure 1-4 A method and system for detecting driver emotions in a cockpit environment, comprising the following steps: Step S101: Establish a dedicated emotion classification system for driving scenarios, dividing abnormal emotions into two dimensions: impulsive and inhibitory. Step S102: Simultaneously acquire facial features, voice features, and body pressure features via hardware; Step S103: Use a spatiotemporal alignment algorithm to perform fusion calculation on multimodal features; Step S104: Output the sentiment classification result based on the fusion features; Step S105: Trigger a graded response strategy based on the emotion level.

[0024] Furthermore, such as Figure 1-4 As shown, the spatiotemporal alignment algorithm in step S103 includes: Visual feature extraction and AU action unit encoding; Calculate the MFCC+ΔF0 parameters for the speech features; Analysis of body pressure characteristics reveals the trajectory of the pressure center; Multimodal feature stitching is achieved through timestamp alignment.

[0025] Furthermore, such as Figure 1-4 As shown, the hierarchical response strategy in step S105 includes: When the emotion level is 3-4, AIGC voice reassurance is triggered; When the emotional level is 5-6, the vehicle comfort control system is activated. When the emotional level reaches 7, the emergency stop protocol is activated.

[0026] Furthermore, such as Figure 1-4 As shown, it also includes personalized learning steps: Establish a user-specific feature database; The model is fine-tuned using a triplet loss function. Incremental learning and updates are performed every 24 hours.

[0027] It is worth noting the above: Multimodal synchronous acquisition system: Hardware-triggered synchronization is used (jitter < 50 μs). Sampling frequency: Visuals: 60fps@800×600 Audio: 48kHz / 16bit Body pressure: 100Hz.

[0028] Embedded converged computing unit: Run a lightweight fusion model (<3MB) Supports INT8 quantization inference (latency < 15ms).

[0029] Personalized learning mechanism: Local storage user characteristic templates (200 sets / person) The model is updated using a federated learning framework.

[0030] The present invention also provides a driver emotion detection system in a cockpit environment, the system comprising: The system includes a multimodal sensing module, an edge computing module with hardware synchronization capabilities, and an execution control module with an integrated vehicle bus interface.

[0031] Furthermore, such as Figure 1-4 As shown, the multimodal sensing module includes: 3D vision unit: A global shutter camera with 940nm infrared illumination; Acoustic sensing unit: 4-microphone beamforming array; Body pressure detection unit: 16-point capacitive pressure sensor network.

[0032] Furthermore, such as Figure 1-4 As shown, the edge computing module includes: Supports μs-level synchronization of multimodal data; Lightweight fusion model with less than 3MB of runtime parameters; A federated learning framework is used for model updates.

[0033] Preferably, the execution control module includes: The vehicle's comfort system is linked via the CAN bus. Integrated AIGC speech synthesis unit (response latency < 50ms).

[0034] It is worth noting that: Example 1: Device installation and configuration A 3D vision unit (15° tilt angle) is installed behind the steering wheel, a body pressure detection unit is embedded in the driver's seat cushion, and an acoustic sensing unit is arranged in the ceiling. Each unit is connected to the edge computing module of the center console via an LVDS interface.

[0035] Example 2: Anger Detection When the device detects: Visual: AU4 activity level > 70% for 2 seconds; Audio: Fundamental frequency > 250Hz and harmonic distortion > 15%; Body pressure: Pressure center offset >10cm.

[0036] Sending commands via CAN bus: Air conditioning system: Temperature reduced to 20℃; Infotainment: Play preset soothing music; Dashboard: Displays a "Please remain calm" message.

[0037] Example 3: Fatigue Prediction

[0038] The device obtains information via the OBD interface: Steering wheel angle standard deviation (<5° for 5 minutes); Vehicle speed fluctuation rate (<2km / h); The fatigue probability is calculated using the local feature library. When P > 0.75: AIGC generated the following message: "A drop in attention has been detected. It is recommended to take a break immediately."

[0039] In practical use, the system specifically reclassifies emotions that require special attention while driving, simplifying the classification of abnormal emotions to be identified: impulsiveness (anger, tension, excessive pleasure); inhibition (fatigue, pain). This simplified classification helps reduce algorithm complexity and improve recognition speed and accuracy.

[0040] Multimodal fusion analysis is performed on facial features, voice features, and posture features. Abnormal emotions are classified based on multimodal feature points to improve classification accuracy. Multimodal feature points are collected simultaneously and calculated in one go, reducing calculation steps and improving calculation speed.

[0041] For each individual driver, multimodal features during their driving process are collected. A reward mechanism is introduced to actively learn from the individual's abnormal emotional features, forming a personalized edge model to further improve accuracy.

[0042] After identifying abnormal emotions, the system uses AIGC technology to provide timely voice reassurance and offer further action suggestions. For example, when impulsive (anger) emotions are identified, the system uses a calm tone to persuade the driver to calm down; when suppressed (fatigue) emotions are identified, the system suggests that the driver buy a cup of coffee or rest at the highway rest area ahead.

[0043] By collecting data on the vehicle's OBD (On-Board Diagnostics) to obtain the driver's driving habits, and collecting information on road conditions and weather during the vehicle's journey to obtain vehicle driving environment data, combined with in-depth analysis of the driver's abnormal emotional changes, a predictive algorithm is established to effectively prevent abnormal driver emotions. For example, if it predicts that the driver will have impulsive (nervous) emotions in rainy weather, it can remind the driver in advance to check the windshield wipers, maintain a safe distance, and avoid sudden braking.

[0044] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A method for detecting driver emotions in a cockpit environment, characterized in that, Includes the following steps: Step S101: Establish a dedicated emotion classification system for driving scenarios, dividing abnormal emotions into two dimensions: impulsive and inhibitory. Step S102: Simultaneously acquire facial features, voice features, and body pressure features via hardware; Step S103: Use a spatiotemporal alignment algorithm to perform fusion calculation on multimodal features; Step S104: Output the sentiment classification result based on the fusion features; Step S105: Trigger a graded response strategy based on the emotion level.

2. The method for detecting driver emotions in a cockpit environment according to claim 1, characterized in that, The spatiotemporal alignment algorithm in step S103 includes: Visual feature extraction and AU action unit encoding; Calculate the MFCC+ΔF0 parameters for the speech features; Analysis of body pressure characteristics reveals the trajectory of the pressure center; Multimodal feature stitching is achieved through timestamp alignment.

3. The method for detecting driver emotions in a cockpit environment according to claim 1, characterized in that, The graded response strategy in step S105 includes: When the emotion level is 3-4, AIGC voice reassurance is triggered; When the emotional level is 5-6, the vehicle comfort control system is activated. When the emotional level reaches 7, the emergency stop protocol is activated.

4. The method for detecting driver emotions in a cockpit environment according to claim 1, characterized in that, It also includes personalized learning steps: Establish a user-specific feature database; The model is fine-tuned using a triplet loss function. Incremental learning and updates are performed every 24 hours.

5. A driver emotion detection system in a cockpit environment, characterized in that, The system includes: The system includes a multimodal sensing module, an edge computing module with hardware synchronization capabilities, and an execution control module with an integrated vehicle bus interface.

6. A driver emotion detection system in a cockpit environment according to claim 5, characterized in that, The multimodal sensing module includes: 3D vision unit: A global shutter camera with 940nm infrared illumination; Acoustic sensing unit: 4-microphone beamforming array; Body pressure detection unit: 16-point capacitive pressure sensor network.

7. A driver emotion detection system in a cockpit environment according to claim 5, characterized in that, The edge computing module includes: Supports μs-level synchronization of multimodal data; Lightweight fusion model with less than 3MB of runtime parameters; A federated learning framework is used for model updates.

8. A driver emotion detection system in a cockpit environment according to claim 5, characterized in that, The execution control module includes: The vehicle's comfort system is linked via the CAN bus. Integrated AIGC speech synthesis unit (response latency <50ms).

Images