Driving emotion intervention and control system based on multi-mode perception

Abstract

The invention relates to the technical field of intelligent automobiles and automobile safety, in particular to a driving emotion intervention and control system based on multi-modal perception, which comprises a data acquisition module used for synchronously acquiring multi-source heterogeneous data; the edge analysis module is used for carrying out preprocessing, feature extraction and parallel analysis on the multi-source heterogeneous data so as to output an emotion probability value and passenger existence confidence; the scenarized decision-making module is used for constructing a hierarchical decision-making model based on the emotion probability value and the passenger existence confidence so as to complete emotion state judgment and driving scene classification; according to the system and the method, the emotion of the driver is accurately and comprehensively perceived, layering, grading and humanization can be executed based on deep scene understanding, and a forced safety bag is achieved.

Description

A driving emotion intervention and control system based on multimodal perception Technical Field

[0001] This invention relates to the field of intelligent vehicles and vehicle safety technology, and in particular to a driving emotion intervention and control system based on multimodal perception. Background Technology

[0002] With the rapid development of intelligent and connected vehicles, driving safety has transcended the traditional scope of mechanical performance, and the driver's mental and physical state has become one of the core variables affecting driving safety. Among them, anger during driving (i.e., "road rage") is an important factor inducing dangerous driving behaviors (such as aggressive following, malicious cutting off, sudden acceleration and braking, etc.). Statistics show that a considerable proportion of traffic accidents are directly or indirectly related to drivers' loss of emotional control.

[0003] Existing technologies have explored solutions for driver emotion monitoring and intervention, but they generally suffer from the following bottlenecks: Limited perception dimensions and high false alarm rates: Many solutions rely on a single modality, such as facial images or heart rate monitoring alone. Single data sources are susceptible to environmental interference (e.g., changes in lighting, poor sensor contact) and struggle to comprehensively reflect complex emotional states, leading to frequent false alarms or missed detections; Disconnect between perception and decision-making, lacking contextual understanding: While some solutions integrate data from multiple sensors, they only perform simple data or feature layer fusion, failing to deeply integrate perception results with specific driving scenarios (e.g., in-vehicle conflict, external interference, solo driving). This makes interventions lack specificity, like "scratching an itch through a boot," unable to effectively reach the source of the emotion; Inflexible intervention methods and poor human-machine collaboration: Existing interventions mostly remain at a passive level, such as voice reminders or playing music. Such gentle approaches are ineffective when emotions are severely out of control; while some proposed solutions involving forced vehicle takeover are too harsh and may provoke resentment and resistance from drivers, or even create new social conflicts and safety risks in certain scenarios (such as when there are other passengers in the vehicle), lacking the art of balancing safety and respect for driver autonomy. Summary of the Invention

[0004] The purpose of this invention is to provide a vehicle software upgrade system and method to solve at least one of the problems existing in the prior art.

[0005] This invention provides the following solution:

[0006] A driving emotion intervention and control system based on multimodal perception, comprising:

[0007] The data acquisition module is used to simultaneously acquire heterogeneous data from multiple sources.

[0008] The edge analysis module is used to preprocess, extract features and perform parallel analysis on multi-source heterogeneous data to output sentiment probability values ​​and occupant presence confidence.

[0009] The scenario-based decision-making module is used to build a hierarchical decision-making model based on the probability value of emotions and the confidence level of the presence of occupants, so as to complete the determination of emotional state and the classification of driving scenarios.

[0010] The safety execution module is used to execute safety intervention strategies based on the results of emotional state determination and driving scenario classification.

[0011] Furthermore, the edge analysis module includes:

[0012] The speech analysis unit is used to build a speech emotion analysis model based on audio data in order to analyze the probability value of speech emotion.

[0013] The operation analysis unit is used to build an operation emotion analysis model based on tactile data in order to analyze the probability value of operation emotion.

[0014] The visual analysis unit is used to build a facial emotion analysis model based on visual data in order to analyze the probability value of facial emotions.

[0015] The occupant determination unit is used to build an occupant analysis model based on visual and tactile data in order to analyze the confidence level of occupant presence.

[0016] Furthermore, the speech analysis unit performs speech activity detection on the audio data, filters out non-human voices and environmental noise, preprocesses the effective speech frames, and extracts key acoustic features that can characterize emotions. The key acoustic features include Mel frequency cepstral coefficients and their first and second order differences. The key acoustic feature sequence of a continuous audio data is input into a deep temporal neural network for modeling to obtain a speech emotion analysis model. It adopts a hybrid structure of convolutional neural network and long short-term memory network. The CNN layer is responsible for extracting timbre features from the features of each frame, and the LSTM layer learns the evolution of speech emotion in the time dimension. Finally, a fully connected layer and a sigmoid activation function are connected in the speech emotion analysis model. The speech emotion analysis model outputs a continuous value in [0,1], which is used as the speech emotion probability value. A speech emotion probability value of 0 indicates extreme calmness, and a speech emotion probability value of 1 indicates extreme excitement.

[0017] Furthermore, the operation analysis unit sets a time window of 5 seconds and extracts features from the tactile data within the window to obtain temporal features, frequency features, and morphological features. The temporal features include the mean, variance, peak-to-peak value, and number of overshoots of the tactile data. The frequency features are the proportion of signal energy in the 2-8Hz frequency band after performing a Fast Fourier Transform on the signal. The morphological features include the mean and variance of the first-order difference of the tactile data. The temporal features, frequency features, and morphological features are fused into an operation feature vector, and the operation feature vector is input into an ensemble learning classifier to obtain the operation emotion probability value.

[0018] Furthermore, the visual analysis unit uses a face detection algorithm to locate the driver's facial region, standardizes the driver's facial region, and inputs it into a lightweight convolutional neural network. The lightweight convolutional neural network is trained on a facial expression recognition dataset, and the terminal outputs the probability distribution of facial expressions through a Softmax function, taking the probability value of the facial expression as the anger category as the facial emotion probability value.

[0019] Furthermore, the occupant determination unit uses an improved lightweight YOLO target detection model to perform real-time inference on visual data, detects and outputs the bounding boxes of all "person" targets and their presence confidence scores, reads the tactile data from the seat pressure sensor, and if the average pressure value in the seat area exceeds 10 kg for 3 consecutive seconds, a pressure flag is triggered. The occupant presence is determined based on the presence confidence score and the pressure flag. When the presence confidence score is greater than 0.8 and the pressure flag is triggered, a passenger is determined and the occupant presence confidence score is set to 1. When the presence confidence score is greater than 0.8 or the pressure flag is triggered, a redundancy check mechanism is triggered to perform a secondary determination based on the seat belt status. If the seat belt status is "fastened," the occupant presence confidence score is set to 1. If the seat belt status is "unfastened," the occupant presence confidence score is set to 0. When the presence confidence score is less than or equal to 0.8 and the pressure flag is not triggered, the occupant presence confidence score is set to 0.

[0020] Furthermore, the scenario-based decision-making module sets an independent warning threshold for each emotion probability value. When any emotion probability value exceeds its corresponding warning threshold, an abnormal emotion warning is immediately triggered.

[0021] The scenario-based decision-making module analyzes the comprehensive emotion index based on the emotion probability value and determines the emotional state based on the comprehensive emotion index. If the comprehensive emotion index is less than 0.3, the emotional state is determined to be normal. If the comprehensive emotion index is greater than or equal to 0.3 and less than 0.6, the emotional state is determined to be level two abnormal. If the comprehensive emotion index is greater than or equal to 0.6, the emotional state is determined to be level three abnormal.

[0022] Furthermore, the scenario-based decision-making module categorizes driving scenarios based on emotion probability values ​​and occupant confidence levels, classifying driving scenarios into in-vehicle social conflict, external traffic interference, and emotional fluctuations while driving alone.

[0023] Furthermore, when the emotional state is at level two abnormality, the safety execution module combines the driving scenario and emotional state into a composite key, retrieves the local hierarchical voice template library, and initiates scenario-adaptive voice reassurance.

[0024] Furthermore, when the emotional state is at level three abnormal, the safety execution module initiates collaborative confirmation when the occupant's confidence level is 1, sets a 10-second waiting confirmation timeout, and if consent confirmation is received within this time limit, the system performs auxiliary restricted takeover; when the occupant's confidence level is 0, a forced takeover warning is issued and forced safety protection is performed, and after the warning, the system automatically takes over control.

[0025] During the intervention, the safety execution module continuously analyzes the comprehensive emotion index. When the emotional state is at level two abnormal and the comprehensive emotion index is less than 0.3 for 60 seconds, or when the emotional state is at level three abnormal and the comprehensive emotion index is less than 0.3 for 180 seconds, the intervention is triggered to be lifted. Upon receiving confirmation from the driver when the intervention is lifted, the system gradually restores control in stages.

[0026] The beneficial effects of this invention are as follows: Firstly, by deeply integrating multimodal information such as voice, operational behavior, facial expressions, and occupant status, and employing dynamic weighted fusion and conflict arbitration mechanisms, the system achieves high-precision and robust early identification and quantitative assessment of driver anger, significantly reducing the risk of misjudgment due to single sensor failure or environmental interference. Secondly, the system innovatively and intelligently correlates and categorizes the quantitative results of emotions with specific driving scenario causes, enabling the system not only to determine whether the driver's emotions are "abnormal," but also to understand "why the emotions are abnormal," thus providing a highly insightful decision-making basis for subsequent intervention. Thirdly, the system designs a progressive intervention system ranging from "scenario-adaptive flexible persuasion" to "occupant status-differentiated mandatory takeover." At the highest risk level, this system creatively implements safety measures of varying intensities based on in-vehicle social situations, constructing an impenetrable safety barrier in extreme cases while maximizing respect for driving autonomy and in-vehicle interpersonal interaction, achieving an organic unity of safety rigidity and human flexibility. Finally, the system has a complete dynamic evaluation and smooth recovery mechanism, which can intelligently release the intervention and return control based on the driver's emotional state, ensuring the closed-loop and humanized nature of the entire intervention process. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 is a schematic diagram of the driving emotion intervention and control system based on multimodal perception in this embodiment.

[0029] Figure 2 is a schematic diagram of the edge analysis module in this embodiment. Detailed Implementation

[0030] The vehicle software upgrade system and method disclosed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined with each other to achieve better technical effects. In the accompanying drawings of the following embodiments, the same reference numerals appearing in each drawing represent the same features or components, which can be applied to different embodiments. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0031] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes and to aid those skilled in the art in understanding and reading the invention. They are not intended to limit the conditions under which the invention can be implemented. Any modifications to the structure, changes in proportions, or adjustments to size, provided they do not affect the effectiveness or purpose of the invention, should fall within the scope of the technical content disclosed in the invention. The scope of the preferred embodiments of the present invention includes other implementations, wherein functions may be performed not in the order stated or discussed, including substantially simultaneously or in reverse order, depending on the functions involved. This should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0032] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0033] In the description of the embodiments of this application, " / " means "or", and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" means: A and B exist alone, B exists alone, and A and B exist simultaneously. In the description of the embodiments of this application, "multiple" refers to two or more embodiments.

[0034] Please refer to Figure 1, which shows the driving emotion intervention and control system based on multimodal perception in this embodiment, including:

[0035] The data acquisition module is used to simultaneously collect multi-source heterogeneous data, including audio data, tactile data, and visual data, which reflect the driver's emotions and the in-vehicle environment. The audio data uses a microphone array located in the car ceiling to collect the voice signals of the entire cabin at a sampling rate of not less than 16kHz, for analyzing voice content, tone, and emotions. The tactile data includes a piezoelectric pressure sensor array embedded in the steering wheel grip, which collects the driver's grip strength, tapping, twisting, and other operating forces and patterns at a frequency of not less than 50Hz, as well as a distributed thin-film pressure sensor laid in the driver's seat cushion, for monitoring seat pressure distribution and duration to assist in judging the occupant's state. The visual data uses the on-board driver monitoring system camera and the cabin wide-angle camera to capture the driver's facial expressions, gaze direction, and visual information of other occupants in the vehicle at a frame rate of not less than 25fps.

[0036] Specifically, the data acquisition module in this embodiment integrates a microphone array, a distributed pressure sensor array, and a multi-view camera to achieve non-invasive, multi-dimensional, synchronous acquisition of the driver's physiological state, operational behavior, and in-vehicle environment. This design overcomes the limitations of traditional single sensors, simultaneously capturing three key modalities: sound, touch, and vision, providing a rich, complementary, and high-fidelity data foundation for subsequent in-depth analysis. By employing mature and highly reliable automotive sensors, this module significantly reduces overall hardware costs and integration complexity while ensuring system performance, making this solution highly feasible for engineering implementation and laying a solid physical foundation for building a robust driving emotion perception system.

[0037] Please refer to Figure 1. The driving emotion intervention and control system based on multimodal perception also includes:

[0038] The edge analysis module is used to preprocess, extract features, and perform parallel analysis on multi-source heterogeneous data to output sentiment probability values ​​and occupant presence confidence.

[0039] Please refer to Figure 2, which shows the edge analysis module in this implementation, including:

[0040] The speech analysis unit is used to build a speech emotion analysis model based on audio data in order to analyze the probability value of speech emotion.

[0041] Specifically, in this embodiment, the speech analysis unit performs speech activity detection on the audio data, filters out non-human voices and environmental noise, preprocesses the effective speech frames, and extracts key acoustic features that can characterize emotions. The key acoustic features include Mel frequency cepstral coefficients and their first and second order differences. The key acoustic feature sequence of a continuous audio data is input into a deep temporal neural network for modeling to obtain a speech emotion analysis model. It adopts a hybrid structure of convolutional neural network and long short-term memory network. The CNN layer is responsible for extracting timbre features from the features of each frame, and the LSTM layer learns the evolution of speech emotions in the time dimension. Finally, a fully connected layer and a sigmoid activation function are connected in the speech emotion analysis model. The speech emotion analysis model outputs a continuous value in [0,1], which is used as the speech emotion probability value. A speech emotion probability value of 0 indicates extreme calmness, and a speech emotion probability value of 1 indicates extreme excitement.

[0042] Specifically, in this embodiment, the audio data is preprocessed by pre-emphasis, framing, and windowing. In the framing process, the frame length is set to 25ms, the MFCC order is 13, and the time window length is 2 seconds. The voice emotion analysis model outputs the voice emotion probability value every 0.5 seconds. The voice emotion analysis model needs to be trained using a large amount of in-vehicle voice data with continuous emotion labels. The loss function adopts mean squared error to approximate the real emotion intensity.

[0043] Please refer to Figure 2 for further details. The edge analysis module also includes:

[0044] The operation analysis unit is used to build an operation emotion analysis model based on tactile data in order to analyze the probability value of operation emotion.

[0045] Specifically, in this embodiment, the operation analysis unit sets a time window of 5 seconds and extracts features from the tactile data within the window to obtain temporal features, frequency features, and morphological features. The temporal features include the mean, variance, peak-to-peak value, and overshoot count of the tactile data. The frequency features are the proportion of signal energy in the 2-8Hz frequency band after performing a Fast Fourier Transform on the signal. The morphological features include the mean and variance of the first-order difference of the tactile data. The temporal features, frequency features, and morphological features are fused into an operation feature vector, and the operation feature vector is input into an ensemble learning classifier to obtain the operation emotion probability value.

[0046] Specifically, in this embodiment, the ensemble learning classifier has been trained on a labeled dataset. The ensemble learning classifier can use random forest and gradient boosting decision tree. The labeled dataset is a dataset of tactile data labeled as normal operation and angry operation. The operation emotion probability value is the probability that the operation feature vector belongs to the angry operation. The overshoot count is the number of times the pressure value exceeds the set pressure safety threshold. The pressure safety threshold is set to 50N. An overshoot count exceeding 3 times / 5 seconds is considered an angry operation.

[0047] Please refer to Figure 2 for further details. The edge analysis module also includes:

[0048] The visual analysis unit is used to build a facial emotion analysis model based on visual data in order to analyze the probability value of facial emotions.

[0049] Specifically, in this embodiment, the visual analysis unit uses a face detection algorithm to locate the driver's facial region, standardizes the driver's facial region, and inputs it into a lightweight convolutional neural network. The lightweight convolutional neural network is trained on a facial expression recognition dataset, and the terminal outputs the probability distribution of facial expressions through a Softmax function, taking the probability value of the facial expression as the anger category as the facial emotion probability value.

[0050] Please refer to Figure 2 for further details. The edge analysis module also includes:

[0051] The occupant determination unit is used to build an occupant analysis model based on visual and tactile data in order to analyze the confidence level of occupant presence.

[0052] Specifically, in this embodiment, the occupant determination unit uses an improved lightweight YOLO target detection model to perform real-time inference on visual data, detects and outputs the bounding boxes of all "person" targets and their presence confidence scores, reads the tactile data from the seat pressure sensor, and if the average pressure value of the seat area exceeds 10 kg for 3 consecutive seconds, a pressure flag is triggered. The occupant presence is determined based on the presence confidence score and the pressure flag. When the presence confidence score is greater than 0.8 and the pressure flag is triggered, it is determined that there is a passenger and the occupant presence confidence score is set to 1. When the presence confidence score is greater than 0.8 or the pressure flag is triggered, a redundancy check mechanism is triggered to perform a secondary judgment in conjunction with the seat belt status. If the seat belt status is "fastened", the occupant presence confidence score is set to 1. If the seat belt status is "unfastened", the occupant presence confidence score is set to 0. When the presence confidence score is less than or equal to 0.8 and the pressure flag is not triggered, the occupant presence confidence score is set to 0.

[0053] Specifically, the edge analysis module described in this embodiment achieves efficient and real-time processing of multi-source heterogeneous data by running multiple customized lightweight intelligent analysis models in parallel within the onboard computing unit. This module creatively decomposes the complex emotion recognition task into multiple parallel sub-tasks, including speech, operation, vision, and occupant presence detection, and employs optimal algorithms such as CNN-LSTM, Random Forest, lightweight CNN, and improved YOLO for modeling based on the characteristics of each modality of data. This "divide and conquer, parallel collaboration" architecture fully utilizes the limited computing power of the vehicle and effectively avoids the computational latency and resource bottlenecks that may arise from a single complex model. Simultaneously, the independent analysis output before model fusion provides interpretable intermediate results for upper-level decision-making, enhancing system transparency and debugging convenience, and significantly improving the accuracy and timeliness of emotion perception.

[0054] Please refer to Figure 1. The driving emotion intervention and control system based on multimodal perception also includes:

[0055] The scenario-based decision-making module is used to build a hierarchical decision-making model based on the probability value of emotions and the confidence level of the presence of occupants, so as to complete the determination of emotional state and the classification of driving scenarios.

[0056] Specifically, in this embodiment, the scenario-based decision-making module sets an independent warning threshold for each emotion probability value. When any emotion probability value exceeds its corresponding warning threshold, an abnormal emotion warning is immediately triggered.

[0057] Specifically, in this embodiment, when the scenario-based decision-making module identifies a preset set of extremely negative keywords in the audio data, it immediately triggers an abnormal emotion warning. In this embodiment, the warning threshold corresponding to the voice emotion probability value is set to 0.4, the warning threshold corresponding to the operation emotion probability value is set to 0.45, and the warning threshold corresponding to the facial emotion probability value is set to 0.35.

[0058] Specifically, in this embodiment, the scenario-based decision-making module analyzes the comprehensive emotion index based on the emotion probability value. The expression of the comprehensive emotion index is: E=wv×Pv+wo×Po+wf×Pf, where E represents the comprehensive emotion index, wv represents the audio weight, Pv represents the voice emotion probability value, wo represents the tactile weight, Po represents the operation emotion probability value, wf represents the visual weight, Pf represents the facial emotion probability value, and wv+wo+wf=1.

[0059] Specifically, in this embodiment, the audio weight is set to 0.4, the tactile weight is set to 0.4, and the visual weight is set to 0.2. Those skilled in the art can dynamically set the audio weight, tactile weight, and visual weight according to the driving environment. For example, if the camera is blocked or the lighting is extremely poor, resulting in a decrease in the reliability of the facial emotion probability value, the visual weight is reduced to 0.05, and the freed-up weight is proportionally added to the audio weight and tactile weight. If the vehicle is on a continuously bumpy road section with high steering wheel signal noise, the tactile weight is reduced to 0.2, and the weights of the audio weight and visual weight are increased. If the microphone detects continuous strong wind noise, the audio weight is reduced, etc. Normalization is performed after each adjustment to ensure that wv+wo+wf=1.

[0060] Specifically, in this embodiment, the scenario-based decision-making module determines the emotional state based on a comprehensive emotion index. If the comprehensive emotion index is less than 0.3, the emotional state is determined to be normal. If the comprehensive emotion index is greater than or equal to 0.3 and less than 0.6, the emotional state is determined to be level two abnormal. If the comprehensive emotion index is greater than or equal to 0.6, the emotional state is determined to be level three abnormal.

[0061] Specifically, in this embodiment, the scenario-based decision-making module classifies driving scenarios based on the probability value of emotions and the confidence level of the occupants' presence, classifying driving scenarios into in-vehicle social conflict, external traffic interference, and emotional fluctuations while driving alone.

[0062] Specifically, in this embodiment, the core judgment condition for the scenario-based decision-making module to set in-vehicle social conflict is: the occupant's confidence level is 1 and the probability value of voice emotion rises sharply in a short period of time, and the semantics in the audio data point to the occupants, such as recognizing personal pronouns such as "you" and "we" and conflicting content; the core judgment condition for setting external traffic interference is: the probability value of voice emotion increases and the semantics in the audio data point to external events, such as recognizing "car", "cutting off", "running a red light", etc., and typical defensive or aggressive operation characteristics appear during driving, such as the characteristic waveforms of sudden turns and sudden braking; the core judgment condition for setting emotional fluctuations when driving alone is: the occupant's confidence level is 0 and the emotional state is level two or three abnormal.

[0063] Specifically, the scenario-based decision-making module described in this embodiment achieves an intelligent leap from data to understanding by constructing a hierarchical and progressive decision-making model that includes "single-modal early warning, multi-modal dynamic weighted fusion, emotion level determination, and scenario classification." This module not only captures sudden emotional signals with high sensitivity but also adaptively responds to complex interference environments in real driving by introducing a dynamic weight adjustment mechanism based on signal credibility, thus ensuring the robustness and reliability of the final emotion index E. Crucially, this module is the first to deeply integrate occupant status with emotion cause analysis, enabling precise identification of the specific scenario of emotional outbursts (such as in-vehicle conflict or external interference). This provides indispensable and insightful decision-making basis for subsequent implementation of highly targeted intervention measures, enabling the system's decision-making to move from "knowing what" to "knowing why."

[0064] Please refer to Figure 1. The driving emotion intervention and control system based on multimodal perception also includes:

[0065] The safety execution module is used to execute safety intervention strategies based on the results of emotional state determination and driving scenario classification.

[0066] Specifically, in this embodiment, when the emotional state is at level two abnormality, the safety execution module combines the driving scenario and emotional state into a composite key, retrieves the local hierarchical voice template library, and initiates scenario-adaptive voice soothing. The local hierarchical voice template library is a structured voice template library, with templates indexed by "driving scenario × emotional state". For example, for "in-car social conflict - level two abnormality", the template "Arguments affect judgment, safety first, let's calm down, okay?" is played; for "external traffic interference - level two abnormality", the template "Others violate the rules, there are traffic regulations to handle it, you should keep a good mindset and a safe distance for safety" is played. At the same time, the media volume is automatically reduced and the interior lights are brightened to a soothing tone.

[0067] Specifically, in this embodiment, when the emotional state is at level three abnormal, and the passenger has a confidence level of 1, the safety execution module initiates a collaborative confirmation by clearly broadcasting through the vehicle's audio system: "To ensure driving safety, it is recommended to immediately proceed to the nearest service area for rest. Passenger, do you agree? Please say 'agree' or click the screen to confirm." Simultaneously, the central control screen displays alternative safe locations, including their name, distance, and estimated arrival time. A 10-second confirmation timeout is set. If confirmation is received within this time limit, the system executes auxiliary control, which involves navigating to the selected location and issuing a "Safety Escort" mode command via the CAN bus: limiting the maximum throttle opening to 70% and the maximum braking deceleration to 3.5. The system smooths the steering wheel angle rate using a speed of m / s². If confirmation is not received within the specified time or a rejection is received, the system will continuously provide a strong voice reminder, such as "Extremely high safety risk, we strongly recommend that you agree," and record the event. When the occupant's confidence level is 0, a mandatory takeover warning is issued, announcing: "A dangerous driving state has been detected, and the system will intervene to ensure safety." Mandatory safety measures are then implemented. After the warning, the system automatically takes over control, sends commands via the CAN bus to activate lane keeping assist, limits the steering wheel angle to ±15°, switches the powertrain to safety mode, disables accelerator pedal mapping, and takes over power requests to maintain the current speed or decelerate smoothly. The braking system enters standby mode, with the system controlling the deceleration to not exceed 3.0 m / s². The L2+ level autonomous driving system is activated, planning a route within 5 kilometers that is marked as safe. The system announces the current status and intentions every 30 seconds to alleviate driver anxiety.

[0068] Specifically, in this embodiment, the safety execution module continuously analyzes the comprehensive emotion index during the intervention execution. When the emotional state is level two abnormal and the comprehensive emotion index is less than 0.3 for 60 seconds, or when the emotional state is level three abnormal and the comprehensive emotion index is less than 0.3 for 180 seconds, the intervention is triggered to be lifted. When the intervention is lifted, the system announces "You have been calmed down. Do you want to exit the safety mode?". After receiving confirmation from the driver, the system restores control in stages and gradually. In the first 0-3 seconds, it restores all steering wheel steering angle permissions. In the 3rd-6th seconds, it restores all accelerator pedal travel mapping. After the 6th second, it exits the automatic driving mode, and the vehicle control is completely returned to the driver.

[0069] Specifically, the safety execution module in this embodiment proposes a layered, graded, and highly scenario-based safety intervention and recovery strategy, achieving a seamless connection and smooth transition from gentle persuasion to rigid protection. In Level 2 anomalies, its voice reassurance based on accurate scenario recognition directly addresses emotional pain points and provides effective psychological guidance. In Level 3 anomalies, it initiates "collaborative confirmation-based restricted takeover" and "mandatory safety protection-based takeover" depending on whether there are passengers in the vehicle, perfectly balancing the absolute bottom line of driving safety with the social attributes of interpersonal collaboration within the vehicle, avoiding secondary conflicts or psychological resistance that may be caused by forced intervention. Furthermore, the module's progressive control recovery mechanism ensures a smooth and natural handover of human-machine control, greatly improving the driver's acceptance and comfort after intervention, forming a complete and humanized safety closed loop of "perception-decision-intervention-removal."

[0070] In the above description, the disclosure of this invention is not intended to limit itself to these aspects. Rather, within the scope of the objectives of this disclosure, components can be selectively and operationally combined in any number. Furthermore, terms such as “comprising,” “encompassing,” and “having” should be interpreted by default as inclusive or open-ended, rather than exclusive or closed, unless explicitly defined as such. All technical, scientific, or other terms are to be understood by those skilled in the art, unless defined as such. Public terms found in dictionaries should not be interpreted in the context of the relevant technical documents in an overly idealistic or impractical manner, unless explicitly defined as such in this disclosure. Any modifications or alterations made by those skilled in the art based on the foregoing disclosure are within the scope of the claims.

Claims

1. A driving emotion intervention and control system based on multimodal perception, characterized in that, include: The data acquisition module is used to simultaneously acquire heterogeneous data from multiple sources. The edge analysis module is used to preprocess, extract features and perform parallel analysis on multi-source heterogeneous data to output sentiment probability values ​​and occupant presence confidence. The scenario-based decision-making module is used to build a hierarchical decision-making model based on the probability value of emotions and the confidence level of the presence of occupants, so as to complete the determination of emotional state and the classification of driving scenarios. The safety execution module is used to execute safety intervention strategies based on the results of emotional state determination and driving scenario classification.

2. The driving emotion intervention and control system based on multimodal perception according to claim 1, characterized in that, The edge analysis module includes: a voice analysis unit, used to construct a voice emotion analysis model based on audio data to analyze the probability value of voice emotion; an operation analysis unit, used to construct an operation emotion analysis model based on tactile data to analyze the probability value of operation emotion; a visual analysis unit, used to construct a facial emotion analysis model based on visual data to analyze the probability value of facial emotion; and an occupant judgment unit, used to construct an occupant analysis model based on visual and tactile data to analyze the confidence level of occupant presence.

3. The driving emotion intervention and control system based on multimodal perception according to claim 2, characterized in that, The speech analysis unit detects speech activity in the audio data, filters out non-human voices and environmental noise, preprocesses valid speech frames, and extracts key acoustic features that characterize emotions. These key acoustic features include Mel-frequency cepstral coefficients and their first and second-order differences. The key acoustic feature sequence of a continuous audio data segment is input into a deep temporal neural network for modeling to obtain a speech emotion analysis model. This model adopts a hybrid structure of convolutional neural networks and long short-term memory networks. The CNN layer is responsible for extracting timbre features from each frame's features, and the LSTM layer learns the evolution of speech emotions over time. Finally, a fully connected layer and a sigmoid activation function are connected in the speech emotion analysis model. The speech emotion analysis model outputs a continuous value in [0,1], which is used as the speech emotion probability value. A speech emotion probability value of 0 indicates extreme calmness, and a speech emotion probability value of 1 indicates extreme excitement.

4. The driving emotion intervention and control system based on multimodal perception according to claim 2, characterized in that, The operation analysis unit sets a time window of 5 seconds and extracts features from the tactile data within the window to obtain temporal features, frequency features, and morphological features. The temporal features include the mean, variance, peak-to-peak value, and number of overshoots of the tactile data. The frequency features are the proportion of signal energy in the 2-8Hz frequency band after performing a Fast Fourier Transform on the signal. The morphological features include the mean and variance of the first-order difference of the tactile data. The temporal features, frequency features, and morphological features are fused into an operation feature vector, which is then input into an ensemble learning classifier to obtain the operation emotion probability value.

5. The driving emotion intervention and control system based on multimodal perception according to claim 2, characterized in that, The visual analysis unit uses a face detection algorithm to locate the driver's facial region, standardizes the driver's facial region, and inputs it into a lightweight convolutional neural network. The lightweight convolutional neural network is trained on a facial expression recognition dataset, and the terminal outputs the probability distribution of facial expressions through a Softmax function. The probability value of the facial expression being classified as anger is taken as the probability value of the facial emotion.

6. The driving emotion intervention and control system based on multimodal perception according to claim 2, characterized in that, The occupant determination unit uses an improved lightweight YOLO target detection model to perform real-time inference on visual data, detects and outputs the bounding boxes of all "person" targets and their presence confidence scores, reads tactile data from the seat pressure sensor, and if the average pressure value in the seat area exceeds 10 kg for 3 consecutive seconds, a pressure flag is triggered. The occupant presence is determined based on the presence confidence score and the pressure flag. If the presence confidence score is greater than 0.8 and the pressure flag is triggered, a passenger is determined and the occupant presence confidence score is set to 1. If the presence confidence score is greater than 0.8 or the pressure flag is triggered, a redundancy check mechanism is triggered to perform a secondary determination based on the seat belt status. If the seat belt status is "fastened", the occupant presence confidence score is set to 1. If the seat belt status is "unfastened", the occupant presence confidence score is set to 0. If the presence confidence score is less than or equal to 0.8 and the pressure flag is not triggered, the occupant presence confidence score is set to 0.

7. The driving emotion intervention and control system based on multimodal perception according to claim 1, characterized in that, The scenario-based decision-making module sets an independent warning threshold for each emotion probability value. When any emotion probability value exceeds its corresponding warning threshold, an abnormal emotion warning is immediately triggered. The scenario-based decision-making module analyzes the comprehensive emotion index based on the emotion probability value and determines the emotional state based on the comprehensive emotion index. If the comprehensive emotion index is less than 0.3, the emotional state is determined to be normal. If the comprehensive emotion index is greater than or equal to 0.3 and less than 0.6, the emotional state is determined to be level two abnormal. If the comprehensive emotion index is greater than or equal to 0.6, the emotional state is determined to be level three abnormal.

8. The driving emotion intervention and control system based on multimodal perception according to claim 7, characterized in that, The scenario-based decision-making module categorizes driving scenarios based on the probability value of emotions and the confidence level of the occupants, classifying driving scenarios into in-vehicle social conflict, external traffic interference, and emotional fluctuations while driving alone.

9. The driving emotion intervention and control system based on multimodal perception according to claim 1, characterized in that, When the emotional state is at level two abnormality, the safety execution module combines the driving scenario and emotional state into a composite key, retrieves the local hierarchical voice template library, and initiates scenario-adaptive voice reassurance.

10. The driving emotion intervention and control system based on multimodal perception according to claim 9, characterized in that, When the emotional state is Level 3 abnormal, the safety execution module initiates collaborative confirmation when the occupant's confidence level is 1, setting a 10-second waiting timeout for confirmation. If consent is received within this time limit, the system performs auxiliary restricted takeover. When the occupant's confidence level is 0, a mandatory takeover warning is issued, and mandatory safety measures are implemented. After the warning, the system automatically takes over control. During the intervention, the safety execution module continuously analyzes the comprehensive emotional index. When the emotional state is Level 2 abnormal and the comprehensive emotional index remains below 0.3 for 60 seconds, or when the emotional state is Level 3 abnormal and the comprehensive emotional index remains below 0.3 for 180 seconds, the intervention is deactivated. Upon receiving confirmation from the driver, the system gradually restores control in stages.

Images