Intelligent intervention system and method based on multi-modal driving behavior analysis

Abstract

The invention discloses an intelligent intervention system and method based on multi-mode driving behavior analysis, and belongs to the field of intelligent traffic. The system comprises a multi-dimensional perception module, a behavior analysis engine, a risk assessment matrix and a self-adaptive intervention module. The multi-dimensional sensing module is integrated with a multi-source heterogeneous sensor and is used for acquiring physiological characteristics of a driver, driving operation data and environment information; the behavior analysis engine analyzes multi-modal data based on spatial-temporal feature fusion, and digs a driving behavior mode; the risk assessment matrix judges danger levels according to the multi-dimensional driving risk quantification result level by level; and the self-adaptive intervention module implements a grading progressive correction strategy. The system realizes active prevention and control of driving risks through cross-modal data association, behavior chain prediction and environment-behavior coupling analysis. The system solves the problems that a traditional system is single in sensing dimension, coarse in risk assessment, rigid in intervention strategy and the like, and effectively improves the driving safety.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation, and in particular to an intelligent intervention system and method based on multimodal driving behavior analysis. Background Art

[0002] With the rapid development of intelligent transportation technology, driver safety assistance systems are evolving from single-sensor warning systems to multimodal data fusion. In the early days, single-sensor technologies such as radar ranging and camera fatigue detection provided a certain level of driving safety by monitoring vehicle distance and the driver's facial state. However, existing technologies have many limitations.

[0003] Single-dimensional perception. Traditional systems rely heavily on visual or physical sensors, making it difficult to fully capture the dynamic correlation between the driver's physiological state (e.g., EEG, breathing), operational behavior (e.g., steering wheel grip), and environmental information. For example, visual sensors suffer from reduced accuracy in poor lighting or when obscured. Physical sensors, however, focus on vehicle status and are unable to integrate multimodal data, resulting in incomplete risk assessments.

[0004] Risk assessment is coarse-grained. Existing methods often determine risk levels based on a single metric (such as lane departure frequency) or static thresholds, failing to consider the spatiotemporal correlations and scenario adaptability of multi-dimensional anomalies. Driving risk characteristics vary significantly across different scenarios. For example, even a minor lane departure on a highway can lead to a serious accident, while a high frequency of lane departures in congested urban areas may not necessarily increase risk. Existing methods are unable to accurately capture these correlations, resulting in inaccurate assessment results.

[0005] Intervention strategies are rigid. Existing intervention methods are mostly fixed patterns (such as continuous alerts or emergency braking), lacking the ability to dynamically respond to evolving risks. Continuous alerts can easily disrupt driving and cause resistance, while emergency braking can be mistakenly triggered, leading to secondary accidents. Furthermore, these interventions lack the flexibility to adapt to risk levels and individual driver differences.

[0006] To address these issues, this paper proposes an intelligent intervention system based on multimodal driving behavior analysis. This system proactively controls driving risks through cross-modal data association, behavioral chain prediction, and environment-behavior coupling analysis. This system establishes a closed-loop framework encompassing multidimensional perception, feature analysis, risk quantification, and graded intervention, effectively addressing the shortcomings of existing technologies. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this application provides an intelligent intervention system and method based on multimodal driving behavior analysis. This system proactively controls driving risks through cross-modal data association, behavior chain prediction, and environment-behavior coupling analysis. It establishes a closed-loop framework of multidimensional perception, feature analysis, risk quantification, and graded intervention, effectively addressing the shortcomings of existing technologies.

[0008] To achieve the above objectives, this application provides the following solutions: An intelligent intervention system and method based on multimodal driving behavior analysis includes: a multi-dimensional perception module that integrates multi-source heterogeneous sensors inside and outside the cockpit to collect driver physiological characteristics, driving operation data, and environmental information; Behavior analysis engine, which integrates and analyzes multimodal data based on spatiotemporal features to mine driving behavior patterns; Risk assessment matrix, which classifies the danger level based on the quantitative calculation results of multi-dimensional driving risks; Adaptive intervention module, which implements graded and progressive correction strategies based on risk levels; The system achieves active prevention and control of driving risks through cross-modal data association, behavior chain prediction and environment-behavior coupling analysis.

[0009] Furthermore, the multi-dimensional perception module includes: a binocular infrared camera, a steering wheel tactile sensor, an on-board millimeter-wave radar, an environmental sensor, a breathing rate sensor, and a brain wave monitoring headband.

[0010] Binocular infrared cameras are used to monitor head posture, pupil status, and micro-expressions, with a frame rate of ≥60fps and support for low-light environments and sunglasses penetration; Steering wheel tactile sensor, based on piezoelectric film technology, detects grip force changes, vibration frequency and pressure distribution with a resolution of 0.1N; Vehicle-mounted millimeter-wave radar, operating at 77GHz, with a detection range of ≥200m and a penetration rate of ≥85% in rain and snow, is used to monitor following distance, relative speed, and lane deviation; Environmental sensors, including light and weather sensors, are used to collect various environmental information during driving in real time; The respiratory rate sensor uses a non-contact microwave radar with an error of ≤0.5 times / minute to monitor the chest fluctuation amplitude and respiratory cycle stability; EEG monitoring headband, dry electrode design, 1000Hz sampling rate, used to identify alpha / beta wave activity, attention index and early fatigue characteristics.

[0011] Furthermore, the behavior analysis engine includes: The feature engineering module constructs a 12-dimensional driving behavior feature vector, including head deviation angle, pupil closure, steering wheel grip fluctuation, accelerator pedal acceleration, lane keeping deviation, breathing interval duration, EEG concentration index, ambient light intensity, road curvature, traffic flow density, mobile phone interaction frequency, and voiceprint emotion parameters; Dynamic weight allocation mechanism automatically adjusts feature weights based on driving scenarios such as highway, urban, nighttime, and inclement weather; The hybrid model architecture combines the CNN-LSTM fusion network to process temporal action continuity, the graph neural network (GNN) to model multi-sensor spatial correlation, and simplifies the decision logic through knowledge distillation technology.

[0012] Furthermore, the risk assessment matrix establishes a four-level risk determination system: The comprehensive risk value R is calculated by the following formula: ; In the formula, wi is the dynamic weight of the i-th dimension feature, xi is the standardized feature value, and n is the number of feature dimensions involved in the calculation; The judgment rules are as follows: Level 1 risk: R≤T1 and environmental status index E≥0.8, where T1 is the level 1 risk threshold; Secondary risk: T1 <R≤T2且至少两维度异常关联度≥0.7,T2为二级风险阈值; Level 3 risk: T2 <R≤T3且三维度异常叠加熵值H≥0.5,T3为三级风险阈值; Level 4 risk: R>T3 and systemic abnormality index S≥0.9.

[0013] Furthermore, in the dynamic weight allocation mechanism, the function of the feature weight wi changing with the driving scene S is: wi(S)=wi0·(1+αi·ΔSi); Where wi0 is the benchmark weight, αi is the scene sensitivity coefficient, and ΔSi is the significance increment of the i-th dimension feature under scene S.

[0014] Furthermore, the adaptive intervention module includes: Level 1 warning: Multi-modal vibration reminder, which transmits warnings through local vibration coding of seats and steering wheels; Level 2 correction: The autonomous driving system temporarily takes over steering and braking control; Level 3 control: limit vehicle speed and activate vehicle-wide safety warning (flashing lights + high-pitched alarm); Level 4 response: Initiate the emergency designated driver agreement, automatically contact the nearest service area or trigger roadside assistance.

[0015] Furthermore, the adaptive intervention module further includes: Attention resource competition mechanism, when mobile phone use is detected, reverse tactile interference is applied through steering wheel resistance pulses; Neurofeedback training, which uses brain-computer interface to conduct attention-enhancing training and enhance beta wave feedback; Environmentally adaptive reminders dynamically adjust the warning volume based on noise levels and switch to visual warnings in scenes such as airport runways.

[0016] Furthermore, an intelligent intervention method based on multimodal driving behavior analysis is characterized by comprising the following steps: Step 1: Data acquisition: acquiring raw driving behavior data through the multi-dimensional perception module; Step 2: Feature fusion: preset feature construction rules, extract 12-dimensional driving behavior feature vectors from the original data, and define the 12-dimensional driving behavior features; Step 3: Execute a dynamic weight allocation mechanism; use the behavior analysis engine to extract spatiotemporal features and dynamically allocate weights to the multimodal data, identify the current driving scenario, calculate dynamic weights, and output a weighted 12-dimensional feature vector; Step 4: Hybrid model construction and processing; use CNN-LSTM fusion network and graph neural network GNN to build a driving mode feature model, build a lightweight decision model through knowledge distillation technology, and output driving behavior patterns and risk assessment results.

[0017] Step 5: Establish a risk assessment matrix and classify driving behavior into four risk levels; Step 6: Risk level determination: Based on the driving behavior pattern and risk assessment results, the driving behavior pattern risk level is determined; Step 7: Graded intervention; triggering an adaptive intervention strategy based on the risk level of the driving behavior pattern and adjusting the intervention intensity in real time.

[0018] Furthermore, it also includes: Behavior chain prediction: using a CNN-LSTM fusion network to predict potentially dangerous operations 800ms in advance; Environment-behavior coupling analysis, dynamically adjusting the judgment threshold based on the risk assessment model of 16 typical scenarios.

[0019] Beneficial effects of the technical solution of this application: The system integrates multiple sensors to comprehensively collect the driver's physiological characteristics, driving operation data, and environmental information, overcoming the limitations of traditional single sensors and achieving more comprehensive risk perception. It also integrates and analyzes multimodal data based on spatiotemporal characteristics to construct a multidimensional driving behavior feature vector and employs a dynamic weighting mechanism. Combined with a four-level risk assessment system, the system accurately assesses driving risks, avoids misjudgments and missed detections, and improves the accuracy of risk warnings.

[0020] The system implements a graded and progressive intervention strategy based on risk level, encompassing four levels of intervention, from multimodal vibration alerts to emergency designated drivers. Combining attention resource competition with neurofeedback training, the system reduces distractions and adapts to individual differences, ensuring precise and appropriate intervention measures in different risk scenarios, effectively reducing driving risk. Leveraging a CNN-LSTM fusion network, the system predicts potentially dangerous maneuvers in advance and dynamically adjusts judgment thresholds based on a risk assessment model for 16 typical scenarios, improving reliability in complex environments and enabling proactive control of driving risks and early accident prevention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is an overall architecture diagram of an intelligent intervention system based on multimodal driving behavior analysis; Figure 2 It is a flow chart of the behavior parsing engine of the intelligent intervention system based on multimodal driving behavior analysis; Figure 3 Flowchart of an intelligent intervention method based on multimodal driving behavior analysis. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0024] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0025] The intelligent intervention system based on multimodal driving behavior analysis in this paper primarily consists of the following modules: a multidimensional perception module, a behavior analysis engine, a risk assessment matrix, and an adaptive intervention module. These modules collaborate to achieve comprehensive monitoring, accurate assessment, and intelligent intervention of driving behavior.

[0026] The multi-dimensional perception module includes binocular infrared cameras, a steering wheel tactile sensor, an onboard millimeter-wave radar, a respiratory rate sensor, and an EEG monitoring headband. The binocular infrared cameras are mounted above the instrument panel directly in front of the driver and on the right A-pillar, ensuring comprehensive capture of the driver's head posture, pupil state, and micro-expressions. Utilizing infrared imaging technology, the binocular infrared cameras achieve clear images in low-light conditions with a frame rate of ≥60 fps. Using binocular vision, they accurately measure the driver's three-dimensional head posture, including pitch, yaw, and roll angles. Furthermore, a pupil detection algorithm monitors pupil size and closure in real time, capturing subtle micro-expressions such as blinking and frowning. Furthermore, the camera features a sunglasses-penetrating function, enabling accurate eye monitoring even when the driver is wearing sunglasses. Output data, including three-dimensional head posture angles, pupil diameter, pupil closure frequency, and micro-expression feature vectors, is transmitted to the behavior analysis engine at a high frequency and in real time.

[0027] Steering wheel tactile sensors are embedded in the inner and outer surfaces of the steering wheel, covering the driver's usual grip positions. Based on piezoelectric film technology, the sensors sense pressure changes when the driver's hands make contact with the steering wheel and convert these pressure signals into electrical signals. High-precision signal processing circuitry enables precise measurement of grip force, vibration frequency, and pressure distribution with a resolution of 0.1N. Output data includes real-time steering wheel grip force, grip force change rate, vibration frequency, and pressure distribution, enabling analysis of driver anxiety, operational stability, and concentration.

[0028] The onboard millimeter-wave radar is installed at the front of the vehicle, and its radar beam covers the road ahead. It offers high resolution and strong anti-interference capabilities, operating at a frequency of 77 GHz, with a detection range of 200 meters or greater. Even in adverse weather conditions such as rain and snow, its penetration rate reaches 85% or greater. Output data includes the following distance, relative speed, and lane deviation angle of the vehicle ahead, providing crucial traffic environment information for driving behavior analysis.

[0029] Environmental sensors include light sensors and meteorological sensors, which are used to collect real-time information such as light intensity, temperature, humidity, rainfall, and wind speed in the driving environment.

[0030] The respiratory rate sensor is mounted on the driver's seatback or seatbelt, in contact with the driver's back. Using non-contact microwave radar technology, it measures respiratory rate and respiratory cycle stability by detecting the rise and fall of the driver's chest. The sensor accurately captures minute displacement changes, with an error of ≤0.5 breaths per minute. Output data, including respiratory rate and respiratory cycle stability indicators, is used to assess the driver's physiological state and fatigue level.

[0031] The driver wears an EEG monitoring headband, with electrodes in close contact with the scalp. Using a dry electrode design, the system collects EEG signals and processes them through amplification and filtering to extract EEG characteristics such as alpha and beta waves. With a sampling rate of up to 1000Hz, it can monitor EEG activity in real time and analyze the driver's attention index and early signs of fatigue. Output data includes the power spectral density of alpha and beta waves, attention index values, and fatigue characteristic parameters, providing important physiological evidence for driving behavior analysis.

[0032] Based on data collected by the multi-dimensional perception module, a 12-dimensional driving behavior feature vector is constructed. Specifically, it includes head deviation angle, pupil closure, steering wheel grip force fluctuation, accelerator pedal acceleration, lane keeping deviation, breathing interval duration, EEG concentration index, ambient light intensity, road curvature, traffic density, mobile phone interaction frequency, and voiceprint emotion parameters. For each feature, corresponding signal processing and feature extraction algorithms are employed. For example, for head deviation angle, the head's deviation angle relative to the forward direction is calculated using the three-dimensional posture angle data output by the binocular infrared camera. For steering wheel grip force fluctuation, the standard deviation of the grip force change rate is extracted as a fluctuation feature by performing time series analysis on the grip force data output by the steering wheel tactile sensor. The extracted feature values ​​are normalized to the range [0, 1] to facilitate subsequent fusion analysis and model training. Data such as road curvature, traffic density, and ambient light intensity collected by the on-board millimeter-wave radar and driving environment sensors are combined with vehicle speed and navigation information to identify the current driving scenario in real time, such as highway, urban, nighttime, and inclement weather. The feature engineering module automatically adjusts the weights of each feature based on different driving scenarios. The weight adjustment function is: wi(S) = wi0·(1+αi·ΔSi). Here, wi0 is the baseline weight, αi is the scenario sensitivity coefficient, and ΔSi is the significance increment of the i-th dimension feature under scenario S. The scenario sensitivity coefficient and significance increment are derived from statistical analysis of a large amount of driving data and reflect the impact of each feature on driving risk in different scenarios. The adjusted weights are normalized to ensure that the sum of all feature weights is 1, thereby ensuring fair and accurate risk assessment.

[0033] A CNN-LSTM fusion network is used to process temporal action continuity. First, a convolutional neural network (CNN) is used to extract spatial features from multimodal data to obtain a spatial feature map. LSTM effectively captures the temporal dependencies of driving behavior, such as the driver's continuous head movements and steering wheel manipulation, thereby accurately modeling the dynamic changes in driving behavior. The CNN-LSTM fusion network generates joint features through feature concatenation. A graph neural network (GNN) is used to model multi-sensor spatial correlation features. Sensors in the multidimensional perception module are considered nodes in a graph, with the physical location relationships and data correlations between sensors as edges. The GNN learns from the graph-structured data and mines spatial correlation features between different sensor data, such as the potential correlation between steering wheel grip and breathing rate, further enriching the feature representation of driving behavior. The spatiotemporal joint features (T × D1) extracted by the CNN-LSTM and the spatial correlation features (N × D2) generated by the GNN are projected into the same dimensional space through a fully connected layer. The two types of features are then concatenated along the feature dimension to obtain the final feature. The final features include the spatiotemporal evolution of driving actions and multi-sensor coordination patterns, providing a comprehensive representation for subsequent decision-making models.

[0034] To simplify decision-making logic and improve the real-time performance and interpretability of the decision-making model, knowledge distillation technology is used to extract knowledge from the complex hybrid model architecture and train a lightweight decision-making model. This decision-making model can quickly respond to changes in driving behavior while maintaining high accuracy, and output driving behavior patterns and risk assessment results in real time.

[0035] Based on the 12 driving behavior characteristics and dynamic weights output by the behavior analysis engine, the comprehensive risk value R is calculated. The calculation formula is: , where wi is the dynamic weight of the i-th dimension feature, xi is the standardized eigenvalue, and n is the number of feature dimensions involved in the calculation.

[0036] Four-level risk assessment system: Driving risks are divided into four levels based on the comprehensive risk value R and related conditions.

[0037] Level 1 risk: R ≤ T1 and environmental index E ≥ 0.8, where T1 is the level 1 risk threshold. For example, if a driver looks down at their phone for more than 2 seconds, but the surrounding environment is good and traffic conditions are simple, this is considered a level 1 risk.

[0038] Secondary risk: T1 <R≤T2且至少两维度异常关联度≥0.7,T2为二级风险阈值。例如,驾驶员打哈欠且车速波动超过15km / h,这两个异常行为之间存在较高的关联度,判定为二级风险。

[0039] Level 3 risk: T2 < R ≤ T3 and three-dimensional abnormality stacking entropy H ≥ 0.5, T3 is the threshold of level 3 risk. For example, the driver closes his eyes, deviates from the lane, and breathes rapidly, and these three abnormal behaviors are stacked, which is determined as level 3 risk.

[0040] Level 4 risk: R > T3 and systemic abnormality index S ≥ 0.9. For example, the driver's brain fatigue index exceeds the threshold, the environment is bad (such as heavy rain), and the dangerous operation (such as deviating from the lane for a long time) continues for more than 5 seconds, which is determined as level 4 risk.

[0041] The parameters such as T1, T2, T3 and the threshold of systemic abnormality index are set through statistical analysis of a large amount of driving data and expert experience, and can be fine-tuned according to actual driving scenarios and user needs.

[0042] In the design of the adaptive intervention module, when the risk level is level 1, warning information is transmitted through local vibration encoding of the seat and steering wheel. The specific implementation is as follows: miniature vibration motors are installed on the seat backrest and cushion positions and the inner circle of the steering wheel. According to the specific type and severity of the risk, different vibration patterns and intensities are designed. For example, for the abnormal behavior of looking at the mobile phone, the seat backrest produces slight up-and-down vibration with a frequency of 2 Hz and a duration of 3 seconds; for speed fluctuation abnormality, the steering wheel produces left-and-right vibration with a frequency of 3 Hz and a duration of 2 seconds. The vibration patterns are combined through pre-set coding rules, so that the driver can quickly identify the risk type and take appropriate measures.

[0043] When the risk level is level 2, the automatic driving system temporarily takes over part of the control function of the vehicle according to the specific situation of the risk. For example, when yawning and speed fluctuation abnormalities are detected, the automatic driving system automatically adjusts the steering angle of the vehicle to keep it near the center line of the lane, and appropriately reduces the speed to avoid collision risk caused by the driver's inattention. The takeover process is smoothly transitioned to ensure driving safety while minimizing interference with normal driving of the driver.

[0044] When the risk level is level 3, the system takes more stringent control measures. First, through the engine management system and braking system of the vehicle, the speed is limited within a safe range, such as reducing the speed to below 60 km / h (which can be adjusted according to the specific road speed limit and safety requirements). At the same time, the full-vehicle safety warning system is turned on, including flashing lights and high-pitched alarms. The flashing lights flash at a frequency of 5 Hz for 10 seconds; the volume of the high-pitched alarm is dynamically adjusted according to the ambient noise level to ensure that the driver can clearly hear it, with a volume range of 80-100 decibels. Through these measures, the driver is reminded to correct dangerous behavior in a timely manner, and warning is given to surrounding vehicles and pedestrians to avoid serious traffic accidents.

[0045] When the risk level reaches Level 4, the system determines that the driving environment and driver status are extremely dangerous, beyond the driver's control. At this point, the system automatically initiates the emergency designated driver protocol and sends a designated driver request message to the nearest service area through the vehicle's communication module, including detailed information such as the vehicle's current location, direction of travel, and estimated time of arrival. Simultaneously, the roadside assistance system is triggered, sending a distress signal to nearby rescue agencies, ensuring timely professional assistance in an emergency and protecting the safety of the driver and vehicle.

[0046] The adaptive intervention module also includes mobile phone usage detection. A binocular infrared camera monitors the driver's head posture and hand movements, combined with the vehicle's driving status detected by the onboard millimeter-wave radar, to determine whether the driver is using a mobile phone. For example, if the driver's head is frequently turned with their hands close to their face, and the vehicle's driving speed fluctuates abnormally, it is determined to be mobile phone use. When mobile phone use is detected, reverse tactile interference is applied via steering wheel resistance pulses. This is achieved by installing a micromotor and resistance adjustment device inside the steering wheel. When mobile phone use is detected, the micromotor activates, generating a resistance pulse of a certain intensity, causing the steering wheel to produce a brief reverse resistance at a frequency of 1Hz and a duration of 5 seconds. This reverse resistance effectively interferes with the driver's hand operation, prompting the driver to refocus on the driving task, reducing the risk of distraction caused by mobile phone use.

[0047] For neurofeedback training, the driver wears a brainwave monitoring headband, which transmits brainwave signals to the training system via brain-computer interface technology. The training system then designs a personalized attention-enhancing training program based on the driver's brainwave characteristics. For example, if the driver's beta wave activity is detected to be low and their attention is not focused, the training system uses multiple sensory stimulations, such as vision and hearing, to guide the driver in attention training. During training, the system monitors brainwave changes in real time. When beta wave activity increases, positive feedback is provided, such as playing soothing music or displaying encouraging messages. When beta wave activity decreases, appropriate reminders are given or the training difficulty is adjusted. Through long-term neurofeedback training, the driver's attention and fatigue resistance are enhanced, reducing driving risks.

[0048] The vehicle's internal noise sensors monitor the noise level in the cockpit in real time. The warning volume is dynamically adjusted based on the noise level. When the noise level is low, the warning volume is appropriately reduced to avoid unnecessary interference to the driver; when the noise level is high, the warning volume is increased to ensure the driver can clearly hear the warning message. For example, when the noise level is between 40-60 decibels, the warning volume is set to 70 decibels; when the noise level exceeds 70 decibels, the warning volume is increased to 85 decibels. In special scenarios such as airport runways, the warning method is switched to visual warnings, taking into account the unique noise environment and the driver's need for concentration. For example, near airport runways, the system automatically silences the high-pitched alarm and displays a striking red warning sign or text message on the vehicle's central control display, instrument panel, etc., reminding the driver to pay attention and prevent the warning message from being overlooked due to the complex noise environment.

[0049] The specific implementation steps of this system's intelligent intervention method include: Step 1: Data acquisition; The multi-dimensional perception module is activated, including a binocular infrared camera, steering wheel tactile sensor, onboard millimeter-wave radar, respiratory rate sensor, and brainwave monitoring headband. Each sensor, according to preset sampling frequencies and parameters, collects 12-dimensional feature vectors in real time, including the driver's physiological characteristics (such as brainwaves and respiratory rate), driving operation data (such as steering wheel grip force and accelerator pedal acceleration), and environmental information (such as following distance and road curvature). These 12-dimensional driving behavior characteristics are defined and transmitted to the behavior analysis engine via the vehicle's communication network, providing data support for subsequent feature fusion and risk assessment.

[0050] Step 2: feature fusion; After receiving data from the multi-dimensional perception module, the behavior analysis engine first performs feature engineering. Based on pre-set feature construction rules, it extracts a 12-dimensional driving behavior feature vector from the raw data.

[0051] Step 3: Activate the dynamic weighting mechanism to identify the current driving scenario in real time and adjust the weights of each feature accordingly. The behavior parsing engine extracts spatiotemporal features from the multimodal data and dynamically assigns weights. This identifies the current driving scenario, calculates dynamic weights, and outputs a weighted 12-dimensional feature vector. This 12-dimensional weighted feature vector is then fed into the hybrid model architecture.

[0052] Step 4: A CNN-LSTM fusion network extracts temporal features from feature vectors and models action sequences; a graph neural network (GNN) models the spatial correlations among multiple sensors. Finally, knowledge distillation technology simplifies decision logic and outputs driving behavior patterns and risk assessment results.

[0053] Step 5: Establish a risk assessment matrix; The comprehensive risk value R is calculated, and the driving behavior is classified into four risk levels in combination with the comprehensive risk value R and related conditions to establish a judgment rule of the risk assessment matrix.

[0054] Step 6: Risk level judgment; Based on the driving behavior mode and risk assessment result output by the hybrid model architecture, the current driving risk level is determined according to the judgment rule of the risk assessment matrix, which provides a basis for subsequent hierarchical intervention. It is judged whether the risk assessment result is greater than the set threshold. If it is greater, hierarchical intervention is performed; if the result is uncertain, it is marked as “uncertain state” and the hybrid model processing is repeated.

[0055] Step 7: Hierarchical intervention; According to the determined risk level, the adaptive intervention module triggers the corresponding intervention strategy. If it is a first-level risk, a multi-modal vibration reminder is performed to transmit warning information through local vibration encoding of the seat and steering wheel; if it is a second-level risk, the automatic driving system temporarily takes over the steering and braking control to ensure the safety of the vehicle; if it is a third-level risk, the vehicle speed is limited and the whole vehicle safety warning (flashing light + high-pitched alarm) is turned on to remind the driver and surrounding traffic participants to pay attention to safety; If it is a fourth-level risk, an emergency driving service agreement is started, and the nearest service area or road rescue is triggered to ensure the safety of the driver and the vehicle.

[0056] During the intervention process, the changes in driving behavior are monitored in real time. If the risk level decreases, the intervention intensity is adjusted in time; if the risk level increases, higher-level intervention measures are taken to ensure the effectiveness and timeliness of the intervention.

[0057] In the behavior chain prediction, the driver's physiological, behavioral and environmental data collected by the multi-dimensional perception module are input into the CNN-LSTM fusion network. A large amount of driving behavior data is used to train the CNN-LSTM fusion network, so that the model can learn the time sequence characteristics and potential laws of driving behavior. During the training process, the labeled dangerous operation behavior data (such as sudden braking, sharp turning, sudden lane changing, etc.) is used as a supervision signal to optimize the parameters of the model and improve the prediction ability of the model for potential dangerous operations. In actual driving process, the model receives the current driving behavior data in real time, and through the learned time sequence characteristics and laws, it predicts the potential dangerous operations that the driver may perform 800ms in advance. For example, when the driver's head is gradually tilted to the right and the steering wheel has a slight right turning trend, the model predicts that the driver may soon perform a dangerous lane changing operation, and sends an early warning signal to the adaptive intervention module in time to take intervention measures in advance to avoid accidents.

[0058] In the environment-behavior coupling analysis, based on data collected by driving environment sensors, such as road curvature, traffic flow density, and ambient light intensity, combined with vehicle speed and navigation information, 16 typical driving scenarios were identified, such as nighttime driving, highway congestion, driving in rainy weather, and mountainous roads. A corresponding risk assessment model was developed for each typical scenario. This model comprehensively considers the driving environment characteristics and driver behavior characteristics of the scenario, and is trained using a large amount of driving data. For example, in the nighttime driving scenario, the model focuses on the impact of factors such as driver fatigue (such as brain wave fatigue characteristics and respiratory rate changes), vehicle speed control (such as accelerator pedal acceleration and following distance), and road environment (such as road curvature and lighting conditions) on driving risk.

[0059] During actual driving, the thresholds in the risk assessment matrix are dynamically adjusted based on real-time driving scenarios. For example, in congested highway scenarios, due to lower vehicle speeds and closer following distances, the threshold for lane departure frequency can be relaxed, while the thresholds for driver fatigue and concentration may need to be more stringent. This dynamic adjustment mechanism allows risk assessment to more accurately adapt to the risk characteristics of different driving scenarios, improving the overall performance and reliability of the system.

[0060] Take the "nighttime fatigue driving" scenario as an example: Dynamic weight allocation: Environmental state S: night; Significant increment: ΔS pupil closure = 0.3, indicating that the impact of pupil closure on driving risk when driving at night is considered to be 30% higher than during the day.

[0061] Pupil closure weight: The baseline weight is 0.1, the scene sensitivity coefficient is 0.5, and according to the formula wi(S)=wi0·(1+αi·ΔSi), the pupil closure weight is adjusted to wi(S)=0.1·(1+0.5·0.3)=0.115 in the night driving scene.

[0062] 2. Hybrid model processing: CNN extracts the spatial relationship between head posture and lane deviation; LSTM identifies the temporal pattern of multiple consecutive eye closures; and GNN correlates the abnormal synchronization of pupil closure and respiratory rate to analyze the driver's physiological state and fatigue level.

[0063] 3. Comprehensive risk assessment: The calculated comprehensive risk value R is 0.85, the environmental state index E is 0.9, and the abnormal correlation between pupil closure and respiratory rate is 0.8. Based on the risk assessment matrix, the current driving behavior is determined to be a level 2 risk.

[0064] 4. Graded intervention: Based on the identified secondary risk, the adaptive intervention module triggers the appropriate intervention strategy: the automated driving system temporarily takes over steering and braking control. The system automatically adjusts the vehicle's steering angle to maintain it near the lane centerline and reduces speed appropriately to avoid collision risks caused by driver fatigue.

[0065] The present invention's intelligent intervention system and method, based on multimodal driving behavior analysis, comprehensively collects the driver's physiological, behavioral, and environmental data through a multidimensional perception module. A behavioral analysis engine performs spatiotemporal feature fusion analysis on this multimodal data to accurately identify driving behavior patterns. A risk assessment matrix categorizes risk levels based on the multidimensional driving risk quantification calculations. An adaptive intervention module implements a graded and progressive corrective strategy based on risk levels. The system also incorporates functions such as an attention resource competition mechanism, neural feedback training, and environmental adaptive reminders. Through behavioral chain prediction and environment-behavior coupling analysis, it enables proactive prevention and control of driving risks. This system can effectively address the limitations of existing driving safety assistance technologies, improve driving safety, and reduce the incidence of traffic accidents. It has broad application prospects and significant social value.

[0066] It is worth noting that this embodiment only shows a layout form of an intelligent intervention system and method based on multimodal driving behavior analysis, but it does not mean that the intelligent intervention system and method based on multimodal driving behavior analysis provided by this application only has this form. The specific structural form can be adjusted according to the specific design.

[0067] In the description of the present application, it should be understood that the terms "front", "back", "up", "down", "outside", "inside", "horizontal", "top", "bottom", "surface", "bottom layer", "top layer", "upper", "lower", "bottom", "top", "inside", "surface", "center", "right", "middle" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0068] The embodiments described above are merely descriptions of the preferred methods of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by those skilled in the art should fall within the scope of protection determined by the claims of the present application.

Claims

1. An intelligent intervention system based on multimodal driving behavior analysis, characterized in that: It includes: A multi-dimensional perception module that integrates multi-source heterogeneous sensors inside and outside the cockpit to collect driver physiological characteristics, driving operation data, and environmental information; A behavior analysis engine that performs fusion analysis on multi-modal data based on spatio-temporal characteristics to mine driving behavior patterns; A risk assessment matrix that classifies and determines the danger level according to the multi-dimensional driving risk quantification calculation results; An adaptive intervention module that implements a hierarchical progressive correction strategy according to the risk level; The system realizes the active prevention and control of driving risks through cross-modal data association, behavior chain prediction, and environment-behavior coupling analysis.

2. The intelligent intervention system based on multimodal driving behavior analysis according to claim 1, characterized in that: The multi-dimensional perception module includes: a binocular infrared camera, a steering wheel tactile sensor, an in-vehicle millimeter wave radar, an environmental sensor, a respiratory frequency sensor, and an electroencephalogram monitoring headband; The binocular infrared camera is used to monitor head posture, pupil state, and micro-expression; The steering wheel tactile sensor detects grip force changes, vibration frequency, and pressure distribution based on piezoelectric film technology; The in-vehicle millimeter wave radar is used to monitor following distance, relative speed, and lane deviation; The environmental sensor includes a light and meteorological sensor and is used to collect various environmental information during driving in real time; The respiratory frequency sensor uses a non-contact microwave radar to monitor the chest fluctuation amplitude and respiratory cycle stability; The electroencephalogram monitoring headband is used to identify the activity of α / β waves, attention index, and early fatigue characteristics.

3. The intelligent intervention system based on multimodal driving behavior analysis according to claim 1, characterized in that: The behavior analysis engine includes: A feature engineering module that constructs a 12-dimensional driving behavior feature vector, including head deviation angle, pupil closure degree, steering wheel grip force fluctuation, accelerator pedal acceleration, lane keeping deviation, respiratory interval duration, electroencephalogram concentration index, environmental light intensity, road curvature, traffic flow density, mobile phone interaction frequency, and voiceprint emotion parameters; A dynamic weight allocation mechanism that automatically adjusts feature weights according to driving scenarios such as highway, urban area, night, and bad weather; A hybrid model architecture that combines a CNN-LSTM fusion network to process the continuity of temporal actions, uses a graph neural network GNN to model the spatial correlation of multi-sensors, and simplifies the decision logic through knowledge distillation technology.

4. The intelligent intervention system based on multimodal driving behavior analysis according to claim 3 is characterized in that: The risk assessment matrix establishes a four-level risk determination system: The comprehensive risk value R is calculated by the following formula: ; where wi is the dynamic weight of the i-th dimension feature, xi is the standardized feature value, and n is the number of feature dimensions participating in the calculation; The determination rules are as follows: Level 1 risk: R ≤ T1 and the environmental state index E ≥ 0.8, where T1 is the threshold of level 1 risk; Level 2 risk: T1 < R ≤ T2 and the abnormal correlation degree of at least two dimensions ≥ 0.7, T2 is the threshold of level 2 risk; Level 3 risk: T2 < R ≤ T and the entropy value H of the abnormal superposition of three dimensions ≥ 0.5, T3 is the threshold of level 3 risk; Level 4 risk: R > T3 and the systematic abnormal index S ≥ 0.

9.

5. The intelligent intervention system based on multimodal driving behavior analysis according to claim 4 is characterized in that: In the dynamic weight allocation mechanism, the function of the feature weight wi changing with the driving scenario S is: wi(S) = wi0·(1 + αi·ΔSi); where wi0 is the reference weight, αi is the scenario sensitivity coefficient, and ΔSi is the significance increment of the i-th dimension feature under the scenario S.

6. The intelligent intervention system based on multimodal driving behavior analysis according to claim 1, characterized in that: The adaptive intervention module includes: Level 1 warning: Multi-modal vibration reminder, which transmits warnings through local vibration coding of the seat and steering wheel; Level 2 correction: The autonomous driving system temporarily takes over steering and braking control; Level 3 control: limit vehicle speed and activate vehicle-wide safety warning; Level 4 response: Initiate the emergency designated driver agreement, automatically contact the nearest service area or trigger roadside assistance.

7. The intelligent intervention system based on multimodal driving behavior analysis according to claim 1, characterized in that: The adaptive intervention module further includes: Attention resource competition mechanism, when mobile phone use is detected, reverse tactile interference is applied through steering wheel resistance pulses; Neurofeedback training, which uses brain-computer interface to conduct attention-enhancing training and enhance beta wave feedback; Environmentally adaptive reminders dynamically adjust the warning volume based on noise levels and switch to visual warnings in scenes such as airport runways.

8. An intelligent intervention method based on multimodal driving behavior analysis, characterized in that: The following steps are involved: Step 1: Data acquisition: acquiring raw driving behavior data through the multi-dimensional perception module; Step 2: feature fusion; Preset feature construction rules, extract 12-dimensional driving behavior feature vectors from raw data, and define 12-dimensional driving behavior features; Step 3: Execute a dynamic weight allocation mechanism; use the behavior analysis engine to extract spatiotemporal features and dynamically allocate weights to the multimodal data, identify the current driving scenario, calculate dynamic weights, and output a weighted 12-dimensional feature vector; Step 4: Hybrid model construction and processing: Using a CNN-LSTM fusion network and a graph neural network (GNN) to build a driving pattern feature model, and using knowledge distillation technology to build a lightweight decision-making model to output driving behavior patterns and risk assessment results. Step 5: Establish a risk assessment matrix; Classify driving behavior into four risk levels; Step 6: Risk level determination; Based on the driving behavior pattern and risk assessment results, the driving behavior pattern risk level is determined; Step 7: Tiered intervention; Trigger an adaptive intervention strategy based on the risk level of the driving behavior pattern and adjust the intervention intensity in real time.

9. The intelligent intervention method based on multimodal driving behavior analysis according to claim 8, characterized in that: Also includes: Behavior chain prediction: using a CNN-LSTM fusion network to predict potentially dangerous operations 800ms in advance; Environment-behavior coupling analysis, dynamically adjusting the judgment threshold based on the risk assessment model of 16 typical scenarios.

Images