A method and system for proactive intervention in driver safety risks based on multimodal perception

By integrating multimodal perception and deep learning models with dynamic adjustments based on individualized physiological characteristics, we have achieved accurate assessment and progressive intervention of driver safety risks. This solves the problems of incomplete data modalities, coarse modeling, and insufficient progressive intervention in existing technologies, thereby improving driving safety.

CN123104664APending Publication Date: 2026-11-17BEIJING BAILONG MAYUN TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611090347.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-22
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies for driver safety risk management suffer from incomplete data modality coverage, crude risk modeling, lack of personalized adaptation, and insufficient progressiveness of intervention measures, leading to frequent false alarms, missed alarms, and passive interventions.

Method used

By acquiring driver physiological state, vehicle dynamics and external environment data through multimodal perception, feature-level fusion is performed using a multimodal deep learning fusion model, and risk thresholds are dynamically adjusted in combination with individualized physiological characteristics to execute progressive intervention strategies, including sensory alerts, mandatory rest and emergency response.

Benefits of technology

It enables accurate assessment of driving risks and timely and precise proactive intervention, significantly reducing false alarm and missed alarm rates, providing closed-loop risk management throughout the entire process, and ensuring driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123104664A_ABST
    Figure CN123104664A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for proactive intervention in driver safety risks based on multimodal perception, belonging to the field of intelligent connected vehicle safety technology. It aims to address the problems of existing technologies, such as single risk assessment modality, coarse models, lack of individualized adaptation, and passive intervention. The method includes: acquiring first-modal data reflecting the driver's physiological state, second-modal data reflecting vehicle dynamics, and third-modal data reflecting external driving environment risks; performing feature-level fusion through a multimodal deep learning fusion model to generate risk indicators characterizing comprehensive driving risks; dynamically adjusting the risk threshold used to trigger intervention based on the driver's physiological characteristics; and executing a progressive intervention strategy based on the comparison results between the risk indicators and the dynamic threshold. This invention, through multimodal fusion, personalized threshold adjustment, and progressive intervention, achieves comprehensive, accurate, and proactive management of driving risks, significantly improving driving safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle safety technology, specifically to a method and system for proactive intervention in driver safety risks based on multimodal perception. Background Technology

[0002] With the development of intelligent and connected vehicles, driving safety technology has received widespread attention. Especially in operational scenarios such as ride-hailing and long-haul freight, the driver's physiological and psychological state directly affects driving safety. Therefore, developing systems capable of real-time monitoring of driver status, assessing driving risks, and proactively intervening is of great significance for preventing traffic accidents.

[0003] In existing technologies, some solutions attempt to assess risk by collecting data from drivers or vehicles. For example, Chinese patent CN121572988A discloses an intelligent driving assistance system based on multimodal emotion recognition. This system identifies driver emotions by collecting facial images, biosignals, and vehicle operation data, and calculates an emotion risk index. However, this solution mainly focuses on emotion recognition, and its input modalities fail to cover external environmental risks that can be provided by high-precision maps. Its risk model is also relatively simple, and its intervention strategies lack clear grading and progression.

[0004] Another type of solution, such as the one disclosed in Chinese patent CN120766476A, proposes an intelligent intervention system based on multi-dimensional perception and behavior analysis. This system constructs a risk assessment matrix and performs adaptive intervention. However, the dimensions it perceives do not explicitly include data reflecting the driver's physiological state or data reflecting the risks of the external driving environment. Its risk assessment method leans more towards traditional rule engines than deep feature fusion using deep learning models. Furthermore, most of these solutions use fixed risk thresholds, making them unable to adapt to individual differences among drivers.

[0005] In summary, existing technologies have the following shortcomings in driver safety risk management: First, the data modality coverage is incomplete, often neglecting the comprehensive consideration of driver status, vehicle dynamics, and external environmental risks; second, the risk modeling methods are relatively crude, making it difficult to capture the complex correlations between multi-source heterogeneous data; third, fixed alarm thresholds are generally used, lacking dynamic adaptation to individual factors such as driver age and driving time, leading to false alarms or missed alarms; finally, intervention measures are mostly simple alarms, lacking a progressive and humanized proactive intervention system from early prevention to emergency response. Summary of the Invention

[0006] To address the shortcomings of existing technologies in comprehensively, accurately, and individually assessing and intervening in driver safety risks, this invention aims to provide a method and system for proactive intervention in driver safety risks based on multimodal perception, thereby overcoming the deficiencies of existing technologies and achieving accurate assessment and proactive, tiered, and effective intervention in driving risks.

[0007] To achieve the above objectives, the present invention provides a method for proactive intervention in driver safety risks based on multimodal perception, comprising: Acquire first modal data reflecting the driver's physiological state, second modal data reflecting vehicle dynamics, and third modal data reflecting the risks of the external driving environment; Based on the first, second, and third modal data, feature-level fusion is performed through a multimodal deep learning fusion model to generate a risk index characterizing the current comprehensive driving risk level; wherein, the multimodal deep learning fusion model is configured to capture the inherent correlation between different modal data; The threshold of the risk indicator used to trigger the intervention strategy is dynamically adjusted based on at least one static physiological characteristic and at least one dynamic physiological load characteristic related to the driver's physiological state. Based on the comparison between the risk indicator and the dynamically adjusted threshold, a progressive intervention strategy is executed. The progressive intervention strategy includes at least the following: when the risk indicator reaches a first preset level, a sensory reminder is triggered for the driver; when the risk indicator reaches a second preset level higher than the first preset level, a mandatory rest intervention aimed at guiding the driver to suspend driving is triggered; and when the risk indicator reaches a third preset level higher than the second preset level, a multi-party collaborative emergency response involving vehicle control or external communication is triggered.

[0008] In one implementation, the first modal data is driver state monitoring visual data, the second modal data is vehicle controller area network bus behavior data, and the third modal data is high-precision map road environment data. These three modalities cover data from three dimensions: driver internal state, vehicle handling behavior, and external environmental risks, providing a foundation for constructing a comprehensive risk profile.

[0009] Furthermore, at least one of the following is extracted from the driver state monitoring visual data: eye closure duration, gaze direction, or yawning frequency; at least one of the following is extracted from the vehicle controller area network bus behavior data: steering wheel angle standard deviation, frequency of sudden braking / acceleration, or lane departure correction frequency; and at least one of the following is extracted from the high-precision map road environment data: road curvature, historical accident density, current weather level, or nighttime lighting level. By extracting these specific key features, risk factors for each modality can be quantified more accurately.

[0010] Optionally, the multimodal deep learning fusion model is a multimodal Transformer model, which achieves feature-level fusion between different modalities through a cross-attention mechanism; the risk indicator is the risk entropy value. Using a multimodal Transformer model can effectively capture the deep interaction relationships and time-series dependencies between different data modalities, and can more accurately assess comprehensive risk compared to traditional models.

[0011] Furthermore, before performing feature-level fusion, the multimodal Transformer model further includes: encoding the first modality data using a convolutional neural network; encoding the second modality data using a multilayer perceptron; and encoding the third modality data using a graph neural network. Employing the optimal encoder for different types of data maximizes the preservation of the original information of each modality, providing high-quality feature representations for subsequent fusion.

[0012] In one implementation, the static physiological characteristic is the driver's age, and the dynamic physiological load characteristic is the cumulative driving time within a day. These two characteristics are key individualized factors affecting driver fatigue and reaction ability, and incorporating them into the assessment can make risk assessment more targeted.

[0013] Furthermore, the dynamic adjustment is achieved through an attenuation compensation factor α, which is calculated using the formula α = 1 + β_age × f(age) + β_drive × g(drive_hours), where f(age) is a monotonically non-decreasing function related to age, g(drive_hours) is a monotonically non-decreasing function related to the cumulative driving time of the day, and β_age and β_drive are preset coefficients. This dynamic adjustment mechanism allows the risk threshold to adapt to individual differences in drivers and their current physiological load, thereby achieving earlier warnings.

[0014] Optionally, the sensory reminder is a voice reminder; the mandatory rest intervention is a route intervention, which plans and sends a navigation route to a nearby safe rest stop; the multi-party collaborative emergency response is an emergency linkage, which includes at least reducing the vehicle's maximum speed limit or sending alarm information to a pre-set third party. This progressive intervention strategy, from gentle reminders to mandatory rest, and then to emergency linkage, achieves closed-loop risk management.

[0015] Furthermore, the mandatory rest intervention specifically includes: calling the map service application interface to search for service areas or safe rest stops within a predetermined distance, planning the optimal rest route, and disseminating the optimal rest route through the in-vehicle navigation system. This proactive route planning method provides drivers with a concrete and actionable solution.

[0016] Furthermore, the multi-party collaborative emergency response specifically includes at least one of the following: reducing the vehicle's maximum permissible speed via the vehicle control unit; automatically dialing the driver's preset emergency contact number; and sending an alarm to the ride-hailing platform's safety center to notify a safety officer to intervene. In extremely high-risk situations, combining vehicle control with external assistance can minimize the occurrence of accidents or mitigate their consequences.

[0017] The present invention also includes a driver safety risk proactive intervention system based on multimodal perception, the system comprising: The data acquisition unit is used to acquire first modal data reflecting the driver's physiological state, second modal data reflecting vehicle dynamics, and third modal data reflecting the risks of the external driving environment. The central processing unit is configured to perform feature-level fusion based on the first, second, and third modal data using a multimodal deep learning fusion model to generate a risk indicator characterizing the current comprehensive driving risk level; wherein the multimodal deep learning fusion model is configured to capture the intrinsic correlation between different modal data; and to dynamically adjust the threshold of the risk indicator for triggering intervention strategies based on at least one static physiological feature and at least one dynamic physiological load feature related to the driver's physiological state. An active intervention unit is used to execute a progressive intervention strategy based on the comparison result between the risk indicator and the dynamically adjusted threshold. The progressive intervention strategy includes at least: triggering a sensory reminder to the driver when the risk indicator reaches a first preset level; triggering a mandatory rest intervention aimed at guiding the driver to suspend driving tasks when the risk indicator reaches a second preset level higher than the first preset level; and triggering a multi-party collaborative emergency response involving vehicle control or external communication when the risk indicator reaches a third preset level higher than the second preset level.

[0018] Compared with existing technologies, this invention has significant advantages. First, by fusing data from three key modalities and employing a multimodal deep learning fusion model, this invention can more comprehensively and deeply understand complex driving scenarios, capturing potential risks that cannot be detected by a single modality or simple model, thereby significantly improving the accuracy of risk assessment and effectively reducing false alarm and false negative rates. Second, this invention uniquely introduces a dynamic threshold adjustment mechanism based on static and dynamic physiological characteristics, enabling the risk threshold to be adjusted in real time according to the driver's individual physiological characteristics and dynamic load status, achieving more sensitive early warnings for high-risk groups and making intervention more timely, accurate, and humane. Finally, the progressive intervention strategy designed in this invention achieves closed-loop management of the entire process, from gentle guidance in the risk nascent stage to mandatory rest in the risk accumulation stage, and then to emergency handling in extreme dangerous situations. This effectively mitigates risks without affecting the normal driving experience and takes decisive measures at critical moments to maximize driving safety. Attached Figure Description

[0019] Figure 1 This is a functional block diagram of the driver safety risk proactive intervention system involved in this invention.

[0020] Figure 2 This is a flowchart of the driver safety risk proactive intervention method involved in the present invention.

[0021] Figure 3 This is a schematic diagram of the structure of one embodiment of the multimodal deep learning fusion model described in this invention.

[0022] Figure 4 This is a schematic diagram illustrating the effect of the attenuation compensation factor α described in this invention on the risk threshold. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] While progress has been made in driver status monitoring and risk warning technologies in the current fields of intelligent driving and traffic safety, several significant shortcomings remain. For example, existing solutions often rely on single or limited perception modalities, such as using only vision or analyzing only vehicle data, leading to a one-sided understanding of complex driving scenarios and insufficient accuracy in risk assessment. Furthermore, risk models often employ simple rule engines or linear weighting, failing to capture deep correlations between multi-source heterogeneous data, resulting in low model accuracy. More critically, existing solutions generally use fixed alarm thresholds, ignoring individual driver physiological differences (such as age) and dynamic changes (such as fatigue accumulation), leading to frequent false alarms and missed alarms. Simultaneously, intervention measures are mostly passive reminders, lacking the ability to proactively and progressively intervene during the risk accumulation phase, making it difficult to effectively prevent accidents.

[0025] To overcome the shortcomings of existing technologies, this invention proposes an innovative method and system for proactive intervention in driver safety risks. The core idea of ​​this approach is to construct a closed-loop system integrating "three-modal fusion perception + multi-modal deep learning fusion model risk modeling + individualized dynamic threshold adjustment + progressive intervention." By comprehensively integrating data reflecting the driver's physiological state, vehicle dynamics, and external driving environment risks, advanced deep learning models are used for precise risk quantification. Furthermore, the timing of intervention is dynamically adjusted based on the driver's individual characteristics. Ultimately, through a tiered and progressive intervention strategy, proactive management of the entire process from risk emergence to high-risk states is achieved, thereby significantly improving driving safety levels.

[0026] like Figure 1 The diagram shown illustrates the functional modules of the driver safety risk proactive intervention system of this invention. This system can be integrated into an in-vehicle computing platform or implemented through a separate device connected to the vehicle system. The system mainly includes a data acquisition unit 100, a central processing unit 200, and a proactive intervention unit 300. These units work together to form the technical basis for the method of this invention.

[0027] Specifically, the data acquisition unit 100 is responsible for acquiring multi-source heterogeneous information related to driving risks in real time. This unit includes a DMS vision acquisition module 110, a CAN bus acquisition module 120, and a high-precision map data module 130. The DMS vision acquisition module 110 is typically a camera (e.g., an infrared and visible light dual-mode camera) installed inside the vehicle and facing the driver, used to capture the driver's facial and head features. The CAN bus acquisition module 120 reads the vehicle's dynamic operating data at high frequency by connecting to the vehicle's diagnostic interface (such as an OBD interface) or directly accessing the CAN bus. The high-precision map data module 130 obtains high-precision environmental information about the vehicle's current location and planned path from ADAS map services or cloud map platforms via network connection. Through the collaborative work of these three modules, the system can construct a complete and three-dimensional situational awareness picture of the driver, vehicle, and environment.

[0028] The central processing unit 200 is the "brain" of the entire system, responsible for data processing, analysis, and decision-making. This unit integrates a risk modeling module 210, a threshold adjustment module 220, and an intervention decision-making module 230. The risk modeling module 210 receives trimodal data from the data acquisition unit 100 and runs a multimodal deep learning fusion model to generate a quantified risk indicator. The threshold adjustment module 220 dynamically adjusts the risk threshold used to trigger intervention based on pre-stored or real-time acquired individual driver characteristics. The intervention decision-making module 230 compares the real-time risk indicator output by the risk modeling module 210 and the dynamic threshold output by the threshold adjustment module 220, and determines whether to trigger intervention and what level of intervention to take based on preset rules.

[0029] The active intervention unit 300 is the system's "actuator," responsible for translating the decisions of the central processing unit 200 into concrete intervention actions. This unit includes a human-machine interaction module 310, a vehicle control module 320, and a remote communication module 330. The human-machine interaction module 310 primarily uses in-vehicle speakers, displays, and other devices to issue voice or visual reminders to the driver. The vehicle control module 320, when authorized or in emergency situations, can interact with the vehicle's electronic control unit (ECU) to perform operations such as limiting vehicle speed. The remote communication module 330 is responsible for establishing communication with the outside world, such as automatically dialing preset emergency contact numbers in emergencies, or sending alarm information and vehicle status data to a back-end monitoring center (such as the ride-hailing platform's security center).

[0030] The following is combined Figure 2 The specific process of the driver safety risk proactive intervention method involved in this invention is described in detail. For example... Figure 2As shown, the method mainly includes the following steps: First, in step S201, the system collects trimodal data in real time. Next, in step S202, the risk entropy value is calculated using a multimodal Transformer model. Subsequently, in step S203, the system obtains physiological characteristic information such as the driver's age and cumulative driving time. In step S204, the system calculates the attenuation compensation factor based on these physiological characteristics and dynamically adjusts the risk threshold used for alarms. Afterward, the system enters the decision-making stage. In step S205, it determines whether the risk entropy value has reached the highest level three threshold. If yes, then in step S210, L3 emergency linkage intervention is executed. If no, then in step S206, it determines whether the risk entropy value has reached a relatively high level two threshold. If yes, then in step S209, L2 route intervention is executed. If still no, then in step S207, it determines whether the risk entropy value has reached the level one threshold. If yes, then in step S208, L1 voice reminder intervention is executed. If none of the thresholds are reached, the process returns to step S201 and continues monitoring.

[0031] In one embodiment of the present invention, a method for proactive intervention in driver safety risks based on multimodal perception is provided. The method first acquires first-modal data reflecting the driver's physiological state, second-modal data reflecting vehicle dynamics, and third-modal data reflecting the risks of the external driving environment. Then, based on the data from these three modalities, feature-level fusion is performed through a multimodal deep learning fusion model to generate a risk index characterizing the current overall driving risk level. Feature-level fusion here means that the model does not simply piece together data, but rather deeply understands and correlates information from different data sources. This multimodal deep learning fusion model is specifically configured to capture the inherent correlations between different modalities of data; for example, it identifies the higher risk implied by the simultaneous occurrence of driver line-of-sight deviation (first modality) and reduced minor steering wheel corrections (second modality) on road curves (third modality).

[0032] Next, the method dynamically adjusts the threshold of the risk indicator used to trigger the intervention strategy based on at least one static physiological characteristic and at least one dynamic physiological load characteristic related to the driver's physiological state. This step embodies the idea of ​​personalized risk management, acknowledging that different drivers have different risk tolerance under different conditions. Finally, based on the comparison between the risk indicator and the dynamically adjusted threshold, a progressive intervention strategy is executed. This progressive intervention strategy is ingeniously designed, comprising at least three levels: when the risk indicator reaches a first preset level, a relatively mild sensory alert is triggered to the driver; when the risk indicator climbs to a second preset level higher than the first preset level, the system takes further measures, triggering a mandatory rest intervention aimed at guiding the driver to suspend driving tasks; and when the risk reaches the highest level, i.e., a third preset level higher than the second preset level, the system initiates a multi-party collaborative emergency response involving vehicle control or communication with the outside world to minimize or mitigate potential hazards.

[0033] In a preferred implementation, the first modal data specifically comprises Driver Status Monitoring (DMS) visual data, the second modal data comprises Vehicle Controller Area Network (CAN) bus behavioral data, and the third modal data comprises high-precision map road environment data. This specific modal combination offers significant advantages: DMS visual data can intuitively reflect the driver's physiological and psychological states, such as fatigue and distraction; CAN bus behavioral data objectively records the driver's driving style and vehicle control details, such as the smoothness or abruptness of operation; and high-precision map road environment data incorporates risk variables from the external environment, such as road geometry and historical accident data. By fusing these three highly complementary data modalities, the system can construct a comprehensive and in-depth understanding of driving risks, far exceeding traditional methods that rely on only one or two modalities, thus laying a solid data foundation for achieving high-precision risk assessment.

[0034] Furthermore, to improve the accuracy and robustness of risk assessment, this invention extracts a series of key quantitative features from the aforementioned three modalities of data. Specifically, from the Driver Status Monitoring (DMS) visual data, at least one of eye closure duration (PERCLOS), gaze direction, or yawning frequency can be extracted through image processing and computer vision algorithms. For example, the DMS visual acquisition module 110 can be set to acquire images at a frequency of 10fps, and the PERCLOS value can be calculated by analyzing consecutive frame images, or the driver's gaze can be detected as continuously deviating from the road ahead, or the number of yawns per unit time can be counted. From the Vehicle Controller Area Network (CAN) bus behavior data, acquired at a high frequency of, for example, 50Hz, at least one of steering wheel angle standard deviation, frequency of emergency braking / acceleration, or lane departure correction frequency can be extracted. An increase in the steering wheel angle standard deviation is generally associated with overcorrection behavior under fatigue or distraction. From the high-precision map road environment data, at least one of road curvature, historical accident density, current weather level, or nighttime lighting level can be extracted as needed. The introduction of these features enables risk models to make judgments based on specific, objective quantitative indicators, rather than vague qualitative descriptions, which greatly improves the accuracy and reliability of risk identification.

[0035] Based on the above scheme, the multimodal deep learning fusion model is specifically a multimodal Transformer model, which achieves feature-level fusion between different modalities of data through a cross-attention mechanism. Simultaneously, the risk indicator is defined as a specific quantitative value, called the risk entropy value. The Transformer model was chosen because of its powerful sequential data processing capabilities and ability to capture long-distance dependencies, especially its core self-attention and cross-attention mechanisms. The cross-attention mechanism allows the model to "pay attention" to relevant information in other modalities of data while processing data from one modality, thereby achieving true deep fusion. For example, when analyzing a fatigue feature in DMS data, the model can simultaneously pay attention to changes in steering wheel operation in CAN data and the complexity of the current road segment in high-precision map data, comprehensively judging the risk. The final generated risk entropy value, such as a comprehensive score from 0 to 100, provides a clear and unified metric for subsequent intervention decisions.

[0036] like Figure 3The diagram shows the structure of the multimodal Transformer risk entropy model. This model demonstrates how to process and fuse data from different sources. The model's inputs are DMS visual data input 301, CAN bus data input 302, and high-precision map data input 303. To effectively represent the initial features of these heterogeneous data from different sources and with varying structures, the model includes a dedicated encoder for each modality before feature-level fusion. As a preferred implementation, the model uses a Convolutional Neural Network (CNN) as the CNN encoder 311 to encode the first modality data (DMS visual data), leveraging the powerful image spatial feature extraction capabilities of CNNs. Simultaneously, a Multilayer Perceptron (MLP) is used as the MLP encoder 312 to encode the second modality data (CAN bus data), as MLPs are well-suited for processing tabular vector data such as CAN bus data. Furthermore, a graph neural network (GNN) is used as the GNN encoder 313 to encode the third modality data (high-precision map data) because high-precision map data naturally has a graph structure (e.g., intersections are nodes and roads are edges), and GNN can effectively learn the risk information contained in its topology and node attributes.

[0037] After processing by their respective encoders, the data from the three modalities are transformed into high-dimensional feature vectors of a unified dimension. These feature vectors are then fed into the core cross-attention fusion module 320. In this module, the feature vectors of one modality serve as queries, while the feature vectors of the other modalities serve as keys and values. The weight for extracting information from the values ​​is determined by calculating the similarity between the query and the key. This process is performed multiple times and in multiple layers across different modalities, enabling the model to learn complex cross-modal association patterns, such as "on a curve with a curvature of 0.012, when the PERCLOS value reaches 0.42, the standard deviation of the steering wheel angle increases by 35%". The final output of the fusion process is a three-dimensional risk vector output 330, where the three dimensions correspond to specific risk types such as fatigue, distraction, and aggression. Finally, by taking the maximum value of each dimension in this three-dimensional risk vector and normalizing it, the final risk entropy value 340 is obtained. This transformation from multi-dimensional risk assessment to a single risk indicator preserves the interpretability of risk types while providing the simplicity required for decision-making.

[0038] To address the problem of excessive or ineffective alarms caused by fixed thresholds in existing technologies, which fail to adapt to individual differences, this invention introduces an individualized dynamic threshold adjustment mechanism. Specifically, the static physiological characteristic is preferably the driver's age, while the dynamic physiological load characteristic is preferably the cumulative driving time within the current day. Age is chosen as the static characteristic because numerous traffic safety studies have shown that a driver's reaction time and risk perception ability change with age. The cumulative driving time within the current day is chosen as the dynamic characteristic because it directly reflects the driver's accumulated fatigue level within the current driving task cycle. By considering both dimensions simultaneously, the system can more accurately assess the driver's current actual risk tolerance.

[0039] Specifically, the dynamic adjustment is achieved through an attenuation compensation factor α. This design is a core feature that distinguishes this invention from existing technologies. The attenuation compensation factor α is calculated using a predefined formula. In a preferred embodiment, this formula is: α = 1 + β_age × f(age) + β_drive × g(drive_hours). Here, f(age) is a monotonically non-decreasing function related to age, g(drive_hours) is a monotonically non-decreasing function related to the cumulative driving time of the day, and β_age and β_drive are preset weighting coefficients. This functional design provides great flexibility, allowing for fitting and adjustment based on actual statistical data.

[0040] In a more specific implementation, the above general formula is concretized as: α = 1 + β_age × max(0, age-35) / 30 + β_drive × drive_hours / 12. In this formula, α is the final calculated attenuation compensation factor α; β_age is the age attenuation coefficient, which can be preset to 0.15; β_drive is the driving time attenuation coefficient, which can be preset to 0.20; age represents the driver's actual age (in years); and drive_hours represents the driver's cumulative driving time for the day (in hours). The function f(age) is specifically implemented as max(0, age-35) / 30, meaning that for drivers aged 35 and below, the age-related attenuation effect is 0, while after age 35, the attenuation effect increases linearly with age. The function g(drive_hours) is specifically implemented as drive_hours / 12, indicating that the attenuation effect is proportional to the driving time. After calculating α, the final intervention threshold is obtained through the formula "adjusted threshold = base threshold / α". Since α is greater than or equal to 1, this mechanism lowers the intervention threshold for older drivers or those who have been driving for a long time, making the system more sensitive and enabling focused attention and early intervention for high-risk groups.

[0041] like Figure 4 The diagram illustrates the impact of the physiological characteristic attenuation compensation factor on the risk threshold. The horizontal axis represents the cumulative driving time for the day, and the vertical axis represents the risk threshold. Curve 401 represents the unadjusted base risk threshold, a horizontal straight line that does not change with the driver's condition. Curve 402 represents the dynamic threshold curve for a young driver, showing that the threshold decreases slowly with increasing driving time. Curve 403 represents the dynamic threshold curve for an older driver, whose starting threshold is lower than that of a younger driver, and whose threshold decreases at a steeper slope with increasing driving time. This visually demonstrates how the attenuation compensation factor α "tailors" the risk defense line for different drivers at different times based on both age and driving time. Through this dynamic adjustment, the system can automatically increase alertness when the driver's physiological function declines, thereby minimizing interference with normal driving while ensuring safety. This significantly reduces the system's false alarm rate (e.g., from 42% to 9%) and improves driver acceptance and trust.

[0042] After comparing risk indicators with dynamically adjusted thresholds, the system will implement progressive intervention strategies in stages. As a preferred implementation, the sensory-level reminders specifically refer to voice reminders; the forced rest intervention specifically refers to route intervention, the core of which is planning and issuing navigation routes to nearby safe rest stops; and the multi-party collaborative emergency response specifically refers to emergency linkage, which includes at least forceful measures such as reducing the vehicle's maximum speed limit or sending alarm information to a pre-set third party. This three-tiered intervention strategy (L1-L3) is designed to progressively increase in intensity and depth. L1 voice reminders aim to gently guide the driver to correct their behavior independently; L2 route intervention provides clear action guidance when the driver may have lost some self-regulation ability; and L3 emergency linkage is the last line of defense when the system determines there is extreme risk and the driver may be completely incapacitated.

[0043] In Level 1 intervention (L1), when the risk entropy value exceeds the adjusted Level 1 threshold (e.g., the base threshold of 50 divided by α), the intervention decision module 230 instructs the active intervention unit 300 to execute a voice prompt. The human-machine interaction module 310 then broadcasts targeted prompts through the vehicle's audio system. For example, if the main contributor to the risk entropy value is fatigue, the system will broadcast "We have detected that you may be somewhat fatigued; please take a rest"; if the main contributor is distraction, the system will broadcast "Please concentrate and drive safely." This intervention method minimizes interference with the driver and aims to serve as a reminder and warning, prompting the driver to proactively adjust their driving behavior or state.

[0044] In Level 2 intervention (L2), when the risk entropy value further increases and exceeds the adjusted Level 2 threshold (e.g., the base threshold 70 divided by α), the system determines that a simple reminder may not be sufficient to eliminate the risk, and thus initiates route intervention. Specifically, the route intervention includes a series of automated steps: First, the intervention decision module 230 calls the high-precision map data module 130 and the relevant map service application programming interface (API) to search for service areas, parking lots, or other marked safe stops within a predetermined distance (e.g., a radius of 30 kilometers) centered on the vehicle's current location. Next, the system evaluates the accessibility of each searched stop, comprehensively considering factors such as distance, estimated arrival time, current traffic conditions, and vehicle range, to plan an optimal stopping route. Finally, the human-machine interaction module 310 announces the intervention decision to the driver, such as "Your driving condition is not good. A route to service area A ahead has been planned for you. It is recommended to rest for 20 minutes," and sends the optimal stopping route through the in-vehicle navigation system. The navigation automatically switches after the driver confirms or the default timeout (e.g., 30 seconds). This proactive and guided intervention approach transforms "rest recommendations" into concrete and actionable plans, greatly improving the success rate of interventions.

[0045] In Level 3 intervention (L3), when the risk entropy value reaches the highest level Level 3 threshold (e.g., the base threshold 90 divided by α), the system determines that the situation is extremely critical, and the driver may have already lost or is about to lose control of the vehicle. At this time, the system will trigger an emergency linkage. Specifically, the emergency linkage may include at least one of the following operations: First, the intervention decision module 230 instructs the vehicle control module 320 to reduce the vehicle's maximum permissible speed to a safer level (e.g., 60% of the current speed or a fixed low speed value) by interacting with the vehicle's power or braking control system, to prevent the vehicle from speeding or losing control due to the driver's unconscious operation. Second, the remote communication module 330 instructs the remote communication module 330 to automatically dial the emergency contact number preset by the driver in the system and play pre-recorded alarm information. Third, for ride-hailing and other operational vehicles, the remote communication module 330 can also send an alarm containing the vehicle ID, location, real-time risk data, and event type to the ride-hailing platform's safety center to notify the back-end safety officer to intervene in a timely manner. These measures together form a multi-party collaborative emergency response network, aiming to control extreme risks and protect the lives of drivers and passengers through technological means and external rescue forces.

[0046] To enable those skilled in the art to more clearly understand the technical solution of the present invention, two specific embodiments will be described below. It should be noted that these embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0047] In a scenario of fatigued driving, suppose a 52-year-old ride-hailing driver, Mr. Wang, has been driving continuously for 7 hours that day and is currently driving on a highway at 11:00 PM. At this time, the system's data acquisition unit 100 is continuously operating. The DMS visual acquisition module 110 detects that his eye closure duration (PERCLOS) has reached 0.42, far exceeding the normal threshold of 0.15, and he has yawned three times within one minute, with his gaze frequently deviating from straight ahead. Simultaneously, data from the CAN bus acquisition module 120 shows that the standard deviation of his steering wheel angle has increased by 35% compared to normal, and the frequency of lane departure correction actions has increased to four times per minute. Information from the high-precision map data module 130 indicates that the current road segment is a highway curve with a curvature of 0.012, and the nighttime lighting level is rated as "low."

[0048] These multimodal data are input into the risk modeling module 210 of the central processing unit 200. After feature extraction and fusion calculation, the multimodal Transformer model outputs a three-dimensional risk vector of [fatigue = 72, distraction = 35, aggression = 18]. Therefore, the current comprehensive risk entropy value is determined to be the maximum value among these three, namely 72. At the same time, the threshold adjustment module 220 obtains the driver's age (52 years old) and the cumulative driving time on that day (7 hours), and calculates the attenuation compensation factor α according to a preset formula. The calculation process is: α = 1 + 0.15 × (52 - 35) / 30 + 0.20 × 7 / 12 = 1 + 0.15 × 17 / 30 + 0.20 × 0.583 = 1 + 0.085 + 0.117 = 1.202.

[0049] The intervention decision module 230 uses this α value to adjust the base threshold. The adjusted L1 threshold is 50 / 1.202 ≈ 41.6, and the adjusted L2 threshold is 70 / 1.202 ≈ 58.2. Since the current risk entropy value of 72 has exceeded the adjusted L2 threshold of 58.2, the system decides to trigger L2-level route intervention. The active intervention unit 300 then executes the operation, calling the map service to search for a service area A 18 kilometers away from the current location within 30 kilometers ahead, and assesses that the vehicle's range and road conditions are suitable for travel. The system announces to Mr. Wang via voice: "Your driving condition is not good. We have planned a route to the service area A ahead for you. We suggest you rest for 20 minutes." At the same time, the in-vehicle navigation automatically switches to the route to service area A. Ultimately, Mr. Wang follows the guidance and rests at the service area for 25 minutes. After the rest, the system reassesses, and his PERCLOS value drops to 0.05, and the risk entropy value also drops to 22 (within the safe range). The system confirms that the risk has been eliminated and allows him to continue accepting orders and operating. This example fully demonstrates the entire closed-loop process from multimodal perception, risk modeling, individualized threshold adjustment to proactive intervention.

[0050] In another scenario involving distracted driving, suppose a 35-year-old driver is looking down at their phone for an extended period while driving on a highway. The DMS vision acquisition module 110 captures that their gaze remains off the road for 12 seconds, and their hands move off the steering wheel. The CAN bus acquisition module 120 also detects that during this period, the vehicle speed begins to deviate unintentionally from the average speed of the current lane, and there are abnormal fluctuations in lateral acceleration. These characteristics are identified by the risk modeling module 210 as typical high-risk distracted driving behaviors.

[0051] The multimodal Transformer model quickly fuses and analyzes these input features and outputs an extremely high risk entropy value, such as 95. For this 35-year-old driver, since his age is at the baseline age and his cumulative driving time that day is not long, his attenuation compensation factor α will be very close to 1. Therefore, his L3 intervention trigger threshold is close to the base threshold of 90. The current risk entropy value of 95 is obviously far beyond this threshold. The intervention decision module 230 immediately judges the situation to be critical and triggers the L3 level emergency linkage. The active intervention unit 300 immediately executes a series of operations: the vehicle control module 320 first intervenes and limits the vehicle's maximum speed to 60% of the current speed to prevent the vehicle from continuing to accelerate or maintain high speed while the driver is out of control. At the same time, the remote communication module 330 automatically dials the driver's preset emergency contact number and sends an alarm message containing the vehicle's location, speed, risk type (high-risk distraction), and risk value (95) to the platform safety center. The in-vehicle human-machine interface module 310 continuously issues a voice warning at maximum volume: "Danger! Please take immediate control of the vehicle!" until the system detects that the driver's gaze has returned to the front and the risk entropy value has significantly decreased to a safe range. This embodiment demonstrates the invention's rapid, decisive, and multi-layered response capabilities in dealing with sudden and extreme risk events.

[0052] In summary, the present invention, through the organic combination of the above-mentioned technical solutions, can bring many significant benefits compared to existing technologies. These benefits are not only theoretically valid, but have also been verified by extensive simulation and real-vehicle testing data.

[0053] First, this invention achieves a significant improvement in the accuracy of risk detection. By integrating three highly complementary data modalities—DMS vision, CAN bus, and high-precision maps—and utilizing a multimodal Transformer model for deep feature fusion, the system's risk identification capability far surpasses that of single-modal or simple fusion solutions. Test data shows that compared to the approximately 18% false alarm rate of traditional solutions, the false alarm rate of this invention can be reduced to 3%. Simultaneously, by introducing an individualized dynamic threshold adjustment mechanism based on age and driving duration, the system can effectively distinguish the risk baselines of different drivers, thereby successfully reducing the false alarm rate from as high as 42% at a fixed threshold to 9%. This improvement and reduction greatly enhance the system's signal-to-noise ratio and practical value.

[0054] Secondly, this invention is remarkably effective in preventing accidents. Its core value lies in shifting the focus of safety management from "post-accident accountability" to "pre-accident intervention." By proactively and gently intervening in the early stages of risk accumulation (L1 voice reminders) and the development stages (L2 route intervention), it effectively prevents a large number of potential risks from evolving into actual accidents. Statistical data shows that vehicles using this invention have a 67% lower incidence of high-risk driver behaviors (such as severe fatigue and prolonged distraction), and the rate of related accidents directly or indirectly caused by these behaviors has also decreased by 73%.

[0055] Furthermore, this invention demonstrates highly efficient emergency response capabilities when dealing with extreme risks. In extreme situations where the driver may become completely incapacitated, the L3 emergency linkage mechanism ensures that the most effective measures are taken in the shortest possible time. From the moment the system detects that the risk entropy value exceeds the L3 threshold to completing a series of core actions such as speed reduction control, making emergency calls, and sending platform alarms, the average response time is less than 8 seconds. This rapid multi-party collaborative response capability buys precious time for saving lives.

[0056] Finally, the design of this invention fully considers user experience and the feasibility of practical promotion. The progressive and user-friendly intervention design from L1 to L2 avoids driver resentment and resistance caused by frequent and ineffective "boy who cried wolf" type alarms. In a survey on the acceptance of ride-hailing drivers, this solution received a high overall score of 4.2 out of 5.0. High user acceptance means that drivers are more willing to cooperate with the system's guidance, thus ensuring that the technical solution can play its due role in the real world, and has extremely high promotional value and application prospects.

Claims

1. A method for proactive intervention in driver safety risks based on multimodal perception, characterized in that, include: Acquire first modal data reflecting the driver's physiological state, second modal data reflecting vehicle dynamics, and third modal data reflecting the risks of the external driving environment; Based on the first, second, and third modal data, feature-level fusion is performed through a multimodal deep learning fusion model to generate a risk index characterizing the current comprehensive driving risk level; wherein, the multimodal deep learning fusion model is configured to capture the inherent correlation between different modal data; The threshold of the risk indicator used to trigger the intervention strategy is dynamically adjusted based on at least one static physiological characteristic and at least one dynamic physiological load characteristic related to the driver's physiological state. Based on the comparison between the risk indicator and the dynamically adjusted threshold, a progressive intervention strategy is executed. The progressive intervention strategy includes at least the following: when the risk indicator reaches a first preset level, a sensory reminder is triggered for the driver; when the risk indicator reaches a second preset level higher than the first preset level, a mandatory rest intervention aimed at guiding the driver to suspend driving is triggered; and when the risk indicator reaches a third preset level higher than the second preset level, a multi-party collaborative emergency response involving vehicle control or external communication is triggered.

2. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 1, characterized in that, The first modal data is driver status monitoring visual data, the second modal data is vehicle controller local area network bus behavior data, and the third modal data is high-precision map road environment data.

3. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 1, characterized in that, The multimodal deep learning fusion model is a multimodal Transformer model, which achieves feature-level fusion between different modal data through a cross-attention mechanism; the risk indicator is the risk entropy value.

4. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 1, characterized in that, The sensory-level reminder is a voice reminder; The mandatory rest intervention is a route intervention, which plans and issues navigation routes to nearby safe rest points; the multi-party collaborative emergency response is an emergency linkage, which includes at least reducing the maximum speed limit of vehicles or sending alarm information to a preset third party.

5. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 2, characterized in that, Extract at least one of the following from the driver state monitoring visual data: eye closure duration, gaze direction, or yawning frequency; extract at least one of the following from the vehicle controller local area network bus behavior data: steering wheel angle standard deviation, frequency of emergency braking / acceleration, or lane departure correction frequency; and extract at least one of the following from the high-precision map road environment data: road curvature, historical accident density, current weather level, or nighttime lighting level.

6. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 3, characterized in that, Before performing feature-level fusion, the multimodal Transformer model further includes: encoding the first modal data using a convolutional neural network; encoding the second modal data using a multilayer perceptron; and encoding the third modal data using a graph neural network.

7. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 1, characterized in that, The static physiological characteristic is the driver's age, and the dynamic physiological load characteristic is the cumulative driving time of the day; The dynamic adjustment is achieved through an attenuation compensation factor α, which is calculated using the formula α=1+β_age×f(age)+β_drive×g(drive_hours), where f(age) is a monotonically non-decreasing function related to age, g(drive_hours) is a monotonically non-decreasing function related to the cumulative driving time of the day, and β_age and β_drive are preset coefficients.

8. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 4, characterized in that, The route intervention specifically includes: calling the map service application interface to search for service areas or safe rest stops within a predetermined distance, planning the optimal rest route, and issuing the optimal rest route through the vehicle navigation system.

9. The method for proactive intervention in driver safety risks based on multimodal perception according to claim 4, characterized in that, The emergency linkage specifically includes at least one of the following: reducing the vehicle's maximum permissible speed through the vehicle control unit; automatically dialing the driver's preset emergency contact number; and sending an alarm to the ride-hailing platform's safety center to notify the safety officer to intervene.

10. A driver safety risk proactive intervention system based on multimodal perception, characterized in that, The system includes: The data acquisition unit is used to acquire first modal data reflecting the driver's physiological state, second modal data reflecting vehicle dynamics, and third modal data reflecting the risks of the external driving environment. The central processing unit is configured to perform feature-level fusion based on the first, second, and third modal data using a multimodal deep learning fusion model to generate a risk indicator characterizing the current comprehensive driving risk level; wherein the multimodal deep learning fusion model is configured to capture the intrinsic correlation between different modal data; and to dynamically adjust the threshold of the risk indicator for triggering intervention strategies based on at least one static physiological feature and at least one dynamic physiological load feature related to the driver's physiological state. An active intervention unit is used to execute a progressive intervention strategy based on the comparison result between the risk indicator and the dynamically adjusted threshold. The progressive intervention strategy includes at least: triggering a sensory reminder to the driver when the risk indicator reaches a first preset level; triggering a mandatory rest intervention aimed at guiding the driver to suspend driving tasks when the risk indicator reaches a second preset level higher than the first preset level; and triggering a multi-party collaborative emergency response involving vehicle control or external communication when the risk indicator reaches a third preset level higher than the second preset level.

Citation Information

Patent Citations

  • Intelligent intervention system and method based on multi-modal driving behavior analysis

    CN120766476A

  • Intelligent driving assistance system and method based on multi-modal emotion recognition

    CN121572988A