Driver takeover monitoring method, apparatus and vehicle

Abstract

The embodiment of the application belongs to the technical field of automatic driving, and relates to a driver takeover monitoring method and device and a vehicle, comprising: acquiring a plurality of environment data and a plurality of state data of a driver; determining a current scene type and a target collision time of the vehicle based on the plurality of environment data; determining an environment risk level based on the current scene type and the target collision time; determining a takeover prompting level based on the environment risk level; determining a takeover ability level of the driver based on the current scene type and the plurality of state data; and determining a risk execution strategy of the vehicle based on the takeover ability level of the driver. The application improves the perception efficiency and response speed of the driver to the prompt through multi-modal collaborative prompting. The dynamic weight scoring model accurately evaluates the takeover ability of the driver, and the flexibility and safety of small risk execution are realized in combination with the environment risk level strategy.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a driver takeover monitoring method, device, and vehicle. Background Technology

[0002] The driver takeover monitoring system is a core subsystem in advanced driver assistance systems (ADAS) or autonomous driving systems. It monitors in real time whether the driver has the ability to take over vehicle control and triggers alerts or safety policies when necessary. Its core objective is to ensure that the driver can take over vehicle control promptly and safely when the system cannot handle complex scenarios (such as sudden obstacles or sensor failures) or when driver intervention is required, thus preventing accidents.

[0003] Existing driver takeover monitoring methods largely rely on fixed collision time thresholds to assess environmental risks, employing single or ungraded alert methods. Furthermore, they often evaluate driver takeover capabilities based on single-state data, and the execution of risk strategies lacks linkage with the driver's real-time capabilities. These existing technologies suffer from problems such as environmental risk assessments not adapting to different scenarios and one-sided assessments of driver takeover capabilities, leading to unreasonable timing of takeover alerts and poor effectiveness of risk strategy execution. Summary of the Invention

[0004] The purpose of this application is to provide a driver takeover monitoring method, device, computer equipment, and storage medium to solve the problems of environmental risk assessment not being adapted to different scenarios and driver takeover capability assessment being one-sided.

[0005] To address the aforementioned technical problems, this application provides a driver takeover monitoring method, which employs the following technical solution:

[0006] A method for monitoring driver takeover includes the following steps:

[0007] Acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver;

[0008] Based on multiple environmental data, the current scene type and target collision time of the vehicle are determined, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment.

[0009] Based on the current scene type and the target collision time, determine the environmental risk level;

[0010] Based on the environmental risk level, a takeover alert level is determined, wherein the takeover alert includes tactile alerts and visual alerts;

[0011] Based on the current scenario type and multiple status data, the driver's takeover capability level is determined;

[0012] Based on the driver's takeover capability level, a risk execution strategy for the vehicle is determined.

[0013] Furthermore, the determination of the environmental risk level based on the current scene type and the target collision time includes:

[0014] A target threshold table is determined based on the current scenario type, wherein the target threshold table is a threshold range table corresponding to each preset environmental risk level of the current scenario type;

[0015] The environmental risk level is determined based on the target collision time and the target threshold table.

[0016] Furthermore, the determination of the environmental risk level based on the target collision time and the target threshold table includes:

[0017] The target threshold table is corrected based on the driver's historical data to obtain a corrected threshold table;

[0018] The environmental risk level is determined based on the collision time and the correction threshold table.

[0019] Furthermore, the determination of the driver's takeover capability level based on the current scenario type and multiple states includes:

[0020] Based on the current scenario type and multiple state data, a dynamic weight scoring model is used to determine the score and weight of each state data of the driver. The dynamic weight scoring model is used to score each state data of the driver and determine the weight of each state data according to the current scenario type.

[0021] Based on the scores and weights corresponding to each of the aforementioned state data, the driver's takeover capability score is obtained;

[0022] The driver's takeover ability level is determined based on the driver's takeover ability score.

[0023] Furthermore, the aforementioned current scenario types include highways, urban roads, and rural roads, and the driver's multiple state data include operational responses, cognitive states, and physiological signals;

[0024] Under the highway conditions, the weight of the operational response is higher than the weight of the cognitive state and the weight of the physiological signal;

[0025] Under the urban road conditions, the weight of the operational response is equal to the weight of the cognitive state, and the weight of the operational response is higher than the weight of the physiological signal.

[0026] Under the rural road conditions, the weight of the cognitive state is higher than the weight of the operational response and the weight of the physiological signal.

[0027] Furthermore, the aforementioned environmental risk levels include low risk, medium risk, and high risk, and determining the takeover alert level based on the environmental risk levels includes:

[0028] When the environmental risk level is low, the takeover alert level is no alert.

[0029] When the environmental risk level is medium risk, the takeover alert level is Level 1 alert, which includes mild tactile alert and mild visual alert;

[0030] When the environmental risk level is high, the takeover alert level is a level two alert, which includes both strong tactile and strong visual alerts.

[0031] Furthermore, the aforementioned driver takeover capability levels include safe takeover, partial takeover, and no takeover. The determination of the vehicle's risk execution strategy based on the driver's takeover capability level includes:

[0032] When the driver's takeover capability level is the safe takeover, the risk execution strategy is a low-risk strategy, wherein the low-risk strategy includes warning deceleration and auxiliary braking;

[0033] When the driver's takeover capability level is partial takeover, the risk execution strategy is a medium-risk strategy, wherein the medium-risk strategy includes decelerating to a safe speed, lane keeping, slowing down to the side of the road, braking, and automatic avoidance.

[0034] When the driver's takeover capability level is "unable to take over", the risk execution strategy is a high-risk strategy, which includes braking to a stop, calling for rescue, emergency deceleration, and turning on hazard lights.

[0035] Furthermore, the aforementioned tactile cues include steering wheel area vibration and seat vibration; the visual cues include AR projection; the operational responses include steering wheel torque, pedal opening, and operational delay; the cognitive states include eye tracking, pupil diameter changes, and facial features; and the physiological signals include heart rate and skin conductivity.

[0036] To address the aforementioned technical problems, this application also provides a driver takeover monitoring device, which employs the following technical solution:

[0037] A driver takeover monitoring device, comprising:

[0038] The data acquisition module is used to acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver;

[0039] The environmental risk determination module is used to determine the current scene type and target collision time of the vehicle based on multiple environmental data, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment, and is also used to determine the environmental risk level based on the current scene type and the target collision time;

[0040] The takeover alert module is used to determine the takeover alert level based on the environmental risk level, wherein the takeover alert includes tactile alerts and visual alerts;

[0041] The takeover capability determination module is used to determine the driver's takeover capability level based on the current scenario type and multiple state data.

[0042] The execution strategy module determines the risk execution strategy for the vehicle based on the driver's takeover capability level.

[0043] To address the aforementioned technical problems, this application also provides a driver takeover monitoring vehicle solution, employing the following technical solution:

[0044] The device includes a driver takeover monitoring device, which is used to perform the driver takeover monitoring method described in any of the above-mentioned methods.

[0045] Compared with the prior art, the embodiments of this application have the following main advantages:

[0046] The driver takeover monitoring method disclosed in this application acquires multiple environmental data and multiple driver state data; determines the current scenario type and collision time based on the multiple environmental data; determines the environmental risk level based on the current scenario type and the target collision time; determines the takeover alert level based on the environmental risk level; determines the driver's takeover capability level based on the current scenario type and the multiple state data; and determines the vehicle's risk execution strategy based on the driver's takeover capability level. This application enhances the driver's perception efficiency and response speed through multimodal collaborative prompts, and achieves flexibility and safety in low-risk execution by accurately assessing the driver's takeover capability and combining it with the environmental risk level. Attached Figure Description

[0047] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a system architecture diagram in which this application can be applied;

[0049] Figure 2 This is a flowchart illustrating the steps of the driver takeover monitoring method provided in this application;

[0050] Figure 3 This is a table of TTC threshold ranges for the driver takeover monitoring method provided in this application;

[0051] Figure 4 This is a table of minimum risk implementation strategies for the driver takeover monitoring method provided in this application;

[0052] Figure 5 This is a schematic diagram of the driver takeover monitoring device provided in this application. Detailed Implementation

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, system architecture 100 may include terminal devices (e.g., first terminal device 101, second terminal device 102, and third terminal device 103), network 104, and server 105. Network 104 serves as a medium for providing communication links between the first terminal device 101, second terminal device 102, third terminal device 103, and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0057] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptops, and desktop computers, etc.

[0059] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0060] It should be noted that the driver takeover monitoring method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the driver takeover monitoring device is generally installed in the terminal device.

[0061] The server also includes a processor and memory. In some embodiments, the processor may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor is typically used to control and execute the overall operation of the server. In this embodiment, the processor is used to run computer-readable instructions stored in the memory or to process data, such as running computer-readable instructions for the driver takeover monitoring method.

[0062] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0063] With the development of autonomous driving technology, driving systems need to request driver takeover in complex scenarios. Existing technologies have three major pain points: First, the effectiveness of prompts is insufficient. Single modalities (such as vision or touch) are easily ignored, and the prompt patterns are fixed (such as using the same vibration frequency in all scenarios), which cannot enhance the driver's perception efficiency through scenario-level coding, resulting in response delays or high rates of false ignoring. Second, the capability assessment is one-sided, relying on a single indicator (such as steering wheel contact or line of sight deviation) without integrating multi-dimensional data such as operation response, cognitive state, and physiological signals, which easily leads to misjudgments such as "hands on the steering wheel but lack of attention". Third, the strategy execution is rigid, using fixed thresholds to trigger minimum risk strategies (such as direct braking), which cannot dynamically adjust the intervention intensity according to the environmental risk level and the driver's ability, resulting in over-intervention (such as sudden braking in low-risk scenarios affecting comfort) or under-intervention (such as failure to avoid obstacles in high-risk scenarios).

[0064] refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of the driver disengagement monitoring method provided in this application. The driver disengagement monitoring method includes the following steps:

[0065] Step S01: Acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver.

[0066] In this embodiment, the driver takeover monitoring method operates on electronic devices (e.g., Figure 1The terminal device shown can send or receive data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, Wi-Fi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future wireless connection methods.

[0067] Traditional methods relying on a single data dimension (such as environmental data or driver operation data alone) are prone to biased assessments. Multi-source data fusion is needed to achieve collaborative perception between the environment and the driver, avoiding misjudgments due to missing information. Two core types of data are collected through vehicle sensors: environmental data (information about the vehicle's external driving environment) and driver status data (physiological, behavioral, and cognitive information of the driver), providing the input basis for subsequent risk assessment and strategy implementation.

[0068] In this embodiment, sensors such as millimeter-wave radar, cameras, lidar, and high-precision sensors are first used to acquire real-time data on obstacle distance, relative speed, road type (highway / urban / rural), traffic signs, and lane markings. Then, data on the driver's physical actions, attention distribution, and physiological stress responses are collected using sensors such as steering wheel torque sensors, pedal sensors, cameras (for eye movement and panel recognition), heart rate monitors, and electrodermal sensors.

[0069] The collected data are preprocessed as follows: noise reduction (e.g., filtering radar signals), time synchronization (ensuring that the time stamp deviation between the environment and driver data is ≤100ms), and redundancy verification (cross-validation of multi-sensor data, such as the distance deviation between the camera and radar obstacles is ≤5%) are performed on the raw data to form multiple environmental data and multiple driver status data.

[0070] Step S02: Based on multiple environmental data, determine the current scene type and target collision time of the vehicle, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment.

[0071] Traffic complexity and tolerance vary significantly across different scenarios (e.g., high speeds and low tolerance on highways, limited visibility on rural roads), necessitating dynamic adaptation of risk assessment through scenario classification. Time-to-Collision (TTC), the time required for two vehicles to maintain their current speeds until a collision occurs, is a direct indicator of collision risk and reflects the degree of urgency. Key features are extracted from environmental data to identify the current driving scenario type (e.g., highway, urban road), and the TTC between vehicles and obstacles is calculated as a core indicator for environmental risk assessment.

[0072] In this embodiment, based on data such as road features (number of lane lines, radius of curvature, speed limit information), traffic participants (pedestrian / non-motorized vehicle density), and environmental signs (highway entrance / exit, traffic lights), the road is classified into types such as highways, urban roads, and rural roads using deep learning models, such as CNN+LSTM (a hybrid deep learning model that combines convolutional neural networks CNN and long short-term memory networks LSTM).

[0073] Based on the relative velocity (v) and relative distance (d) of the obstacle, TTC is calculated using the formula TTC = d / v (when v ≠ 0). If v = 0 (stationary obstacle), a preset safe distance threshold is used (e.g., 50m corresponds to TTC = 5 seconds); for multi-obstacle scenarios, the minimum TTC value is used as the evaluation basis.

[0074] Step S03: Determine the environmental risk level based on the current scene type and the target collision time.

[0075] Fixed thresholds cannot adapt to different scenarios (e.g., a TTC of 4 seconds on a highway is considered high risk, while a TTC of 4 seconds on a rural road may be considered medium risk). Therefore, dynamic thresholds based on specific scenarios are needed to accurately classify risk levels. By combining scenario-specific thresholds with TTC values, environmental risks can be divided into multiple risk levels, quantifying the urgency of the external environment.

[0076] Based on the characteristics of the scenario type (such as fault tolerance and traffic complexity), set the corresponding rules for collision time and risk level in the scenario (not limited to a fixed threshold table).

[0077] The calculated collision time is compared with the rules of the scenario to determine the risk level of the collision (e.g., TTC ≤ 3 seconds is high risk in high-speed scenarios, and TTC ≤ 2 seconds is high risk in rural roads).

[0078] By combining historical accident data (such as the TTC distribution of 90% of accidents in this scenario) and real-time traffic flow (the threshold is reduced by 20% during congestion), the threshold table is dynamically adjusted. The calculated TTC is compared with the corrected threshold table to output the environmental risk level, such as low risk / medium risk / high risk.

[0079] Step S04: Based on the environmental risk level, determine the takeover alert level, wherein the takeover alert includes tactile alerts and visual alerts.

[0080] Traditional single-modal cues (such as sound-only alerts) are easily ignored by drivers (especially in noisy or distracted environments). Therefore, it is necessary to enhance the efficiency of cues perception through spatiotemporal binding of tactile and visual cues, while avoiding excessive alerts that could interfere with driving. Based on the environmental risk level (low / medium / high risk), differentiated multimodal takeover alerts (tactile + visual coordinated cues) should be triggered to ensure that drivers can quickly perceive and respond to risks.

[0081] Establish a logical correspondence between risk levels and alert levels (not limited to Level 1 and Level 2 alerts), and clearly define the alert combinations for different levels. Tactile alerts can be selected to vibrate the steering wheel in the direction of the risk (e.g., if there is an obstacle in the left front, the left side of the steering wheel will vibrate), or the seat and pedals can be selected as the vibration points. The vibration frequency and intensity can be selected according to different risk levels (e.g., slight vibration for low risk, continuous strong vibration for high risk).

[0082] Visual alerts can be selected using an AR windshield projection with a yellow border (indicating the direction and distance of the risk, such as "obstacle 30m to the left"), with the border flashing frequency synchronized with tactile vibration. Alternatively, the display can be set to a dashboard, with direct display methods (such as icons, text, and colors) and dynamic effects (such as flashing frequency).

[0083] Step S05: Based on the current scenario type and multiple state data, determine the driver's takeover capability level.

[0084] A single indicator (such as only detecting whether the steering wheel is in contact) cannot comprehensively assess a driver's takeover ability (e.g., when "hands are on the steering wheel but attention is distracted"). Accurate assessment requires multi-dimensional data and scenario-appropriate weighting to avoid over- or under-judgment of strategies due to misjudgment. By integrating multiple driver state data, such as operational response, cognitive state, and physiological signals, and combining them with scenario-type differentiated weights, a takeover ability score is calculated and a takeover ability level is assigned, for example, three levels: safe takeover, partial takeover, and no takeover.

[0085] Analyze multi-dimensional driver status data, such as operational response speed, eye focus, and heart rate changes, to assess performance in each dimension, such as the timeliness of operations and the level of concentration. Combine the "weighting requirements" of each status dimension according to the scenario type (e.g., operational response is more important in highway scenarios, while cognitive state is more important in rural roads) to comprehensively judge the driver's overall takeover ability and classify it into levels.

[0086] Step S06: Determine the risk execution strategy for the vehicle based on the driver's takeover capability level.

[0087] Traditional fixed strategies (such as triggering emergency braking regardless of risk level) can easily lead to poor passenger comfort or insufficient risk avoidance. If the driver can safely take over, the system does not need to intervene excessively, only providing assistance to avoid affecting the driver's operation. If the driver can only partially take over, the system needs to provide appropriate assistance (such as lane keeping and supplemental braking) to compensate for the driver's insufficient ability. Therefore, it is necessary to combine the driver's overall takeover ability and the level of environmental risk to implement a minimum-risk strategy.

[0088] After determining the driver's takeover capability level and the environmental risk level, corresponding rules for the execution strategy are set (which can be low, medium, or high risk strategies), clarifying the actions the system should take under different levels. The corresponding actions may include selecting warning deceleration, assisted braking, lane keeping, slow stop at the side of the road, automatic avoidance, emergency braking, and calling for help. The specific actions need to be combined with the current scenario (e.g., prioritizing the use of the emergency lane in highway scenarios and avoiding braking at intersections in urban roads).

[0089] Based on the matched risk execution strategy, the vehicle's execution systems (such as braking system, steering system, and lighting system) are controlled to perform operations, and environmental risks and driver status changes are monitored in real time. If the risk increases or the driver's ability decreases, the strategy can be upgraded; if the driver takes over effectively, the strategy can be downgraded or terminated.

[0090] This application enhances driver perception efficiency and response speed through multimodal collaborative prompts. A dynamic weighted scoring model accurately assesses driver takeover capability, and combined with a strategy based on environmental risk levels, it achieves flexibility and safety in low-risk execution. It can also continuously optimize individual adaptation and scenario response, enhancing the system's reliability and intelligence in complex traffic environments.

[0091] In some optional implementations of this embodiment, the step of determining the environmental risk level based on the current scene type and the target collision time specifically includes:

[0092] A target threshold table is determined based on the current scene type, wherein the target threshold table is a threshold range table corresponding to each preset environmental risk level of the current scene type; the environmental risk level is determined based on the target collision time and the target threshold table.

[0093] Risk perception thresholds vary significantly across different scenarios (e.g., low tolerance on highways, limited visibility on rural roads), necessitating dynamic adjustments to risk assessment criteria based on scenario characteristics to avoid misjudgments. Environmental risk levels are determined based on the current scenario type and time-to-collision (TTC), achieving scenario adaptation through a pre-set target threshold table and dynamic correction mechanism. TTC thresholds are categorized by scenario type, such as… Figure 3 As shown, Figure 3It is the TTC threshold range table of the driver takeover monitoring method provided by this application.

[0094] For highways, it can be: TTC > 5 seconds (low risk), 3 seconds < TTC ≤ 5 seconds (medium risk), TTC < 3 seconds (high risk); for urban roads, it can be: TTC > 4 seconds (low risk), 2.5 seconds < TTC ≤ 4 seconds (medium risk), TTC < 2.5 seconds (high risk). For rural roads, it can be: TTC > 4 seconds (low risk), 2.5 seconds < TTC ≤ 4 seconds (medium risk), TTC < 2 seconds (high risk).

[0095] Combined with real-time traffic flow (the threshold increases by 20% during congestion) and weather conditions (the TTC threshold increases by 1 second on rainy days), the TTC threshold is dynamically adjusted. By comparing TTC with the corrected threshold, the low / medium / high risk levels are output. Example, on a rainy day (visibility 50 meters) on an urban road, the sudden braking of the vehicle in front causes TTC = 2.6 seconds, triggering the rainy day correction (the threshold increases by 10%). The original medium risk threshold from 2.5 seconds to 4 seconds is corrected to 2.75 seconds to 4.4 seconds. Then, if TTC is less than 2.75 seconds, it is a high risk. Since TTC = 2.6 seconds is less than 2.75 seconds, it is determined to be a high risk.

[0096] This application dynamically adjusts the TTC threshold by combining different scenarios with real-time traffic flow and weather conditions. Compared with the strategy of fixed thresholds, it reduces the misjudgment rate and improves the accuracy of risk level determination.

[0097] In some optional implementation manners of this embodiment, the step of determining the environmental risk level based on the target collision time and the target threshold table specifically includes:

[0098] Correct the target threshold table according to the historical data of the driver to obtain a corrected threshold table; based on the collision time and the corrected threshold table, determine the environmental risk level.

[0099] Static thresholds cannot adapt to individual differences of drivers and the dynamic state of the vehicle (an increase in load may lead to an extension of the braking distance). Dynamically correct the target threshold table based on the driver's historical data (reaction time, operation habits) and the vehicle state (vehicle speed, load) to achieve dual adaptation of individuals and scenarios.

[0100] Specifically, record the average reaction time and operation delay of the driver in different scenarios, and combine parameters such as vehicle speed and road curvature radius. Update the correction coefficient every 100 kilometers of driving data. The correction rule can be:

[0101] If the reaction time is greater than 20% of the historical average, the threshold is increased by 10% (e.g., the high-risk TTC on highways is revised from 4 seconds to 4.4 seconds). For example, if the driver's historical reaction time is 1.2 seconds (historical average is 0.8 seconds), the system will revise his rural road TTC threshold from 3.5 seconds to 3.85 seconds to ensure sufficient takeover preparation time.

[0102] If the vehicle speed exceeds 100 km / h, the threshold increases by 10%. For example, in urban roads, if a driver increases the speed from 60 km / h to over 100 km / h, the system adjusts the risk TTC in urban roads from 3 seconds to 3.3 seconds to ensure sufficient takeover preparation time.

[0103] This application dynamically adjusts the target threshold table based on driver history data and vehicle status, thereby improving the accuracy of individual adaptation and reducing the false judgment rate of takeover detection.

[0104] In some optional implementations of this embodiment, determining the driver's takeover capability level based on the current scenario type and multiple state data specifically includes:

[0105] Based on the current scenario type and multiple state data, a dynamic weighted scoring model is used to determine the score and weight corresponding to each state data of the driver. The dynamic weighted scoring model is used to score each state data of the driver and determine the weight corresponding to each state data according to the current scenario type. Based on the score and weight corresponding to each state data, the driver's takeover ability score is obtained. Based on the driver's takeover ability score, the driver's takeover ability level is determined.

[0106] Single indicators (such as steering wheel contact) are prone to misjudgment (hands on the steering wheel but eyes elsewhere), requiring comprehensive multi-source data to improve assessment reliability. By fusing multi-dimensional indicators (operational response, cognitive state, physiological signals) and using a dynamic weighting model, three levels are classified: safe, partially manageable, and unmanageable.

[0107] Multiple driver status data are collected, including operational responses (such as steering wheel torque, pedal opening, and operation delay), cognitive states (such as eye tracking, pupil diameter changes, and facial features), and physiological signals (such as heart rate and skin conductivity). Based on the above response data, a dynamic weighted scoring model is used to analyze and derive cognitive, operational, and physiological scores. For example, the cognitive score can be calculated based on the time from system prompt to the driver's first operation: less than 0.5 seconds earns 1 point, and more than 2 seconds earns 0 points. Alternatively, it can be calculated based on the effective steering wheel torque being greater than or equal to 0.5 N. 1 point is awarded for a driver's visual focus duration (m), otherwise 0 points are awarded. Operational score can be calculated based on the percentage of time the driver's gaze is focused: 1 point for a duration greater than 80%, 0 points for a duration less than 30%. Alternatively, it can be calculated based on the driver's pupil dilation during an emergency: 1 point for a duration greater than 20%, otherwise 0 points. Physiological score can be calculated based on a 20% increase in the driver's heart rate from its resting value (stress activation), 0 points for no change in heart rate (fatigue / distraction). Alternatively, it can be calculated based on a sudden change in skin conductivity: 1 point for a change greater than or equal to 10%, otherwise 0 points.

[0108] Within the same dimension (cognitive, operational, and physiological dimensions), there may be multiple sub-indicators (e.g., the "cognitive dimension" includes three sub-indicators: "percentage of eye focus," "pupil dilation," and "blink frequency"). These need to be combined into a total score for a single dimension using a comprehensive formula, which can be obtained using the following formula.

[0109]

[0110] Where n represents the number of sub-indicators under this dimension. For example, if the score for gaze focus percentage in a cognitive dimension is 1, the score for pupil dilation is 0.5, and the score for blink frequency is 0.8, then the total score for the cognitive dimension is... .

[0111] Finally, a weighted summation method is used to calculate the overall score to determine the driver's takeover capability level. Takeover capability is categorized into levels: a score of 0.7 or higher indicates safe, a score of 0.3-0.7 indicates partial takeover, and a score less than 0.3 indicates no takeover capability. The formula can be:

[0112] The overall takeover capability score is calculated as follows: (Quantitative score of a certain status indicator × Weight of that indicator in the current scenario). This formula integrates multi-dimensional indicators of different importance into a quantitative value that reflects the driver's overall takeover capability.

[0113] For example, the current scene type is rural road, and the weights of each state data in the rural road scene are: cognitive state (weight 0.5) > operational response (weight 0.4) > physiological signal (weight 0.1).

[0114] On rural roads, the driver's eyes were closed for 40% of the time (cognitive score 0.3), the steering wheel torque was 1.2 N·m (operational score 0.8), and the physiological state was normal (physiological score 0.9). The total score was 0.3×0.5+0.8×0.4+0.9×0.1=0.56, which was judged as partial takeover.

[0115] This application improves the accuracy of capability assessment and reduces the false negative rate by integrating multi-dimensional indicators and combining them with a dynamic weighting model.

[0116] In some optional implementations of this embodiment, the current scenario type mentioned above includes highways, urban roads and rural roads, and the driver's multiple state data include operation response, cognitive state and physiological signals;

[0117] On the highway, the operational response weight is higher than the cognitive state weight and the physiological signal weight; on the urban road, the operational response weight is equal to the cognitive state weight, and the operational response weight is higher than the physiological signal weight; on the rural road, the cognitive state weight is higher than the operational response weight and the physiological signal weight.

[0118] Different scenarios place different emphasis on driver capabilities. On highways, high speeds require rapid driver responses in case of danger, making operational response a more critical factor. Urban roads present complex traffic conditions, demanding accurate judgment of traffic situations, where cognitive state and operational response are equally important. Rural roads may have poor visibility and conditions, requiring drivers to maintain high levels of focus, making cognitive state even more crucial. Therefore, setting different weighting relationships for different scenarios better reflects actual driving needs and improves the accuracy of takeover capability assessment. First, the system determines the current scenario type: highway, urban road, or rural road. Then, it determines the driver's takeover level based on the weighting allocation. Specifically, the weighting allocation can be as follows:

[0119] High speed: Operational response (0.5), cognitive state (0.3), physiological signals (0.2);

[0120] City: Operational response (0.4), cognitive state (0.4), physiological signals (0.2);

[0121] Rural areas: cognitive state (0.5), operational response (0.3), physiological signals (0.2).

[0122] For example, when a vehicle is traveling on a rural road, the driver's cognitive state score is 0.9, operational response score is 0.6, and physiological signal score is 0.7. Under rural road conditions, the weights for cognitive state are 0.5, operational response is 0.3, and physiological signal score is 0.2. Therefore, the takeover capability score is 0.9 × 0.5 + 0.6 × 0.3 + 0.7 × 0.2 = 0.45 + 0.18 + 0.14 = 0.77, which falls within the safe takeover level.

[0123] This application sets different sub-weights for specific scenarios by setting key factors in those scenarios, making the dynamic weight scoring model more aligned with the driving needs of different scenarios. This further improves the accuracy and rationality of driver takeover capability assessment and provides more reliable support for subsequent strategy execution.

[0124] In some optional implementations of this embodiment, the aforementioned environmental risk levels include low risk, medium risk, and high risk. The step of determining the takeover alert level based on the environmental risk levels specifically includes:

[0125] When the environmental risk level is low, the takeover alert level is no alert.

[0126] When the environmental risk level is medium risk, the takeover alert level is Level 1 alert, which includes mild tactile alert and mild visual alert;

[0127] When the environmental risk level is high, the takeover alert level is a level two alert, which includes both strong tactile and strong visual alerts.

[0128] Single-modal cues are easily ignored; for example, auditory cues become ineffective in noisy environments. Multimodal spatiotemporal binding is needed to enhance perception efficiency. Tactile and visual collaborative cues are triggered based on the environmental risk level, including a three-tiered alert strategy of low-risk, medium-risk, and high-risk. Specifically, the system first determines the current environmental risk level as low, medium, or high, and then triggers the corresponding takeover alert based on the risk level.

[0129] Low risk: No alerts will be issued.

[0130] Medium risk: Triggering a level one alert, such as the area of ​​the steering wheel corresponding to the risk direction vibrating at a frequency of 2Hz with an intensity of 0.5-1.0G, while a yellow border is projected onto the windshield in the risk direction and the distance is marked, and the projection flashes synchronously with the vibration rhythm.

[0131] High risk: Triggering a level 2 alert, such as the steering wheel vibrating continuously at a high frequency of 5Hz, while the seat vibrates at an intensity of 1.0-2.0G, and a red dynamic arrow and countdown progress bar are projected on the windshield, with a projection response delay of less than or equal to 100ms.

[0132] For example, a vehicle is driving on an urban road, and the environmental risk level is determined to be medium risk. At this time, the system triggers a level one alert, and the left side of the steering wheel (assuming there is a risk in the left front) vibrates at a frequency of 2Hz with an intensity of 0.8G. At the same time, a yellow border is projected onto the left front of the windshield, indicating "30m ahead", and flashes synchronously with the vibration rhythm.

[0133] This application achieves a tiered takeover alert system by setting multiple risk levels, which ensures that drivers receive appropriate alerts at different risk levels while reducing unnecessary interference and improving driving comfort and safety.

[0134] In some optional implementations of this embodiment, the driver's takeover capability level includes safe takeover, partial takeover, and no takeover. The step of determining the vehicle's risk execution strategy based on the driver's takeover capability level specifically includes:

[0135] When the driver's takeover capability level is the safe takeover, the risk execution strategy is a low-risk strategy, wherein the low-risk strategy includes warning deceleration and auxiliary braking;

[0136] When the driver's takeover capability level is partial takeover, the risk execution strategy is a medium-risk strategy, wherein the medium-risk strategy includes decelerating to a safe speed, lane keeping, slowing down to the side of the road, braking, and automatic avoidance.

[0137] When the driver's takeover capability level is "unable to take over", the risk execution strategy is a high-risk strategy, which includes braking to a stop, calling for rescue, emergency deceleration, and turning on hazard lights.

[0138] Fixed strategies (such as direct braking) can easily lead to passenger discomfort or traffic congestion; a balance must be struck between safety and traffic efficiency. A two-dimensional strategy based on environmental risk and takeover capability dynamically executes tiered strategies such as deceleration, avoidance, and braking. Specifically, the system first determines the driver's takeover capability level, and then executes the corresponding minimum-risk strategy based on that level, such as... Figure 4 As shown, Figure 4 This is the minimum risk implementation strategy table for the driver takeover monitoring method provided in this application.

[0139] Safety takeover: Performing pre-warning deceleration (such as braking deceleration of 0.2g) or auxiliary braking (such as braking deceleration of 0.4g), etc.

[0140] Partial takeover: Reduce the vehicle to a safe speed (such as 90km / h on highways, 40km / h on urban roads, and 30km / h on rural roads), while maintaining lane position, or applying braking (such as a braking deceleration of 0.3g), automatic avoidance, etc.

[0141] Unable to take over: Brake the vehicle to a stop and call for assistance, or perform emergency deceleration (such as 0.5g braking deceleration) and turn on the hazard lights, etc.

[0142] For example, if the driver's takeover capability level is partial, the vehicle is on an urban road with a medium-risk environment, a Level 1 warning is triggered (the steering wheel vibrates in the dangerous direction, and a yellow border is projected onto the windshield in the dangerous direction with a distance indicator). Simultaneously, the system executes a medium-risk strategy, reducing the vehicle speed to 40 km / h and activating lane keeping assist to prevent the vehicle from veering off course. If the driver's takeover capability level is no longer possible, the vehicle is on a highway with a high-risk environment, a Level 2 warning is triggered (the entire steering wheel vibrates continuously at a high frequency of 5Hz, and a red dynamic arrow and countdown progress bar are projected onto the windshield). Simultaneously, the system executes a high-risk strategy, performing emergency deceleration with a braking force of 0.5g to bring the vehicle to a stop and activating hazard lights. If the driver's takeover capability level is safe, the vehicle is on a rural road with a low-risk environment, and the system executes a low-risk strategy, performing pre-emptive deceleration and lane keeping assist.

[0143] This application sets up multiple takeover levels and determines the minimum risk execution strategy. It can take appropriate risk response strategies according to the driver's actual takeover ability, ensuring safety while reducing excessive intervention and improving passenger comfort and driving experience.

[0144] In some optional implementations of this embodiment, the tactile alerts include steering wheel area vibration and seat vibration, the visual alerts include AR projection, the operation responses include steering wheel torque, pedal opening and operation delay, the cognitive states include eye tracking, pupil diameter changes and facial features, and the physiological signals include heart rate and skin conductivity.

[0145] To make the entire driver takeover monitoring method clearer, more specific, and easier to understand and implement, and to provide a clear basis for system design, development, and testing, ensuring the system can accurately acquire relevant data and effectively monitor driver takeover capability and environmental risks, the following tactile cues are included: steering wheel area vibration and seat vibration; visual cues include AR projection; operational responses include steering wheel torque, pedal opening, and operation latency; cognitive states include eye tracking, pupil diameter changes, and facial features; and physiological signals include heart rate and skin conductivity. For example, tactile cues, visual cues, operational responses, cognitive states, and physiological signals can be:

[0146] Tactile cues: The vehicle's control system controls the corresponding area of ​​the steering wheel or seat to vibrate, transmitting different information in different vibration modes (frequency, intensity, etc.).

[0147] Visual cues: Using AR projection technology, relevant visual information, such as borders, arrows, and text, can be projected onto the windshield.

[0148] Operation response monitoring: Sensors detect steering wheel torque, pedal opening, and time from prompt to first operation.

[0149] Cognitive state monitoring: The driver's gaze direction and fixation duration are monitored through eye-tracking devices, and changes in pupil diameter and facial features (such as blinking frequency) are detected through cameras.

[0150] Physiological signal monitoring: The driver's physiological indicators such as heart rate and skin conductivity are monitored through corresponding sensors.

[0151] When the secondary haptic alert is triggered, the system controls the entire steering wheel to vibrate continuously at a high frequency of 5Hz, while the seat also vibrates, with the intensity increasing as the TTC decreases. For visual alerts, an AR projection displays a red dynamic arrow pointing in the safe operating direction and a countdown progress bar on the windshield. During operation response monitoring, the sensors detect a steering wheel torque ≥0.5N. m indicates effective contact.

[0152] Further reference Figure 5 , Figure 5 This is a schematic diagram of the driver takeover monitoring device provided in this application, serving as a reference to the above. Figure 2 The implementation of the method shown in this application provides an embodiment of a driver takeover monitoring device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0153] like Figure 5 As shown, the driver takeover monitoring device 500 described in this embodiment includes: a data acquisition module 501, an environmental risk determination module 502, a takeover alert module 503, a takeover capability determination module 504, and an execution strategy module 505. Wherein:

[0154] The data acquisition module 501 is used to acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver.

[0155] The environmental risk determination module 502 is used to determine the current scene type and target collision time of the vehicle based on multiple environmental data, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment, and is also used to determine the environmental risk level based on the current scene type and the target collision time;

[0156] The takeover alert module 503 is used to determine the takeover alert level based on the environmental risk level, wherein the takeover alert includes tactile alerts and visual alerts;

[0157] The takeover capability determination module 504 is used to determine the driver's takeover capability level based on the current scenario type and multiple state data.

[0158] The execution strategy module 505 determines the risk execution strategy for the vehicle based on the driver's takeover capability level.

[0159] The driver takeover monitoring device provided in this application improves the driver's perception efficiency and response speed through multimodal collaborative prompts. A dynamic weighted scoring model accurately assesses the driver's takeover capability, and combined with a two-dimensional strategy based on environmental risk levels, it achieves flexibility and safety in low-risk operations.

[0160] To address the aforementioned technical problems, this application also provides an autonomous vehicle, including the aforementioned driver takeover monitoring device, which is used to execute the driver takeover monitoring method of any of the above embodiments or implementations.

[0161] In complex traffic scenarios, autonomous vehicles need to promptly request driver intervention when the system is unable to handle the situation. The driver's ability to take over and their responsiveness directly impact driving safety. Traditional autonomous vehicles may suffer from limitations such as simplistic alert methods and fixed risk strategies, making it difficult to adapt to different scenarios and individual driver differences. Therefore, equipping vehicles with dedicated driver takeover monitoring devices allows them to more intelligently assess risks, alert the driver, and implement safety strategies, bridging the safety gap between autonomous driving and manual intervention.

[0162] The vehicle integrates multiple sensors (such as cameras, radar, steering wheel torque sensors, eye-tracking devices, physiological signal monitors, etc.) to provide environmental data (such as obstacle distance, relative speed, road type) and driver status data (such as operation response, eye trajectory, heart rate, etc.) to the driver take over the monitoring device.

[0163] The data acquisition module in the driver takeover monitoring device collects the aforementioned sensor data and transmits it to the environmental risk determination module and the takeover capability determination module.

[0164] The environmental risk assessment module determines the scenario type and time of collision (TTC) based on environmental data, and then classifies the environmental risk level; the takeover capability assessment module combines scenario type and driver status data, and evaluates the takeover capability level through a dynamic weighted scoring model.

[0165] The takeover alert module triggers corresponding tactile (steering wheel, seat vibration) and visual (AR projection) alerts based on the environmental risk level; the execution strategy module executes the corresponding minimum risk strategy (such as deceleration, lane keeping, braking to a stop, calling for help, etc.) based on the driver's takeover ability level and the environmental risk level.

[0166] Throughout the process, the device continuously optimizes parameters through a closed-loop learning mechanism to adapt to the habits of different drivers and complex scenarios.

[0167] The autonomous vehicle provided in this application enhances the driver's perception efficiency and response speed to prompts through multimodal collaborative prompts. A dynamic weighted scoring model accurately assesses the driver's takeover capability, and a two-dimensional matrix strategy based on environmental risk levels achieves flexibility and safety in low-risk operations. A full-link closed-loop learning mechanism continuously optimizes individual adaptation and scenario response, enhancing the system's reliability and intelligence in complex traffic environments.

[0168] This application clarifies the specific content of each part, providing clear technical indicators and basis for the implementation of the system, ensuring that the system can accurately and effectively collect data and provide alerts, and improving the operability and reliability of the entire method.

[0169] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0170] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for monitoring driver takeover, characterized in that, The method includes: Acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver; Based on multiple environmental data, the current scene type and target collision time of the vehicle are determined, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment. Based on the current scene type and the target collision time, determine the environmental risk level; Based on the environmental risk level, a takeover alert level is determined, wherein the takeover alert includes tactile alerts and visual alerts; Based on the current scenario type and multiple state data, a dynamic weight scoring model is used to determine the score and weight of each state data of the driver. The dynamic weight scoring model is used to score each state data of the driver and determine the weight of each state data according to the current scenario type. The current scenario type includes highways, urban roads and rural roads. The multiple state data of the driver include operation response, cognitive state and physiological signals. Based on the scores and weights corresponding to each state data, the driver's takeover ability score is obtained. Specifically, on highways, the weight of the operational response is higher than the weight of the cognitive state and the weight of the physiological signal. On urban roads, the weight of the operational response is equal to the weight of the cognitive state, but the weight of the operational response is higher than the weight of the physiological signal. On rural roads, the weight of the cognitive state is higher than the weight of the operational response and the weight of the physiological signal. Based on the driver's takeover ability score, the driver's takeover ability level is determined; Based on the driver's takeover capability level, a risk execution strategy for the vehicle is determined.

2. The method according to claim 1, characterized in that, The determination of the environmental risk level based on the current scene type and the target collision time includes: A target threshold table is determined based on the current scenario type, wherein the target threshold table is a threshold range table corresponding to each preset environmental risk level of the current scenario type; The environmental risk level is determined based on the target collision time and the target threshold table.

3. The method according to claim 2, characterized in that, The determination of the environmental risk level based on the target collision time and the target threshold table includes: The target threshold table is corrected based on the driver's historical data to obtain a corrected threshold table; The environmental risk level is determined based on the collision time and the correction threshold table.

4. The method according to claim 1, characterized in that, The environmental risk levels include low risk, medium risk, and high risk. Determining the takeover alert level based on the environmental risk levels includes: When the environmental risk level is low, the takeover alert level is no alert. When the environmental risk level is medium risk, the takeover alert level is Level 1 alert, which includes mild tactile alert and mild visual alert; When the environmental risk level is high, the takeover alert level is a level two alert, which includes both strong tactile and strong visual alerts.

5. The method according to any one of claims 1-4, characterized in that, The driver's takeover capability level includes safe takeover, partial takeover, and no takeover. The determination of the vehicle's risk execution strategy based on the driver's takeover capability level includes: When the driver's takeover capability level is the safe takeover, the risk execution strategy is a low-risk strategy, wherein the low-risk strategy includes warning deceleration and auxiliary braking; When the driver's takeover capability level is partial takeover, the risk execution strategy is a medium-risk strategy, wherein the medium-risk strategy includes decelerating to a safe speed, lane keeping, slowing down to the side of the road, braking, and automatic avoidance. When the driver's takeover capability level is "unable to take over", the risk execution strategy is a high-risk strategy, which includes braking to a stop, calling for rescue, emergency deceleration, and turning on hazard lights.

6. The method according to claim 1, characterized in that, The tactile cues include steering wheel area vibration and seat vibration; the visual cues include AR projection; the operational responses include steering wheel torque, pedal opening, and operation delay; the cognitive states include eye tracking, pupil diameter changes, and facial features; and the physiological signals include heart rate and skin conductivity.

7. A driver takeover monitoring device, characterized in that, The method for monitoring driver takeover as described in any one of claims 1-6 is characterized by comprising: The data acquisition module is used to acquire multiple environmental data and multiple driver status data, wherein the environmental data is the current driving environment data of the vehicle, and the status data is the current status data of the driver; The environmental risk determination module is used to determine the current scene type and target collision time of the vehicle based on multiple environmental data, wherein the target collision time is the time required for the vehicle to collide with an obstacle in the current driving environment, and is also used to determine the environmental risk level based on the current scene type and the target collision time; The takeover alert module is used to determine the takeover alert level based on the environmental risk level, wherein the takeover alert includes tactile alerts and visual alerts; The takeover capability determination module is used to determine the driver's takeover capability level based on the current scenario type and multiple state data. The execution strategy module determines the risk execution strategy for the vehicle based on the driver's takeover capability level.

8. An autonomous vehicle, characterized in that, The device includes a driver takeover monitoring device, which is used to perform the driver takeover monitoring method according to any one of claims 1 to 6.

Images