Automatic driving control method and system with safety takeover and man-machine cooperation

Abstract

The invention discloses a safe takeover and man-machine cooperation automatic driving control method. The method comprises the following steps: acquiring a face image, eye movement data, a head posture, a sight direction, a blinking frequency and a steering wheel holding state of a driver in real time through a vehicle-mounted camera and a steering wheel torque sensor; the collected driver state information is input into a driver ability evaluation module constructed based on a time sequence model, a driver ability level C is output, and the ability level C is divided into four levels of C0 to C3; and acquiring current driving scene information through an environment sensing sensor carried by the vehicle. According to the invention, all-weather and full-scene environment understanding is realized, the camera provides semantic information of lane lines and traffic signs, the millimeter-wave radar can still accurately measure distance at rainy and foggy nights, the laser radar generates high-precision three-dimensional point cloud, depicts the contour of an obstacle, and injects prior knowledge into a high-precision map.

Description

Technical Field

[0001] This invention belongs to the field of safe autonomous driving technology, specifically a safe takeover and human-machine collaborative autonomous driving control method and system. Background Technology

[0002] With the development of autonomous driving technology, Level 2 to Level 3 autonomous driving functions are gradually being put into use. However, these systems still require driver intervention and takeover in complex road conditions, special weather conditions, and sensor failures. Currently, most existing autonomous driving takeover logic relies on the driver's real-time state judgment, ignoring the combined factors of driver attention duration, fatigue, and environmental complexity. Furthermore, existing takeover systems often lack dynamic takeover decisions and strategies, making them prone to takeover failure or delayed takeover in specific situations, posing potential safety risks. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, the present invention provides a safe takeover and human-machine collaboration automatic driving control method and system to at least partially solve the above-mentioned technical problems.

[0004] The technical solution adopted in this invention is as follows: This invention proposes a safe takeover and human-machine collaboration method for autonomous driving control, comprising the following steps: Step 1: Real-time acquisition of the driver's facial images, eye movement data, head posture, gaze direction, blink frequency, and steering wheel grip status using the vehicle-mounted camera and steering wheel torque sensor. Step 2: Input the collected driver status information into the driver capability assessment module built based on the time series model, and output the driver capability level C, which is divided into four levels from C0 to C3; Step 3: Obtain current driving scene information through the environmental perception sensors on the vehicle. The scene information includes vehicle speed, relative speed to the vehicle in front, time to collision (TTA), road type, weather conditions, lane line clarity, obstacle location, traffic density, and sensor health. Step 4: Input the scenario information into the scenario risk assessment module and output the scenario risk level R, which is divided into four levels from R0 to R3; Step 5: Input the driver's ability level C and the scenario risk level R into the takeover decision module. Determine whether to trigger a takeover request, the prompting method, the prompting intensity, and the takeover timing based on the preset takeover strategy matrix. If no effective manual takeover operation is detected within the preset takeover response time, activate the minimum risk maneuver module to perform deceleration, lane change, or pull over operation, and simultaneously turn on the hazard warning lights.

[0005] In one embodiment of the present invention, the time series model is a Long Short-Term Memory Network (LSTM) or a Convolutional Neural Network (CNN) used to model driver state information within a continuous time window to predict takeover capability within the next 1 to 3 seconds.

[0006] In one embodiment of the present invention, the environmental perception sensor includes a forward-looking camera, a millimeter-wave radar, a lidar, and a GNSS positioning module. Data from each sensor is transmitted to a central domain controller via a CAN bus or Ethernet, and is fused and processed by a scene risk assessment module in the central domain controller.

[0007] In one embodiment of the present invention, the notification method for the takeover request includes voice broadcast, visual warning on the dashboard or head-up display, steering wheel vibration, and seat vibration. The intensity and combination of the multimodal notifications are dynamically adjusted based on the driver's historical response behavior.

[0008] In one embodiment of the present invention, when performing a sidewalk parking maneuver, the minimum risk maneuver module prioritizes selecting the right emergency lane or shoulder and continuously monitors surrounding obstacles during the parking process. If a risk of lateral collision is detected, the lane change action is suspended and the current lane is maintained while decelerating to a stop.

[0009] In one embodiment of the present invention, the system of the safe takeover and human-machine collaborative automatic driving control method includes: The driver status acquisition module includes an infrared camera installed on the top of the cockpit, a steering wheel torque sensor embedded in the steering wheel, and an eye-tracking sensor located on the inside of the A-pillar, which are used to acquire the driver's facial image, gaze direction, blink frequency, head posture, and hand grip status in real time. The driver capability assessment module is electrically connected to the driver status acquisition module and is deployed in the vehicle's central computing unit. It is used to receive and process the driver status information and output the driver capability level C. The scenario risk assessment module is electrically connected to the vehicle's environmental perception sensor group, which includes a forward-facing camera, a forward-facing millimeter-wave radar, a side-facing lidar, and a high-precision map receiving unit, and is used to fuse environmental data and output the scenario risk level R. The takeover decision module is communicatively connected to the driver capability assessment module and the scenario risk assessment module, and is used to determine the takeover strategy by looking up a table based on the capability level C and the risk level R. The multimodal prompting module, controlled by the takeover decision module, includes a speaker integrated into the dashboard, a HUD projection unit, a steering wheel vibration motor, and a tactile actuator at the bottom of the seat; The minimum risk maneuvering module communicates with the vehicle's longitudinal and lateral actuators and is used to control the braking system, steering motor, and hazard warning lights to perform preset safety actions when no effective takeover is detected.

[0010] In one embodiment of the present invention, the infrared camera in the driver state acquisition module is fixed below the sun visor with its optical axis pointing towards the driver's eye area. The eye-tracking sensor is embedded in the interior panel of the left A-pillar, and its field of view covers the driver's eyes and the bridge of the nose area. The steering wheel torque sensor is integrated inside the steering wheel frame and detects the distribution of gripping force of both hands through a strain gauge array.

[0011] In one embodiment of the present invention, the forward-looking camera in the environmental perception sensor group is installed in the center of the inner side of the windshield, the millimeter-wave radar is embedded in the middle of the front bumper, the lidar is arranged on the longitudinal axis of the roof, and the high-precision map receiving unit establishes a communication link with the cloud map server through the vehicle-mounted T-Box.

[0012] In one embodiment of the present invention, the HUD projection unit in the multimodal prompt module is optically coupled to the lower area of ​​the windshield, the speakers are arranged inside the air vents on the left and right sides of the dashboard, the steering wheel vibration motor is fixed inside the steering wheel rim, and the seat tactile actuator is embedded under the foam layer of the driver's seat cushion and backrest.

[0013] In one embodiment of the present invention, the minimum risk maneuvering module is connected to the Electronic Stability Program (ESP) controller, the Electric Power Steering (EPS) controller, and the Body Control Module (BCM) via a CANFD bus. It is used to send braking commands, target steering angle values, and hazard warning light activation signals, and to receive obstacle feedback from the environmental perception sensor group in real time during execution to dynamically adjust the trajectory.

[0014] The beneficial effects of the technical solution of this invention are as follows: This invention detects early signs of fatigue in drivers, such as increased blinking frequency, slow head drooping, and weakened steering wheel grip, even before the driver closes their eyes. The system can predict that the driver will enter level C1 (severe distraction) within 2 seconds, thus initiating a mild warning in advance to avoid a passive response only when the driver has completely lost their ability to function (C0). The LSTM gating mechanism effectively filters out brief disturbances (such as looking down at the central control screen for 1 second), escalating the alarm only when the condition continues to deteriorate. The CNN, through local temporal convolution, quickly identifies sudden distractions (such as suddenly turning the head to look out the window). The two complement each other to improve robustness. The model can be fine-tuned online to gradually learn the baseline behavioral patterns of specific drivers (such as the naturally higher blinking frequency in older drivers), reducing the false trigger rate over long-term use.

[0015] This invention utilizes a forward-looking camera, millimeter-wave radar, lidar, and GNSS / high-precision geodetic homing.Figure 4 The system leverages redundant sensing capabilities to achieve all-weather, all-scenario environmental understanding. Cameras provide semantic information on lane lines and traffic signs, millimeter-wave radar maintains accurate distance measurement even in rain, fog, and nighttime conditions, and lidar generates high-precision 3D point clouds to delineate obstacle outlines. High-precision maps incorporate prior knowledge (such as construction zones ahead or sharp bends without guardrails). Even if one sensor fails (e.g., a camera is covered in mud), the remaining sensors can still maintain basic risk assessment. The scenario risk assessment module not only calculates static indicators (vehicle speed, road type) but also integrates dynamic factors (time-to-collision time (TTA), traffic density change rate, and sensor health decay coefficient) to output structured R0–R3 levels.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a first flowchart of the safe takeover and human-machine collaboration autonomous driving control method proposed in an embodiment of the present invention; Figure 2 This is a second flowchart of the automated driving control method with safe takeover and human-machine collaboration proposed in an embodiment of the present invention. Figure 3 This is a system module diagram of the safe takeover and human-machine collaboration automatic driving control method proposed in an embodiment of the present invention; Figure 4 This is a first functional block diagram of the system of the safe takeover and human-machine collaboration automatic driving control method proposed in an embodiment of the present invention; Figure 5 This is a second functional module diagram of the system for the safe takeover and human-machine collaborative automatic driving control method proposed in this embodiment of the invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] The following description, with reference to the accompanying drawings, illustrates an embodiment of the present invention: a safe takeover and human-machine collaborative automatic driving control method and system.

[0020] like Figures 1 to 5As shown, this embodiment of the invention provides a safe takeover and human-machine collaboration-based autonomous driving control method, including the following steps: Step 1: Real-time acquisition of the driver's facial images, eye movement data, head posture, gaze direction, blink frequency, and steering wheel grip status using the vehicle-mounted camera and steering wheel torque sensor. Step 2: Input the collected driver status information into the driver capability assessment module built based on the time series model, and output the driver capability level C, which is divided into four levels from C0 to C3. Step 3: Obtain current driving scene information through the environmental perception sensors on the vehicle. Scene information includes vehicle speed, relative speed to the vehicle in front, time to collision (TTA), road type, weather conditions, lane line clarity, obstacle location, traffic density, and sensor health. Step 4: Input the scenario information into the scenario risk assessment module and output the scenario risk level R. The risk level R is divided into four levels from R0 to R3. Step 5: Input the driver's ability level C and the scenario risk level R into the takeover decision module. Based on the preset takeover strategy matrix, determine whether to trigger a takeover request, the prompt method, the prompt intensity, and the takeover timing. If no effective manual takeover operation is detected within the preset takeover response time, activate the minimum risk maneuver module to perform deceleration, lane change, or pull over operation, and simultaneously turn on the hazard warning lights.

[0021] In a specific application, the system first uses multi-source sensors arranged in the cockpit to continuously and non-intrusibly collect the driver's physiological and behavioral state. The vehicle-mounted infrared camera is fixed below the sun visor or inside the A-pillar, with its optical axis aligned with the driver's eye area to capture key facial features, pupil position, eyelid opening and closing frequency, and three-dimensional head posture. Simultaneously, a steering wheel torque sensor embedded within the steering wheel frame uses a strain gauge array to detect in real-time the torque distribution applied by the hands to the wheel rim, grip duration, and minute vibration characteristics. The raw signals are synchronously fed into the central computing unit and, after preprocessing (such as denoising, normalization, and temporal alignment), are used as input to a pre-trained time-series neural network model. This model can be a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU), or a Transformer-based temporal encoder. Its function is to predictively model the driver's readiness to take over in the next 1 to 3 seconds. The model output is quantized into four discrete levels C0–C3: C0 represents complete loss of takeover ability (e.g., falling asleep, sudden illness); C1 represents severely distracted attention but strong response to stimuli; C2 represents partial return of attention but significant operational delay; and C3 represents complete wakefulness, good hand-eye coordination, and immediate takeover ability.

[0022] Meanwhile, the vehicle's environmental perception subsystem operates synchronously, constructing a multi-dimensional understanding of the current driving scenario through forward-facing cameras, millimeter-wave radar, lidar, GNSS / IMU integrated navigation, and high-precision map data. The subsystem not only acquires static information (such as road curvature, lane line type, and speed limit signs), but also focuses on extracting dynamic risk factors: including the relative speed between the vehicle and the vehicle in front, time-to-collision (TTA), lateral offset, surrounding vehicle density, obstacle intrusion probability, the impact of weather on sensor performance (such as rain and fog causing sparse lidar point clouds), and the health status of each sensor (such as whether the camera is fogged or whether the radar is subject to electromagnetic interference). The heterogeneous data is integrated through sensor fusion algorithms (such as Kalman filtering, multi-hypothesis tracking, or multimodal feature stitching) and then input into the scene risk assessment module. The module also adopts a hierarchical strategy, dividing the overall risk into four levels: R0-R3. R0 represents low-risk scenarios (such as highway cruising, sparse traffic, and clear weather), R1 represents low to medium risk (such as urban expressways and occasional vehicles cutting in), R2 represents medium to high risk (such as dense traffic, construction zones, and insufficient nighttime lighting), and R3 represents high-risk emergency scenarios (such as the sudden appearance of stationary obstacles ahead, pedestrians crossing the road, and the risk of multi-vehicle rear-end collisions). The risk level is determined by weighted fusion of risk scores from multiple dimensions and the introduction of a context-aware mechanism (for example, even if the vehicle speed is not high in a curve, if the lane lines are blurred and there are no guardrails, the risk level will automatically increase).

[0023] After independently assessing the driver's competence level (C) and the scenario risk level (R), the system enters the core takeover decision-making stage. It uses a pre-defined "takeover strategy matrix" as the decision-making center, with C as the vertical axis and R as the horizontal axis. Each intersecting cell corresponds to a specific takeover strategy, including: whether to immediately trigger a takeover request, the prompting method (voice / visual / tactile), the prompt intensity (volume, flashing frequency, vibration amplitude), the takeover buffer time (the time window from issuing the request to mandatory intervention), and whether to allow degraded operation (such as simply maintaining lane centering instead of full-speed cruise). For example, when C=3 and R=1, the system only needs to provide a slight flashing prompt on the HUD; while when C=1 and R=3, it will simultaneously activate a high-volume voice alarm, high-frequency steering wheel vibration, and pulsed seat vibration, compressing the takeover response time to within 3 seconds. If, within this preset time, the system fails to detect any operation that meets the definition of "effective takeover" (such as continuously gripping the steering wheel for more than 1 second and applying steering force, or pressing the brake pedal for more than a threshold) through the steering wheel torque sensor, accelerator / brake pedal displacement, or eye tracking, it will determine that the takeover has failed and immediately activate the Minimum Risk Maneuver (MRM) module.

[0024] The MRM module dynamically plans a safe exit trajectory based on the current vehicle status and surrounding environment. It first uses a high-precision map and real-time perception data to identify the right-hand emergency lane, shoulder, or the nearest exit ramp as the target parking area. Then, the longitudinal controller gradually reduces the vehicle speed (to avoid sudden deceleration and rear-end collisions), while the lateral controller slowly shifts to the right, ensuring no lateral obstructions. Throughout this process, hazard lights automatically activate, the V2X module broadcasts the vehicle's abnormal status to surrounding vehicles, and the in-vehicle speaker continuously announces, "The vehicle is pulling over; please do not approach." If a high-speed approaching vehicle or non-motorized vehicle is detected on the right during the lane change, the lateral movement is paused, the vehicle prioritizes decelerating to a stop in the current lane, and a reassessment is made to ensure a safe pull-over.

[0025] In one specific implementation, the time series model is a Long Short-Term Memory Network (LSTM) or a Convolutional Neural Network (CNN) used to model the driver's state information within a continuous time window to predict takeover capability in the next 1 to 3 seconds. The environmental perception sensors include a forward-looking camera, millimeter-wave radar, lidar, and a GNSS positioning module. The data from each sensor is transmitted to the central domain controller via a CAN bus or Ethernet, and is fused and processed by the scene risk assessment module in the central domain controller.

[0026] In a specific application of this invention, after the system starts, a sensing front-end deployed in the cockpit continuously collects multi-dimensional state signals of the driver. These signals include, but are not limited to: facial image sequences captured by an infrared camera (used to extract eyelid opening and closing frequency, pupil position, and head pitch / yaw angle); grip force distribution and micro-vibration characteristics output by a torque sensor embedded in the steering wheel; and coordinates of the gaze point provided by an optional eye-tracking module. All raw data are synchronously recorded at a sampling frequency of no less than 30Hz and timestamped. The state sequence within a continuous time window, typically covering historical information from the past 2 to 5 seconds, is fed as input into a temporal prediction model built on a deep learning architecture. The model specifically employs a Long Short-Term Memory (LSTM) network or a one-dimensional convolutional neural network (1D-CNN), both of which excel at capturing non-linear dependencies in the temporal dimension. LSTM effectively mitigates the vanishing gradient problem through its gating mechanism, enabling it to memorize behavioral patterns over long periods (such as the gradual transition from focus to distraction); while 1D-CNN, through sliding convolutional kernels, rapidly extracts local temporal features (such as sudden blinks or head turns), offering greater computational efficiency. Based on current and historical states, it predicts whether the driver possesses the physiological and cognitive abilities to perform an effective takeover operation within the next 1 to 3 seconds. The prediction results are quantified as discrete takeover capability levels (such as C0–C3), essentially a probabilistic estimate of "takeover readiness" rather than a static snapshot, thus endowing the system with the ability to proactively adapt to the uncertainties of human behavior.

[0027] Meanwhile, the vehicle's external environmental perception subsystem operates in parallel, constructing a real-time three-dimensional semantic understanding of the driving scene. A forward-facing camera, mounted in the central area inside the windshield, provides high-resolution visual information for identifying lane line types, traffic signs, the outline of vehicles ahead, and pedestrian postures. A forward-facing millimeter-wave radar, embedded in the center of the bumper, is capable of all-weather operation, measuring the distance, relative speed, and azimuth of targets, especially compensating for visual deficiencies in rain, fog, or strong light conditions. A lidar system positioned along the longitudinal axis of the roof generates dense point clouds for high-precision obstacle geometry modeling and free-space detection. A GNSS positioning module, combined with an IMU inertial unit and high-precision map data, provides centimeter-level positioning and vehicle motion status (such as yaw rate and acceleration). Data streams from these four heterogeneous sensors are transmitted in real-time to the central domain controller via a high-speed in-vehicle communication network (such as CANFD or in-vehicle Ethernet). Here, the scenario risk assessment module performs multimodal data fusion: first, it aligns the data from each sensor in time and space (using timestamps and pose information); then, it employs feature-level or decision-level fusion strategies (such as weighted fusion based on attention mechanisms or Bayesian inference) to generate a unified environmental situation map. This map not only includes static elements (road topology, speed limit information) but also focuses on quantifying dynamic risk factors. For example, it calculates time-to-collision (TTA) using the distance to the vehicle ahead and relative speed, combines this with LiDAR point clouds to determine the probability of an obstacle intruding into the lane, uses camera images to assess lane line clarity and weather visibility, and introduces sensor health indicators (such as radar signal-to-noise ratio, camera exposure anomalies) to attenuate and correct perception confidence. Finally, all risk dimensions are comprehensively mapped into structured scenario risk levels (such as R0–R3), which dynamically reflect the urgency and operational difficulty of human intervention in the current driving situation.

[0028] The engine performs joint reasoning based on a pre-defined "capability and risk matching rule base." For example, if it predicts that the driver will be at level C1 (severely distracted) within the next 2 seconds and the environment suddenly changes to level R3 (sudden stationary obstacle ahead), the system will immediately trigger the highest priority takeover request and simultaneously pre-activate the braking and steering actuators to shorten the response delay. Conversely, if C is C3 (fully alert) and R is R0 (stable cruise on a highway), only a low-interference warning will be maintained or takeover will not be triggered temporarily. The takeover capability prediction output by the LSTM / CNN model is forward-looking, enabling the system to initiate the human-machine handover process before an actual danger occurs, rather than passively waiting for an emergency to trigger. At the same time, the multi-source fusion of environmental perception data ensures the reliability of risk assessment and avoids misjudgment due to the failure of a single sensor.

[0029] In one specific implementation, the takeover request is prompted in the following ways: voice broadcast, visual warning on the dashboard or head-up display, steering wheel vibration, and seat vibration. The intensity and combination of the multimodal prompts are dynamically adjusted according to the driver's historical response behavior. When performing a pull-over, the minimum risk maneuver module prioritizes the right emergency lane or shoulder and continuously monitors surrounding obstacles during the parking process. If a side collision risk is detected, the lane change action is suspended and the current lane is maintained while decelerating to a stop.

[0030] In specific applications, this invention treats the driver as a cognitive subject with individual differences and behavioral habits, and improves their response efficiency by dynamically adjusting the prompting strategy. At the same time, when manual takeover fails, the minimum risk maneuver module executes intelligent risk avoidance actions with environmental perception capabilities to ensure that the system is always within a controllable safety boundary.

[0031] Once the takeover decision module determines that a takeover request needs to be triggered based on driver capability assessment and scenario risk analysis, the system immediately activates the multimodal prompt engine. The engine constructs a personalized response model based on the driver's historical behavior data. For example, the system records the driver's first response time, operation confirmation delay, and frequency of misoperation for different prompt channels (voice, vision, and tactile) in each takeover event, and establishes a preference weight vector accordingly. For drivers who are accustomed to reacting quickly through steering wheel vibration, the system will prioritize increasing the intensity of the tactile channel in high-risk scenarios (e.g., increasing the frequency of the vibration motor to 150Hz and extending its duration), while supplementing it with concise voice commands (e.g., "Please take over immediately!") and a high-contrast red border flashing on the HUD; while for users who are more sensitive to visual information, a dynamic arrow is superimposed in the central area of ​​the instrument panel, and the voice volume is reduced to avoid startling them. Dynamic adjustment is not only reflected in channel selection but also in the gradient evolution of prompt intensity: if no effective operation (e.g., change in steering wheel torque or pedal displacement) is detected within 500 milliseconds after the first prompt, the system will automatically upgrade the prompt level. For example, the linear resonant actuator (LRA) at the bottom of the seat is changed from a single pulse to a continuous waveform vibration. At the same time, the HUD warning area is expanded and accompanied by a higher beep tone. The entire prompting process has a "progressive wake-up" characteristic in terms of time, which avoids overstimulation and panic, and ensures that attention is recalled within the critical window period.

[0032] At the same time, the system background simultaneously starts the takeover response monitoring module to continuously verify whether the driver has truly completed effective takeover. The module not only relies on the steering wheel grip state, but also integrates eye tracking (confirming that the gaze has returned to the road), pedal operation (braking or acceleration intention), and consistency judgment of the vehicle's lateral / longitudinal control input. Only when multiple channel signals are aligned in time and the amplitude exceeds the preset threshold is the takeover considered successful; otherwise, it is considered a takeover failure.

[0033] If a valid takeover is not confirmed within the preset takeover response time (typically 3–8 seconds, dynamically set according to the risk level), the system immediately and seamlessly switches to Minimum Risk Maneuver (MRM) mode. At this point, the vehicle no longer requires human intervention but autonomously executes an emergency degrading strategy aimed at a "safe stop." The MRM module first utilizes high-precision map data and real-time perception information to identify whether there is an emergency lane, hard shoulder, or open area for temporary parking on the right side of the road. On highways or urban expressways, the system prioritizes planning a rightward deviation trajectory, aiming to bring the vehicle to a complete stop on the shoulder without obstructing traffic on the main road. However, the lane-changing maneuver involves continuous lateral obstacle monitoring throughout the process: lateral millimeter-wave radar and blind-spot cameras scan the adjacent lane and shoulder area on the right in real time to detect approaching vehicles, non-motorized vehicles, guardrail ends, or construction cones at high speed. If a lateral collision risk is detected early in the lane change (e.g., a motorcycle approaching from the right at a relative speed >20 km / h), the MRM controller will immediately abort the lateral movement command, maintain the current lane center, and activate the longitudinal deceleration strategy. It will apply progressive braking force through the Electronic Stability Program (ESP) to control the deceleration within a comfortable threshold (typically ≤3 m / s²), while simultaneously activating the hazard warning lights and broadcasting the vehicle's abnormal status to the V2X network. During deceleration to a complete stop, the system continues to monitor the distance and speed of vehicles approaching from behind. If a high risk of a rear-end collision is determined, the final stopping position can be fine-tuned (e.g., slightly closer to the right but not completely changing lanes) to maximize rear buffer space.

[0034] In one specific implementation, the system of the safe takeover and human-machine collaborative automatic driving control method includes: The driver status acquisition module includes an infrared camera installed on the top of the cockpit, a steering wheel torque sensor embedded in the steering wheel, and an eye-tracking sensor located on the inside of the A-pillar, which are used to acquire the driver's facial image, gaze direction, blink frequency, head posture, and hand grip status in real time. The driver competence assessment module is electrically connected to the driver status acquisition module and is deployed in the vehicle's central computing unit. It is used to receive and process driver status information and output the driver competence level C. The scenario risk assessment module is electrically connected to the vehicle's environmental perception sensor group, which includes a forward-facing camera, a forward-facing millimeter-wave radar, a side-facing lidar, and a high-precision map receiving unit. It is used to fuse environmental data and output the scenario risk level R. The takeover decision module is connected to the driver capability assessment module and the scenario risk assessment module respectively, and is used to determine the takeover strategy by looking up a table based on the capability level C and the risk level R. The multimodal prompting module, controlled by the takeover decision module, includes speakers integrated into the dashboard, a HUD projection unit, a steering wheel vibration motor, and haptic actuators under the seats; The minimum risk maneuvering module communicates with the vehicle's longitudinal and lateral actuators and is used to control the braking system, steering motor, and hazard warning lights to perform preset safety actions when no effective takeover is detected.

[0035] In a specific application of this invention, after the system is started, a multi-source sensor array deployed in the cockpit first collects the driver's state non-intrusively and continuously. An infrared camera installed below the sun visor or in the center of the roof continuously captures facial images of the driver at a frame rate of 30–60Hz. Its wavelength is optimized to clearly identify the eyelid contours and pupil center even in strong light or nighttime environments. The steering wheel torque sensor embedded in the steering wheel frame detects the magnitude, direction, and distribution pattern of the torque applied to the wheel rim by both hands in real time through a distributed strain gauge array, thereby determining whether the driver is in an active grip state. The eye-tracking sensor located inside the interior panel of the left A-pillar uses a near-infrared light source and a high-speed CMOS imager to accurately capture the gaze point, fixation duration, and scanning path of both eyes.

[0036] The evaluation module runs on a high-performance domain controller in the vehicle (such as a computing platform based on Orin or Thor chips), and integrates a pre-trained deep temporal neural network model (such as LSTM or 1D-CNN). The model extracts features and models the context of a multi-dimensional state sequence over the past 2–5 seconds, not only identifying the current state (such as whether the blinking frequency exceeds a threshold), but also predicting whether the driver has the cognitive and physiological ability to complete effective steering or braking operations within the next 1–3 seconds. The output is structured into four capability levels C (C0–C3), each level corresponding to a different confidence interval for takeover readiness, and serves as a key input variable for subsequent decisions.

[0037] Meanwhile, the vehicle's external environmental perception subsystem operates in parallel. A forward-facing camera, mounted on the inner optical center of the windshield, provides semantic segmentation and target detection information for the road ahead. A forward-facing millimeter-wave radar, embedded in the center of the front bumper, measures the distance, speed, and orientation of obstacles ahead in all weather conditions. Lateral lidar is positioned on the roof or fenders to generate high-density point clouds to characterize the geometry of adjacent lanes and shoulders. A high-precision map receiving unit maintains a connection with the cloud server via a T-Box, acquiring centimeter-level road topology, speed limit information, and construction zone markings. Heterogeneous data is aggregated to the same central computing unit via CANFD or in-vehicle Ethernet, where the scenario risk assessment module performs spatiotemporal alignment and multimodal fusion. The module employs a fusion algorithm based on attention mechanisms or Bayesian inference, comprehensively considering vehicle speed, time-to-collision (TTA), lane visibility, traffic density, the impact of weather on sensor performance, and the sensor's own health status. Ultimately, it outputs a structured scenario risk level R (R0–R3), quantifying the urgency and operational difficulty of human intervention in the current driving situation.

[0038] The takeover decision module, acting as the system's "central nervous system," receives C-level and R-level signals from the capability assessment and risk assessment modules in real time and performs joint lookup reasoning based on a preset takeover strategy matrix. This matrix has an online-updable strategy space, where each (C,R) combination corresponds to a complete takeover response plan, including whether to trigger a prompt, the prompt channel combination, the intensity level, the maximum takeover buffer time, and whether degraded operation is allowed. For example, when C=2 (partial attention regression) and R=2 (medium-high risk), the system activates a flashing red border on the HUD, mid-frequency vibration on the steering wheel, and a gentle voice prompt, setting a 5-second response window; while when C=0 (no takeover capability) and R=3 (high-risk emergency), it immediately initiates a high-intensity alert across all channels and sets the MRM (Management Response Manager) standby status to "ready."

[0039] Once a takeover request is triggered, the multimodal prompt module immediately executes the command, with speakers integrated inside the left and right air vents of the dashboard playing customized voice commands; the HUD projection unit overlays dynamic warning icons in the lower area of ​​the windshield; the eccentric rotary motor inside the steering wheel generates tactile feedback with controllable frequency and amplitude; and the linear resonant actuators (LRAs) under the seat cushion and backrest output pulsed vibrations to awaken the body's senses.

[0040] Throughout the warning period, the system continuously runs a takeover validity verification mechanism in the background, cross-verifying whether the driver has truly taken over using multi-dimensional signals such as steering wheel torque changes, pedal displacement, and eye-tracking road regression. If no valid operation is confirmed within a preset time, the system seamlessly switches to Minimum Risk Maneuvering (MRM) mode. At this time, the Minimum Risk Maneuvering module takes over the vehicle's underlying control, sending deceleration commands to the ESP brake controller via the CANFD bus, sending the target steering angle to the EPS steering motor, and simultaneously activating the BCM to control the hazard warning lights. MRM dynamically plans a safe stopping path based on real-time perception data: prioritizing the right emergency lane or shoulder, but continuously monitoring lateral obstacles during lane changes; once a high-speed approaching vehicle or non-motorized vehicle is detected on the right, lateral movement is immediately suspended, maintaining the current lane and decelerating to a stop, and continuously broadcasting V2X warning information after stopping.

[0041] In one specific implementation, the infrared camera in the driver state acquisition module is fixed below the sun visor, with its optical axis pointing towards the driver's eye area. The eye-tracking sensor is embedded in the interior panel of the left A-pillar, and its field of view covers the driver's eyes and the bridge of the nose area. The steering wheel torque sensor is integrated inside the steering wheel frame and detects the distribution of grip force by both hands through a strain gauge array. The forward-facing camera in the environmental perception sensor group is installed in the center of the inside of the windshield. The millimeter-wave radar is embedded in the middle of the front bumper. The lidar is arranged on the longitudinal axis of the roof. The high-precision map receiving unit establishes a communication link with the cloud map server through the vehicle-mounted T-Box.

[0042] In practical applications of this invention, after system startup, the perception subsystem in the cockpit first establishes continuous monitoring of the driver's physiological and behavioral state. An infrared camera fixed below the sun visor aligns its optical axis with the driver's eye area. This positioning balances unobstructed vision and resistance to ambient light interference. The sun visor itself blocks direct sunlight from above, while infrared imaging effectively avoids the impact of visible light variations (such as entering / exiting tunnels or nighttime lighting) on ​​image quality. The camera outputs a high-definition near-infrared video stream at a rate of no less than 30 frames per second, used to extract key features in real time, including eyelid opening and closing frequency, pupil center position, head pitch angle, and yaw angle. Simultaneously, an eye-tracking sensor embedded in the interior panel of the left A-pillar covers the driver's eyes and the triangular area of ​​the bridge of the nose with its customized optical field of view, achieving high-precision capture of the direction of gaze. Since the A-pillar is located at the edge of the driver's natural field of vision, its placement avoids physical obstruction of the road ahead while ensuring a stable geometric relationship between the sensor lens and the human eye, maintaining high tracking accuracy even with slight head movements.

[0043] The torque sensor integrated inside the steering wheel supplements the aforementioned visual information from a tactile perspective. The sensor consists of an array of multiple micro-strain gauges distributed around the steering wheel frame, capable of distinguishing the left and right hand grip positions, the direction of applied torque (clockwise / counterclockwise), and the duration of grip. For example, when the driver lightly holds the steering wheel with only one hand and has no active steering intention, the system can identify it as a "non-intervention state"; while when both hands apply a stable counter-torque symmetrically and are accompanied by slight high-frequency vibrations (reflecting muscle tension), it is determined to be "active control".

[0044] The aforementioned internal perception data is transmitted in real time to the central computing unit via a high-speed in-vehicle communication bus (such as in-vehicle Ethernet), and is spatiotemporally fused with environmental perception information from outside the vehicle. The forward-facing camera is installed in the central area inside the windshield, adjacent to the rearview mirror base. Its installation angle is calibrated to ensure that the main optical axis is parallel to the vehicle's longitudinal axis, thereby maximizing the imaging integrity of lane lines, traffic signs, and obstacles ahead. The millimeter-wave radar is embedded in the center of the front bumper, and its metal shell is integrated with the body structure, ensuring unobstructed transmission / reception of electromagnetic beams and meeting collision safety regulations. The lidar is arranged on the longitudinal axis of the roof, usually located in front of the shark fin antenna or integrated into the spoiler. The high-position installation significantly expands the horizontal field of view and reduces the risk of being obscured by mud or snow splashed up by the vehicle in front.

[0045] In one specific implementation, the HUD projection unit in the multimodal prompt module is optically coupled to the lower area of ​​the windshield, the speakers are arranged inside the air vents on the left and right sides of the dashboard, the steering wheel vibration motor is fixed inside the steering wheel rim, the seat tactile actuator is embedded under the foam layer of the driver's seat cushion and backrest, and the minimum risk maneuver module is connected to the Electronic Stability Program (ESP) controller, the Electric Power Steering (EPS) controller, and the Body Control Module (BCM) via the CANFD bus to send braking commands, steering angle target values, and hazard warning light activation signals, and receives obstacle feedback from the environmental perception sensor group in real time during execution to dynamically adjust the trajectory.

[0046] In practical applications of this invention, when the system determines that a takeover request needs to be triggered, the multimodal prompting module immediately activates a spatially oriented, intensity-controlled, and physiologically adapted composite warning mechanism. The HUD projection unit projects key information (such as red borders, takeover countdown, and directional guidance icons) onto the lower area of ​​the windshield via optical coupling, within a range of 5°–10° below the driver's natural eye level. This avoids obstructing key areas of the road ahead while leveraging the human eye's high sensitivity to downward motion stimuli for rapid attention capture. Simultaneously, the speaker arrays located inside the air vents on both sides of the dashboard employ directional sound field technology, making the voice prompts (such as "Please take over the steering wheel immediately!") appear to come from the front of the vehicle. This aligns with human spatial perception habits of hazards, enhancing alertness without triggering a startle reflex. The steering wheel vibration motor, fixed inside the steering wheel rim, generates tactile feedback with controllable frequency (typically 30–200Hz) and amplitude through eccentric rotation or linear resonance. Its energy is directly transmitted to Pacinian corpuscles in the driver's palm skin. These receptors are extremely sensitive to high-frequency vibrations, effectively transmitting warning signals even when visual or auditory input is interrupted. Furthermore, tactile actuators (typically multiple linear resonant motors, LRAs) embedded beneath the foam layers of the driver's seat cushion and backrest can independently control vibration patterns in different areas. For example, when a right lane change is needed to avoid danger, only the right side of the seat cushion vibrates, creating a sense of spatial guidance. All these warning channels are not synchronously output with uniform intensity. Instead, the takeover decision engine dynamically modulates their combination, timing, and intensity based on the current risk level, the driver's historical response preferences, and the ambient noise level, forming a "progressive, multi-channel, context-adaptive" wake-up strategy.

[0047] If no valid human intervention (such as changes in steering wheel torque, pedal displacement, or eye-tracking return to the road) is detected within the preset takeover window, the system immediately and seamlessly switches to the Minimum Risk Maneuver (MRM) mode. At this time, the Minimum Risk Maneuver module establishes a low-latency command channel with the controllers in the chassis and body domains via the vehicle's high-speed communication network, specifically the CANFD bus. First, it sends a target deceleration or braking pressure command to the Electronic Stability Program (ESP) controller to achieve smooth and controllable longitudinal deceleration. Simultaneously, it sends the target steering angle and angular velocity curve to the Electric Power Steering (EPS) controller, driving the front wheels to execute a pre-planned lateral trajectory. In parallel, it sends a high-priority signal to the Body Control Module (BCM) to forcibly activate the hazard warning lights and activate continuous in-vehicle voice announcements (such as "The vehicle is automatically pulling over; please do not operate") to prevent accidental driver intervention.

[0048] Meanwhile, throughout the process, the MRM continuously receives real-time obstacle feedback from the environmental perception sensor array. The forward-facing camera, millimeter-wave radar, and lidar constantly update the position, speed, and trajectory prediction results of obstacles in front and to the sides. After being processed by the central domain controller, the data is fed back to the MRM trajectory planner with a delay of less than 50ms. For example, if the system originally planned to change lanes to the right and stop in the emergency lane, but the lateral lidar suddenly detects a motorcycle approaching at high speed from the blind spot at 60km / h, the MRM controller will immediately interrupt the lateral movement command, maintain the current lane centering, and only perform longitudinal deceleration. If a stationary obstacle appears in front, the braking intensity is dynamically adjusted in conjunction with TTA (Time-to-Collision) to avoid rear-end collisions caused by sudden braking.

[0049] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0050] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for safe takeover and human-machine collaboration in automatic driving control, characterized in that, Includes the following steps: Step 1: Real-time acquisition of the driver's facial images, eye movement data, head posture, gaze direction, blink frequency, and steering wheel grip status using the vehicle-mounted camera and steering wheel torque sensor. Step 2: Input the collected driver status information into the driver capability assessment module built based on the time series model, and output the driver capability level C, which is divided into four levels from C0 to C3; Step 3: Obtain current driving scene information through the environmental perception sensors on the vehicle. The scene information includes vehicle speed, relative speed of the vehicle in front, time to collision (TTA), road type, weather conditions, lane line clarity, obstacle location, traffic density, and sensor health. Step 4: Input the scenario information into the scenario risk assessment module and output the scenario risk level R, which is divided into four levels from R0 to R3; Step 5: Input the driver's capability level C and the scenario risk level R into the takeover decision module, and determine whether to trigger a takeover request, the prompting method, the prompting intensity, and the takeover timing based on the preset takeover strategy matrix; If no effective manual takeover operation is detected within the preset takeover response time, the minimum risk maneuver module is activated to perform deceleration, lane change, or pull over to the side of the road operation, and the hazard warning lights are turned on simultaneously.

2. The safe takeover and human-machine collaborative automatic driving control method according to claim 1, characterized in that, The time series model is a Long Short-Term Memory Network (LSTM) or a Convolutional Neural Network (CNN) used to model driver state information within a continuous time window to predict takeover capability within the next 1 to 3 seconds.

3. The safe takeover and human-machine collaborative automatic driving control method according to claim 1, characterized in that, The environmental perception sensors include a forward-looking camera, millimeter-wave radar, lidar, and GNSS positioning module. Data from each sensor is transmitted to the central domain controller via CAN bus or Ethernet, where it is fused and processed by the scene risk assessment module in the central domain controller.

4. The safe takeover and human-machine collaborative automatic driving control method according to claim 1, characterized in that, The takeover request can be prompted via voice broadcast, visual warnings on the dashboard or head-up display, steering wheel vibration, and seat vibration. The intensity and combination of the multimodal prompts are dynamically adjusted based on the driver's historical response behavior.

5. The automated driving control method for safe takeover and human-machine collaboration according to claim 1, characterized in that, When performing a sidewalk maneuver, the minimum risk maneuver module prioritizes the right emergency lane or shoulder and continuously monitors surrounding obstacles during the parking process. If a risk of lateral collision is detected, the lane change action is suspended and the vehicle maintains its current lane while decelerating to a stop.

6. A system for a safe takeover and human-machine collaboration automatic driving control method according to any one of claims 1-5, characterized in that, include: The driver status acquisition module includes an infrared camera installed on the top of the cockpit, a steering wheel torque sensor embedded in the steering wheel, and an eye-tracking sensor located on the inside of the A-pillar, which are used to acquire the driver's facial image, gaze direction, blink frequency, head posture, and hand grip status in real time. The driver capability assessment module is electrically connected to the driver status acquisition module and is deployed in the vehicle's central computing unit. It is used to receive and process the driver status information and output the driver capability level C. The scenario risk assessment module is electrically connected to the vehicle's environmental perception sensor group, which includes a forward-facing camera, a forward-facing millimeter-wave radar, a side-facing lidar, and a high-precision map receiving unit, and is used to fuse environmental data and output the scenario risk level R. The takeover decision module is communicatively connected to the driver capability assessment module and the scenario risk assessment module, and is used to determine the takeover strategy by looking up a table based on the capability level C and the risk level R. The multimodal prompting module, controlled by the takeover decision module, includes a speaker integrated into the dashboard, a HUD projection unit, a steering wheel vibration motor, and a tactile actuator at the bottom of the seat; The minimum risk maneuvering module communicates with the vehicle's longitudinal and lateral actuators and is used to control the braking system, steering motor, and hazard warning lights to perform preset safety actions when no effective takeover is detected.

7. The system of the safe takeover and human-machine collaborative automatic driving control method according to claim 6, characterized in that, The infrared camera in the driver state acquisition module is fixed below the sun visor, with its optical axis pointing towards the driver's eye area. The eye-tracking sensor is embedded in the interior panel of the left A-pillar, and its field of view covers the driver's eyes and the bridge of the nose area. The steering wheel torque sensor is integrated inside the steering wheel frame and detects the distribution of grip force of both hands through a strain gauge array.

8. The system of the safe takeover and human-machine collaborative automatic driving control method according to claim 6, characterized in that, The environmental perception sensor group includes a forward-facing camera installed in the center of the inside of the windshield, a millimeter-wave radar embedded in the center of the front bumper, a lidar arranged on the longitudinal axis of the roof, and a high-precision map receiving unit that establishes a communication link with the cloud map server through an onboard T-Box.

9. The system of the safe takeover and human-machine collaborative automatic driving control method according to claim 6, characterized in that, The HUD projection unit in the multimodal prompt module is optically coupled to the lower area of ​​the windshield, the speakers are arranged inside the air vents on the left and right sides of the dashboard, the steering wheel vibration motor is fixed inside the steering wheel rim, and the seat tactile actuator is embedded under the foam layer of the driver's seat cushion and backrest.

10. The system of the safe takeover and human-machine collaborative automatic driving control method according to claim 6, characterized in that, The minimum risk maneuvering module is connected to the Electronic Stability Program (ESP) controller, the Electric Power Steering (EPS) controller, and the Body Control Module (BCM) via the CANFD bus. It is used to send braking commands, target steering angle values, and hazard warning light activation signals. During execution, it receives obstacle feedback from the environmental perception sensor group in real time to dynamically adjust the trajectory.

Images