Intelligent lane departure early warning and correction method based on driver behavior prediction

Abstract

The invention belongs to a man-machine cooperative control technology in the field of intelligent driving, and particularly discloses a lane departure early warning and correction method based on driver behavior prediction. And in combination with vehicle state parameters (such as vehicle speed, yaw velocity, lane departure distance and the like), a Transformer-CNN hybrid model is adopted to predict the steering intention of the driver, so that accurate behavior judgment is realized. Multi-source data are fused through a graph neural network (GNN), the lane departure risk is dynamically evaluated, integration with an electric power steering system (EPS) is achieved, damping and torque of a steering wheel are adjusted in real time, and a vehicle is actively corrected to a safe lane. Meanwhile, the system provides multi-mode early warning such as sound, vibration and head-up display (HUD) according to the risk level, and it is ensured that a driver responds in time.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine collaborative control in the field of intelligent driving, and specifically relates to an intelligent lane departure warning and correction method based on driver behavior prediction. Background Art

[0002] With the development of intelligent driving technology, the Lane Departure Warning System (LDWS) has become an important tool for improving driving safety. Traditional LDWS usually detects lane lines through on-vehicle cameras, and uses image processing algorithms to judge whether the vehicle deviates from the lane and issues a warning when deviation occurs. However, the existing technologies have the following deficiencies:

[0003] 1. Insufficient environmental adaptability and limited robustness:

[0004] Traditional methods highly rely on visual detection of lane lines (such as cameras or lidar), but under conditions of blurred, missing lane markings or bad weather (such as rain, snow, haze, strong backlight), the detection accuracy significantly decreases. For example, algorithms based on image segmentation are vulnerable to noise interference in low-contrast scenes, resulting in false detection or missed detection; while multi-camera stitching technology, although expanding the detection range, is prone to image distortion in dynamic curve scenes, and the hardware cost is also high. Solutions for lane-line-free scenarios mostly rely on inferring the driving path of the host vehicle from the trajectory of the preceding vehicle, but when the stability of the preceding vehicle's trajectory is insufficient (such as frequent lane changes or dense traffic flow), the warning reliability is greatly reduced.

[0005] 2. Lack of dynamic prediction of intentions in the static threshold mechanism:

[0006] Existing systems mostly use fixed-time thresholds (such as the lane departure time TLC) or empirical parameters to judge risks, and cannot distinguish between driver's active operations (such as overtaking, avoidance) and unconscious deviations in real time. Models based on driving habit classification (such as fuzzy clustering, neural network) can achieve a coarse-grained division of driving styles (conservative, general, aggressive), but do not combine real-time behavior data to predict the driver's future intentions. This results in the system still being likely to trigger false alarms when the driver actively changes lanes, or failing to give timely warnings when the driver is distracted and deviates briefly, reducing the system's intelligence level and user trust.

[0007] 3. Single intervention strategy and weak closed-loop control ability:

[0008] Traditional warning systems mostly adopt passive prompts (such as sounds and vibrations), and do not form a closed loop with vehicle control systems (such as Electric Power Steering EPS and Electronic Stability Program ESP), so they cannot achieve active trajectory correction. Some studies attempt to intervene in the vehicle driving direction by adjusting steering torque or damping, but their control parameters are mostly fixed values and are not dynamically optimized in combination with real-time risk levels. For example, when the driver continuously deviates due to fatigue, fixed torque intervention may cause driving conflicts due to overcorrection, or may not be able to effectively correct the trajectory due to insufficient force.

[0009] 4. Rough human-computer interaction and insufficient multimodal collaboration:

[0010] Existing warning methods are single and lack personalized adaptation, making it difficult to meet the requirements of complex driving scenarios. For example, voice prompts are easily ignored in noisy environments, and fixed-frequency vibration feedback may cause the driver's sensitivity to gradually decrease. If visual prompts (such as Head-Up Display HUD arrows) do not dynamically adjust display parameters (such as length and refresh rate) according to the risk level, they may distract the driver's attention and instead increase the operation burden.

[0011] The core problems of existing Lane Departure Warning Systems (LDWS) can be summarized into the following four points:

[0012] (1) Strong environmental dependence: Traditional systems overly rely on visual detection of lane lines, and their performance significantly degrades in scenarios with blurred markings, bad weather, or no lane lines.

[0013] (2) Lack of intention prediction: Static thresholds and offline models cannot capture the dynamic changes of driver behavior in real time, resulting in a high false alarm rate.

[0014] (3) Insufficient closed-loop control: Passive warnings are disconnected from the vehicle execution system, and the intervention strategies are single and lack adaptive optimization.

[0015] (4) Rough interaction experience: The warning methods are single, and multimodal collaboration and personalized adaptation are not achieved. Summary of the Invention

[0016] To solve the above problems, the present invention proposes an intelligent lane departure warning and correction method based on driver behavior prediction. The method specifically includes:

[0017] Step S1, collect multimodal raw data during vehicle driving. The multimodal raw data includes driver head pose angle, driver line-of-sight coordinates, driver gesture force, steering wheel angle, vehicle speed, yaw rate, lane departure distance, lane curvature, and road surface adhesion coefficient. The multimodal raw data consists of real road data and data generated by a simulator;

[0018] Step S2, preprocess the collected multimodal raw data to obtain a multimodal feature vector;

[0019] Step S3: Generate a sample data set based on the multi-modal feature vectors, construct a Transformer-CNN model, train the Transformer-CNN model using the sample data set, and optimize the trained model to obtain a trained Transformer-CNN model;

[0020] Step S4: Input the sample data set into the trained Transformer-CNN model to obtain a driver behavior prediction result;

[0021] Step S5: Use a monorail vehicle dynamics model to adjust the risk quantification index based on the driver behavior prediction result and driver intention, and fuse the multi-modal feature vectors based on the risk quantification index using a graph neural network to generate a dynamic risk level;

[0022] Step S6: Perform lane departure warning and correction based on the dynamic risk level.

[0023] Optionally, in step S2, the process of preprocessing the collected multi-modal raw data to obtain multi-modal feature vectors specifically includes:

[0024] Use the method of multi-channel time window sampling to continuously sample the multi-modal raw data at a sampling rate of 30 Hz;

[0025] Perform data cleaning on the sampled multi-modal raw data and then perform data normalization to obtain the multi-modal feature vectors.

[0026] Optionally, in step S3, the process of obtaining a trained Transformer-CNN model specifically includes:

[0027] Input the sample data in the sample data set into a CNN model to extract spatial features and obtain a feature map of a preset size;

[0028] Input the feature map of the preset size into a Transformer to extract temporal features, and optimize the obtained temporal features to obtain a trained Transformer-CNN model.

[0029] Optionally, in step S5, the process of generating a dynamic risk level specifically includes:

[0030] Calculate the driver intention classification probability based on the driver behavior prediction result;

[0031] Input the driver intention classification probability and driver intention into a monorail vehicle dynamics model to predict the vehicle driving trajectory;

[0032] Calculate the time to collision (TTC) simply based on the predicted vehicle driving trajectory, and optimize the TTC using the driver's intention to obtain a risk quantification index, where the risk quantification index includes driver characteristics, vehicle status, and environmental information.

[0033] Calculate the adjusted time to collision value based on the driver's intention and the risk quantification index.

[0034] Normalize the adjusted time to collision value to obtain a normalized time to collision value.

[0035] Input the normalized time to collision value into a graph neural network for multi-modal fusion to generate a dynamic risk level.

[0036] Optionally, the single-track vehicle dynamics model is:

[0037]

[0038] where d is the lateral deviation distance of the vehicle from the center of the lane, is the time derivative of the lateral deviation distance of the vehicle from the center of the lane, ψ is the vehicle heading angle, r is the yaw rate, v is the vehicle speed, L is the vehicle wheelbase, is the time derivative of the vehicle heading angle, and δ is the steering wheel angle.

[0039] Optionally, the calculation process of the simple time to collision TTC is:

[0040]

[0041] Optionally, the method of optimizing the simple time to collision using the driver's intention to obtain the adjusted time to collision TTC adj includes:

[0042] TTC adj = TTC·(1 + 0.5ConsistencyFactor);

[0043] where ConsistencyFactor is the consistency factor.

[0044] Optionally, the calculation process of the consistency factor is:

[0045]

[0046] where P left , P right ∈[0,1], are the intention probabilities of the driver to change lanes to the left and to the right.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] The present invention provides an intelligent lane departure warning and correction method based on driver behavior prediction. This method realizes accurate lane departure risk assessment and effective intervention by fusing multi-modal data, predicting driver intentions, and performing adaptive control. Specifically:

[0049] (1) By fusing multi-modal data, the dependence on visual detection of lane lines is reduced, and the detection accuracy (above 90%) and robustness in complex environments are significantly improved;

[0050] (2) Based on the intention prediction of Transformer-CNN (accuracy 94.9%), active operations and unconscious deviations are accurately distinguished, the false alarm rate is reduced, and user trust is enhanced;

[0051] (3) Through the dynamic risk assessment of GNN (accuracy 94.0%) and EPS integration, active trajectory correction (correction time 1.8 seconds) is realized, and driving safety is enhanced;

[0052] (4) Multi-modal collaborative warning (response time 0.5 seconds) optimizes the human-machine interaction, reduces driver fatigue, and improves the user experience. Brief Description of the Drawings

[0053] In order to more clearly illustrate the technical solutions of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is the method flow chart of the intelligent lane departure warning and correction method based on driver behavior prediction in the embodiment of the present invention;

[0055] Figure 2 It is the schematic diagram of the hardware deployment in the embodiment of the present invention;

[0056] Figure 3 It is the schematic diagram of the Transformer-CNN model structure in the embodiment of the present invention;

[0057] Figure 4 It is the schematic diagram of the process of the risk assessment and multi-modal fusion model in the embodiment of the present invention;

[0058] Figure 5 It is the schematic diagram of the confusion matrix of the intention prediction test samples in the embodiment of the present invention;

[0059] Figure 6 It is the schematic diagram of the confusion matrix of the risk assessment and multi-modal fusion model test samples in the embodiment of the present invention. Detailed Embodiments

[0060] To make the above objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] To facilitate the understanding of the technical solution of the present invention, the following defines and explains the core technical terms involved in the specification:

[0062] Multimodal data:

[0063] Refers to heterogeneous data collected from multiple sources, including driver behavior data (head pose, steering wheel angle, line-of-sight coordinates, gesture strength) and vehicle state data (vehicle speed, yaw rate, lane departure distance, lane curvature, road surface adhesion coefficient) in the present invention.

[0064] Infrared camera data:

[0065] Infrared camera data refers to the driver's facial and eye information collected by an in-vehicle infrared camera for monitoring the driver's behavior state. The data includes raw pixel information such as the line-of-sight direction, eyelid closure degree, and head pose, with the unit being pixel value (dimensionless). In the system, the infrared camera data is used to extract driver behavior features, such as the head pose angle α and the line-of-sight trajectory (gaze point coordinates (x, y)), providing inputs for intention prediction and risk assessment.

[0066] CAN bus data:

[0067] CAN bus data refers to the vehicle state information collected through the Controller Area Network (CAN bus) of the vehicle, including real-time signals such as vehicle speed v, yaw rate r, and steering wheel angle δ, with the units being m / s, rad / s, and rad respectively. These data are generated by the vehicle's sensors (such as wheel speed sensors, gyroscopes, and steering wheel rotation sensors) and are used in the system for vehicle dynamics modeling and state feature extraction to ensure the accuracy of risk assessment.

[0068] Electric Power Steering System (EPS):

[0069] A system that assists steering through an electric motor, which is used in the present invention to achieve adaptive correction and assist the vehicle to return to a safe trajectory by adjusting the steering wheel damping (according to ΔD) and torque (according to M).

[0070] HUD (Head-Up Display):

[0071] HUD is the abbreviation of "Head-Up Display", which is usually translated as "Head-Up Display" in Chinese. In the automotive field, HUD is a technology that projects key driving information (such as vehicle speed, navigation, warning prompts, etc.) onto the windshield in front of the driver or a dedicated display screen, enabling the driver to obtain information without having to look down, thereby improving driving safety and convenience. In the present invention, it is used to display risk warnings, such as green (low risk), yellow (medium risk), and red (high risk) prompts.

[0072] Driver intention probability P:

[0073] The driver intention probability P refers to a probability vector representing the driver's behavioral intention, which includes three states: no intention, left turn, and right turn, in the form of The values of each component are between [0,1], and their sum is 1, without units. This probability is predicted by the Transformer-CNN model based on the driver's behavior and vehicle operation information, and is used as the input of the multi-modal feature map in the system, affecting the evaluation of the risk level R.

[0074] Yaw rate r:

[0075] The yaw rate r refers to the angular velocity of the vehicle around the vertical axis, reflecting the rotation rate of the vehicle. It is obtained by the gyroscope sensor in the CAN bus, with the unit of radian per second (rad / s). In the system, r is used as a vehicle state feature and input into the single-track model (state equation ) to predict the vehicle trajectory and also as a node feature of the multi-modal feature map.

[0076] Lane departure distance d:

[0077] The lane departure distance d refers to the lateral offset distance of the vehicle center relative to the lane center line. It is calculated by in-vehicle sensors (such as forward-looking cameras or lidar) and lane line detection algorithms, with the unit of meter (m). d>0 indicates a right offset, and d<0 indicates a left offset.

[0078] Steering wheel angle δ:

[0079] The steering wheel angle δ refers to the rotation angle of the steering wheel. It is obtained by the steering wheel sensor of the vehicle state sensor through the CAN bus, with a range of [-12.57, 12.57], in radians (rad), approximately [-720°, 720°]. Positive values indicate a right turn, and negative values indicate a left turn. In the system, δ is converted into the front wheel angle through the steering system as the control input of the single-track model (state equation ), directly affecting the dynamic change of the yaw rate r.

[0080] Lane curvature ρ:

[0081] The lane curvature ρ refers to the reciprocal of the lane curvature radius, which is obtained from on-vehicle sensors (such as cameras) or map data, with the unit of 1 / meter (1 / m). This indicator reflects the degree of lane bending and serves as an environmental information feature in the system, affecting the edge weight between the vehicle state and the environment, used for constructing a multi-modal feature map and indirectly influencing risk assessment.

[0082] Lane width d lane :

[0083] Lane width d lane refers to the width of the lane, with a default value of 3.5 m, which can be dynamically adjusted according to GPS and map data, and the unit is meter (m). In the system, d lane is used to calculate the adjusted time to collision TTC adj (formula ), reflecting the spatial limit of the vehicle deviating from the lane.

[0084] Road surface adhesion coefficient μ:

[0085] The road surface adhesion coefficient μ, also known as the friction coefficient, is a dimensionless physical quantity that measures the frictional force between the vehicle tires and the road surface, representing the ratio of the maximum static frictional force between the tires and the road surface to the vertical load. It can be estimated by combining a forward-looking camera with on-vehicle weather sensors to determine the road surface condition (such as wet or snowy).

[0086] Time to collision threshold TTC:

[0087] A risk quantification indicator that represents the time when the vehicle deviating from the center line of the lane may lead to a collision, with the unit of seconds, and the calculation formula is where d is the deviation distance, v is the vehicle speed, and ψ is the heading angle.

[0088] Adjusted time to collision TTC adj :

[0089] Adjusted time to collision TTC adj refers to the estimated time required for the vehicle to deviate from the lane and reach the lane boundary, and the calculation formula is If there is no collision risk, it is ∞, with the unit of seconds (s). Among them, ∈ = 0.01 m / s is a small constant to prevent division by zero. In the system, TTC adj serves as a key feature for risk assessment, affecting the risk level R and maintaining physical consistency with R through soft constraints.

[0090] Heading angle ψ:

[0091] The heading angle ψ refers to the angle between the longitudinal axis of the vehicle and the center line of the lane, reflecting the deflection direction of the vehicle, which is obtained by calculation from the dynamic model, with the unit of radian (rad). In the system, ψ is obtained through the state equation of the single-track model Calculation, used to calculate the lane departure rate Indirectly affects TTC adj .

[0092] Cornering stiffness C of the front and rear tires f , C r :

[0093] Cornering stiffness C of the front and rear tires f and C r Refers to the rigidity of the front and rear tires under lateral force, which are 60000 N / rad and 65000 N / rad respectively, with the unit of Newton per radian (N / rad). In the system, these two parameters are used in the state equation of the single-track model Affects the dynamic change of the yaw rate r

[0094] Moment of inertia I of the vehicle about the vertical axis z : Moment of inertia I of the vehicle about the vertical axis z Refers to the moment of inertia of the vehicle about the vertical axis, with a value of 3000 kg·m 2 , with the unit of kilogram square meter (kg·m 2 ). This parameter reflects the inertial characteristics of the vehicle's rotation and is used in the state equation of the single-track model in the system, affecting the calculation to ensure the physical accuracy of trajectory prediction

[0095] Risk level R: The risk level R refers to the level of lane departure risk, with a value range of {0, 1, 2, 3}, representing no risk, low risk, medium risk, and high risk respectively, without a unit. In the system, R is generated by the GNN model through feature mapping (R = argmax i P(R i )) and is kept consistent with TTC adj through soft constraints, and is used to guide correction and warning

[0096] Correction damping adjustment amount ΔD: The correction damping adjustment amount ΔD refers to the damping increment adjusted by the electric power steering system (EPS) according to the risk level R. The calculation formula is ΔD = 0.3R, with the unit of dimensionless (proportional coefficient). In the system, ΔD is used to control the intensity of the correction torque to ensure that the correction force matches the risk level and improve driving safety

[0097] Such as Figure 6As shown in the figure, the Confusion Matrix: The confusion matrix is a table used to show the correspondence between the prediction results of a classification model and the actual labels, usually represented in matrix form to show the prediction distribution of each category. It contains the following elements: True Positive (TP): The actual positive class and the predicted positive class. False Positive (FP): The actual negative class but predicted as the positive class. True Negative (TN): The actual negative class and the predicted negative class. False Negative (FN): The actual positive class but predicted as the negative class. The confusion matrix provides comprehensive information for evaluating the performance of the model and is the basis for calculating accuracy, precision, recall, and F1 score.

[0098] Accuracy: Accuracy refers to the proportion of the number of correctly predicted samples in all predictions of a classification model to the total number of samples. It is an indicator for evaluating the overall prediction effect of the model, and the calculation formula is:

[0099]

[0100] It is applicable to scenarios where the class distribution is relatively balanced, but may have limitations in the case of class imbalance.

[0101] Precision: Precision refers to the proportion of samples that actually belong to a certain category among the samples predicted by the model as that category. It reflects the accuracy of the model's prediction of the positive class, and the calculation formula is:

[0102]

[0103] High precision means a low false positive rate and is often used in scenarios where false positive predictions need to be reduced.

[0104] Recall: Recall refers to the proportion of all samples that actually belong to a certain category correctly identified by the model, also known as sensitivity or true positive rate. It reflects the coverage ability of the model for the positive class, and the calculation formula is:

[0105]

[0106] High recall means a low false negative rate and is applicable to scenarios where all positive examples need to be identified as much as possible.

[0107] F1 Score: The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate the balance performance of the model between precision and recall. The calculation formula is:

[0108]

[0109] The F1-score is particularly useful in the case of class imbalance, and a higher value indicates better model performance.

[0110] Example:

[0111] An intelligent lane departure warning and correction method based on driver behavior prediction, as Figure 1 shown, the method includes:

[0112] Step S1, collect multi-modal raw data during vehicle driving, where the multi-modal raw data includes the driver's head pose angle, the driver's line-of-sight coordinates, the driver's gesture strength, the steering wheel angle, the vehicle speed, the yaw rate, the lane departure distance, the lane curvature, and the road surface adhesion coefficient. The multi-modal raw data consists of real road data and data generated by a simulator.

[0113] The device deployment diagram is as Figure 2 shown. Camera installation location: The infrared camera is installed at the top of the A-pillar in the cockpit, and the front-view camera is installed in the upper-middle part of the windshield. Function: The infrared camera captures the driver's head pose (deflection angle α, unit: degree, range [-90°, 90°]) and the line-of-sight trajectory (coordinates (x, y), unit: pixel, based on the in-vehicle coordinate system). The front-view camera captures the lane departure distance (d, unit: m), the lane curvature (ρ, unit: 1 / m), and combines with the in-vehicle weather sensor to estimate the road surface adhesion coefficient (μ, unitless, range [0, 1]).

[0114] Preprocessing: Use the distortion correction algorithm to eliminate the deformation of the wide-angle lens, and smooth the line-of-sight trajectory noise through Kalman filtering. Pressure sensor information: Installation location: On both sides of the steering wheel grip area, sampling frequency 30Hz, range 0 - 50N. Function: Collect the gesture strength (F, unit: N). Preprocessing: Moving window mean filtering (window length 5 seconds) to eliminate instantaneous jitter.

[0115] Vehicle status data: Function: Combine with in-vehicle sensors to obtain the vehicle speed (v, unit: m / s), the yaw rate (r, unit: rad / s), and the steering wheel angle δ (unit: rad, range [-12.57, 12.57], approximately [-720°, 720°]) in real time through the CAN bus.

[0116] Vehicle status data: Function: Combine with in-vehicle sensors to obtain the vehicle speed (v, unit: m / s), the yaw rate (r, unit: rad / s), and the steering wheel angle δ (unit: rad, range [-12.57, 12.57], approximately [-720°, 720°]) in real time through the CAN bus.

[0117] Feature tensor: The dimension is corresponding to 10 feature channels (α, x, y, F, δ, v, r, d, ρ, μ).

[0118] Step S2: Preprocess the collected multi-modal raw data to obtain multi-modal feature vectors.

[0119] The process of obtaining the multi-modal feature vectors specifically includes: using the method of multi-channel time-window sampling to continuously collect the multi-modal data at a sampling rate of 30 Hz; performing data cleaning on the multi-modal data and then data normalization to obtain the multi-modal feature vectors. The input tensor is a 10-dimensional vector in the form of:

[0120] x t = [α t , x t , y t , F t , δ t , v t , r t , d t , ρ t , μ t ;

[0121] Specifically as follows:

[0122] α: Head yaw angle (degrees, range [-90°, 90°]), collected by an infrared camera.

[0123] (x, y): Line-of-sight offset coordinates (pixels, based on the in-vehicle coordinate system), collected by an infrared camera.

[0124] F: Gesture force (N), collected by a steering wheel pressure sensor.

[0125] δ: Steering wheel angle (degrees, range [-540°, 540°]), collected by a steering wheel sensor.

[0126] v: Vehicle speed (m / s, range [0, 40]), collected by the vehicle CAN bus.

[0127] r: Yaw rate (rad / s, range [-0.5, 0.5]), collected by an in-vehicle IMU sensor.

[0128] d: Lane departure distance (m, range [-1.5, 1.5]), calculated by a front-view camera.

[0129] ρ: Lane curvature (m -1 , range [0, 0.1]), calculated by a front-view camera or high-precision map.

[0130] μ: Road surface adhesion coefficient (unitless, range [0, 1]), estimated by a front-view camera and an in-vehicle weather sensor.

[0131] Data is continuously collected at a sampling rate of 30 Hz, generating 30 frames of data per second. Temporal slicing is performed using a fixed window partition. The window length is 5 seconds, corresponding to 150 frames (5 × 30 = 150). The window overlap rate is 50%, that is, the input tensor is updated every 2.5 seconds (75 frames), and the starting point of the new window is the midpoint of the previous window.

[0132] Therefore, the shape of the input tensor is 150, 10, representing 10-channel data for 150 time steps within a 5-second window.

[0133] Denoising (mean filtering), missing value filling (linear interpolation), and outlier removal (based on reasonable thresholds) are performed on the collected data. When performing sliding window slicing, ensure that the 150 frames of data in each window are complete, and the insufficient parts at the beginning and end can be processed by zero padding or truncation.

[0134] The data of each channel is normalized to eliminate dimensional differences: Head pose angle α: Normalized to [-1, 1] (corresponding to ±90°). Line-of-sight coordinates x, y: Normalized to [0, 1] (based on the display coordinate system). Gesture force F: Normalized to [0, 1] (based on the maximum expected grip force). Steering wheel angle δ: Normalized to [-1, 1] (corresponding to the maximum steering angle range). Vehicle speed v: Normalized to [0, 1] (based on the range of 0 - 40 m / s, referring to typical vehicle speeds). Yaw angular velocity r: Normalized to [-1, 1] (based on the range of ±0.5 rad / s, referring to vehicle dynamics). Lane departure distance d: Normalized to [0, 1] (based on the range of 0 - 2 m, referring to lane width). Lane curvature ρ: Normalized to [0, 1] (based on the range of 0 - 0.1 m -1 range, referring to actual road data). Road surface adhesion coefficient μ: Does not need to be normalized, with a range of [0, 1].

[0135] Step S3: Generate a sample data set based on the multi-modal feature vectors, construct a Transformer-CNN model, use the sample data set to train the Transformer-CNN model, and optimize the trained model to obtain a trained Transformer-CNN model.

[0136] The process of obtaining the trained Transformer-CNN model specifically includes: Composing simulation data from existing real road data and data generated by the simulator; Inputting the simulation data into the CNN model to extract spatial features and obtain a feature map of a preset size; Inputting the feature map of the preset size into the Transformer to extract temporal features, and optimizing the obtained temporal features to obtain a trained Transformer-CNN model.

[0137] The driver behavior prediction model is the core module of the present invention, aiming to predict the steering intention of the driver through multi-modal data. To achieve high-precision intention prediction, the present invention adopts a Transformer-CNN hybrid model. As Figure 3 shown, the specific structure of the model is as follows:

[0138] (1) CNN feature extraction layer: ① Hierarchical configuration: Input: The input dimension is 150×10 (time steps × number of features), corresponding to 10 feature channels (α, x, y, F, δ, v, r, d, ρ, μ). Among them, α: Head deflection angle (degrees); (x, y): Line-of-sight coordinates (pixels); F: Hand force (N); δ: Steering wheel angle (degrees); v: Vehicle speed (m / s); r: Yaw rate (rad / s); d: Lane departure distance (m); ρ: Lane curvature (m -1 ); μ: Road surface adhesion coefficient (dimensionless).

[0139] Layer 1: Convolution kernel size 3×1, stride 1, number of channels 32, activation function LeakyReLU, output feature map size 148×32. Layer 2: Convolution kernel size 3×1, stride 1, number of channels 64, activation function LeakyReLU, output feature map size 146×64. Layer 3: Convolution kernel size 3×1, stride 1, number of channels 128, activation function LeakyReLU, output feature map size 144×128.

[0140] ② Function: Extract local spatio-temporal features from the original multi-modal data (head pose α, line-of-sight coordinates (x, y), hand force F, steering wheel angle δ, vehicle speed v, yaw rate r, lane departure distance d, lane curvature ρ, road surface adhesion coefficient μ), and capture the instantaneous behavior patterns of the driver (such as sudden head deflection, continuous line-of-sight deviation) and the interactive effects of vehicle dynamics and environmental factors (such as the correlation between vehicle speed and lane curvature).

[0141] ③ Optimization basis: The convolution kernel size 3×1, stride 1, and number of channels 32 / 64 / 128 are determined by grid search experiments. Experiments show that the 3×1 convolution kernel is suitable for capturing local features of time series, and the gradually increasing number of channels (32 / 64 / 128) achieves a balance between computational efficiency and feature expression ability.

[0142] (2) Transformer encoder: ① Encoder structure: Input: The output dimension of the CNN is 144×128.

[0143] Embedding layer: The feature dimension is mapped from 128 to the Transformer hidden dimension of 512 through a linear layer, resulting in 144×512. Stacked in 6 layers, each layer contains the following modules: a. Multi-head self-attention mechanism: 8 attention heads, with a single-head dimension of 64 and a total hidden dimension of 512 (8×64). The attention calculation formula: where (Q, K, V) are the query, key, and value vectors respectively, and d k = 64 (the dimension of each head). b. Feed-forward network: Two fully connected layers, with a hidden layer dimension of 2048 and the activation function GELU. c. Residual connection and layer normalization: Each sub-layer (self-attention, feed-forward network) is followed by a residual connection and layer normalization (LayerNorm).

[0144] ② Positional encoding: Learnable positional embeddings are used to inject the temporal information of the time series into the model, with a dimension of 144×512, matching the input dimension.

[0145] ③ Function: As Figure 5 shown, it models the long-term temporal dependencies of the driver's behavior (such as the correlation between continuous line-of-sight deviation and steering operations), and at the same time captures the interaction effects between features (such as vehicle speed v, lane curvature ρ, road surface adhesion coefficient μ) and behavior patterns (for example, changes in steering behavior under low adhesion coefficients), enhancing the coherence and robustness of intention prediction.

[0146] ④ Optimization basis: The number of layers 6 and the number of attention heads 8 are determined through experimental tuning. Experiments show that this configuration achieves the optimal balance between the accuracy of the intention prediction model and the computational complexity. The hidden dimension of 512 and the feed-forward network dimension of 2048 refer to the standard Transformer configuration to ensure that the model has an appropriate capacity to handle the information increment brought by the new feature channels.

[0147] (3) Output layer: Global average pooling: Compresses the Transformer output (dimension 144×512) into a 512-dimensional vector. Fully connected layer: Maps the 512-dimensional vector to a 3-dimensional output. Softmax layer: Generates the intention classification probability P, divided into three categories: no intention (P no ), left turn (P left ), right turn (P right ), and satisfies P no +P left +P right = 1.

[0148] (4)Basis for parameter optimization: All hyperparameters (including convolutional kernel size, number of channels, number of Transformer layers, and number of attention heads) are optimized through cross-validation and grid search methods to ensure the best performance of the model on the training set and validation set. Construction of the training dataset: It consists of simulator data + real road data: 100,000 pieces of data are generated based on the UC-win / Road platform, and 50,000 pieces of data are collected through real vehicle tests. The driving scenarios include urban roads, highways, curves, and bad weather, covering different driver genders, ages, and driving habits. Labels are assigned to the intentions, and 3 experts jointly annotate them based on video playback, divided into "no intention", "left turn", and "right turn", with an annotation consistency of 98%.

[0149] ① Loss function: Cross-Entropy Loss, and the calculation formula is:

[0150]

[0151] where y c is the one-hot encoding of the true label, and P c is the predicted probability.

[0152] ② Optimizer: Adam optimizer, with an initial learning rate of 0.001, adopting the cosine annealing strategy (cycle of 50 epochs), and a weight decay coefficient of 1e-4.

[0153] ③ Regularization: Dropout: A Dropout layer (ratio 0.2) is set before the fully connected layer. Label smoothing: The smoothing factor is 0.1 to prevent the model from overfitting. Hardware platform: NVIDIA Jetson AGX Xavier, accelerating with FP16 precision, and the single-frame inference time ≤ 8 ms, meeting the real-time requirement (update at 30 Hz).

[0154] Step S4: Input the sample dataset into the trained Transformer-CNN model to obtain the driver behavior prediction result.

[0155] Step S5: Using a single-track vehicle dynamics model, adjust the risk quantification index based on the driver behavior prediction result and driver intention, and fuse the multi-modal feature vectors based on the risk quantification index and using a graph neural network to generate a dynamic risk level. The risk assessment module of the present invention is based on a single-track vehicle dynamics model (Single-Track Model), predicts the driving trajectory within the next 3 seconds through real-time vehicle state parameters, and combines a graph neural network (GNN) for multi-source data fusion to achieve dynamic risk assessment. The following are the specific implementation details of this module: ① Ignore the roll and pitch motions of the vehicle, only consider the lateral and yaw motions, and simplify the four wheels into two equivalent front and rear wheels. ② Assume that the tire side force is linearly related to the side slip angle (applicable to small angle approximation).

[0156] The process of obtaining the risk level specifically includes: calculating the driver intention classification probability based on the driver behavior prediction result; predicting the vehicle driving trajectory by inputting the driver intention classification probability and the multi-modal feature vector into the single-track vehicle dynamics model; calculating the time to collision simply based on the predicted vehicle driving trajectory, optimizing the time to collision simply using the driver intention classification probability to obtain a risk quantification index, and performing risk grading based on the risk quantification index to obtain a preliminary risk level R initial 。

[0157] The single-track vehicle dynamics model is as follows:

[0158]

[0159] where d is the lateral deviation distance of the vehicle from the center of the lane, is the time derivative of the lateral deviation distance of the vehicle from the center of the lane, ψ is the vehicle heading angle, r is the yaw rate, v is the vehicle speed, L is the vehicle wheelbase, is the time derivative of the vehicle heading angle.

[0160] Use the 4th-order Runge-Kutta method (RK4) to perform numerical integration on the dynamic equation, with a step size h = 0.033 seconds, predict the vehicle trajectory in the next 3 seconds (about 90 time points), and generate a position sequence y(t). The prediction formula is:

[0161] d = y(t + 3) - y lane ;

[0162] where: y lane is the position of the center line of the lane (unit: m), detected by the front view camera; y(t + 3) is the predicted position of the vehicle after 3 seconds in the future (unit: m). The prediction result generates the future position sequence of the vehicle For subsequent risk quantification. Based on the predicted trajectory, the Time to Collision (TTC) is calculated, and the predicted result of the driver's intention is introduced to adjust the TTC, generating an adjusted risk quantification index, i.e., the adjusted Time to Collision (TTC). adj , and finally the risk is classified.

[0163] The calculation process of the simple Time to Collision (TTC) is as follows:

[0164]

[0165] Optimize the simple Time to Collision using the driver's intention to obtain the adjusted Time to Collision (TTC). adj The method includes:

[0166] TTC adj = TTC · (1 + 0.5 · ConsistencyFactor);

[0167] where ConsistencyFactor is the consistency factor. The calculation process of the consistency factor is as follows:

[0168]

[0169] where P left , P right ∈[0, 1], is the intention probability of the driver to change lanes to the left or right, as Figure 4 shown.

[0170] Step S6: Based on the dynamic risk level, perform lane departure warning and correction. d is the lateral deviation distance (unit: m), d < 0 indicates deviation to the left, and d > 0 indicates deviation to the right.

[0171] When the driver's intention is consistent with the deviation direction (such as changing lanes to the right and the vehicle deviating to the right, i.e., P right > P left and d > 0), then ConsistencyFactor = |P left - P right | > 0, TTC adj > TTC, TTC increases, indicating a lower risk and reducing false intervention.

[0172] Since the value range of TTC adj may be large (for example, when v sin(ψ) is close to 0, TTC may tend to infinity), it is necessary to normalize it to ensure consistency with the value ranges of other feature nodes in the GNN.

[0173]

[0174] After normalization, The larger the value, the lower the risk. After normalization, is used as the input for the GNN. According to the unnormalized TTC adj , the risk is divided into 4 levels, and the specific rules for the initial risk level are as follows: No risk (R initial = 0): TTC adj ≥ 3 seconds; Low risk (R initial = 1): 2 ≤ TTC adj < 3 seconds; Medium risk (R initial = 2): 1 ≤ TTC adj < 2 seconds; High risk (R initial = 3): TTC adj < 1 second. Here, the initial risk level R initial is used as a reference for subsequent GNN dataset label calibration.

[0175] The graph neural network (GNN) is used to fuse multi-source data and generate comprehensive feature H. The following is the specific implementation of the GNN: (1) Node definition (V): The heterogeneous feature map contains the following four types of nodes:

[0176] ① Driver feature node: Driver intention probabilities P no , P left , P right (no lane change, left lane change, right lane change), head yaw angle α, sight offset coordinates x, y, gesture force F. ② Dynamics node: Vehicle speed v, yaw rate r, steering wheel angle δ, lane departure distance d. ③ Environment node: Lane curvature ρ, road surface adhesion coefficient μ. ④ Risk quantification node:

[0177] (2) Edge definition (E): Edges are divided into two categories: ① Physical association edge: Reflects the dynamic relationship between variables. For example, the association between vehicle speed v and yaw rate r, and the weight is initialized based on physical formulas; ② Semantic association edge: Reflects the interaction between intention and environment. For example, the relationship between intention probability P and lane curvature ρ (intention sensitivity is enhanced in curve scenarios), and the weight is trained through data-driven methods.

[0178] (3) Graph convolution operation: A 2-layer graph convolutional network (GCN) is used to update node features:

[0179] ① The first layer:

[0180]

[0181] ② The second layer:

[0182]

[0183] Among them, is the initial feature of node v; N(v) is the neighbor set of node v; W(0) , W (1) , W e (0) , are trainable parameters; the output dimension of each layer is 64, and the Dropout ratio is 0.2.

[0184] (4) Graph-level feature extraction: After performing global average pooling on all node features , generate the graph-level feature H:

[0185]

[0186] is the graph-level feature, which synthesizes intention, dynamics, environment, and risk information. Dynamic risk level mapping. Based on the feature H output by the GNN, the following formula is used to generate the risk level:

[0187] ① Calculate logits:

[0188] logits = W h ·H + b h ;

[0189] where, W h and b h are the weights and biases of the fully connected layer,

[0190] ② Softmax mapping:

[0191]

[0192] where, P(R) is a four-dimensional vector representing the probability distribution of risk levels 0 (no risk), 1 (low risk), 2 (medium risk), and 3 (high risk).

[0193] Final risk level:

[0194] R = arg max(P(R));

[0195] Select the category with the highest probability as the final risk level.

[0196] Model training and inference strategy:

[0197] (1) Dataset construction:

[0198] ① Data source: Consists of simulator data + real-road data: Generate 100,000 pieces of data based on the UC-win / Road platform, collect 50,000 pieces of data through real vehicle tests, and the driving scenarios include urban roads, highways, curves, and bad weather, covering different driver genders, ages, and driving habits.

[0199] ② Annotation method: Based on TTCadj The threshold generates R labels, which are jointly corrected and annotated by 3 experts based on video playback.

[0200] (2) Model training:

[0201] ① Loss function: Weighted cross-entropy loss, the formula is:

[0202]

[0203] where y i，c is the one-hot encoding of the true risk level, P i，c is the predicted probability, w c is the class weight, set according to the reciprocal of the class ratio.

[0204] ② Optimizer: Adam optimizer, initial learning rate 0.001.

[0205] ③ Regularization: Dropout: Set a Dropout layer (ratio 0.2) before the fully connected layer. Label smoothing: Smoothing factor 0.1 to prevent the model from overfitting.

[0206] (3) Inference and deployment: Hardware platform: NVIDIA Jetson AGX Xavier, accelerated with FP16 precision, single-frame inference time ≤ 8ms, meeting the real-time requirement (30Hz update).

[0207] The adaptive correction module adjusts the damping and steering torque of the steering wheel through the Electric Power Steering System (EPS) according to the risk level R output by the risk assessment and multi-modal fusion model and the adjusted time-to-collision threshold TTC, assisting the vehicle to return to a safe trajectory. The following is the specific implementation and execution process of the correction strategy.

[0208] Formula: ΔD = 0.3R where: ΔD: Damping adjustment amount, dimensionless ratio, used to adjust the damping coefficient of the steering wheel. R: Risk level, range [0, 3].

[0209] Source and derivation of the formula: The purpose of damping adjustment is to dynamically increase the steering resistance of the steering wheel according to the risk level R, reminding the driver and slowing down potential deviation behaviors. The selection of the coefficient 0.3 is based on experimental tests: when R = 3, a 90% increase in damping (i.e., ΔD = 0.3 × 3 = 0.9) can effectively remind the driver while avoiding excessive intervention.

[0210] Experimental verification: The experiment was carried out on a simulated driving platform and in a real vehicle environment. The test vehicle was a mid-size sedan with a wheelbase of 2.8m.

[0211] Experimental setup: Different vehicle speeds (30 km / h, 60 km / h, 90 km / h) and risk levels were set, and the subjective feedback of the driver on different damping adjustment amounts and the correction effect were recorded.

[0212] Result: The coefficient 0.3 makes the damping adjustment linearly related to the risk level, which can effectively remind the driver without affecting the comfort of normal operation.

[0213] Steering torque adjustment strategy:

[0214] Formula:

[0215] Where: M: Steering torque, unit is Newton-meter (N·m). TTC adj : Adjusted time-to-collision threshold, unit is second, and the range is [0, 3].

[0216] Source and derivation of the formula: The purpose of the steering torque is to apply an additional torque through the EPS system to assist the vehicle to return to a safe trajectory. The magnitude of the torque is inversely proportional to TTC adj : The smaller the TTC adj , the higher the risk and the greater the torque.

[0217] The selection of the coefficient 5 is based on the experimental optimization of torque comfort and correction effect: 5 N·m is the maximum torque acceptable to most drivers during emergency correction.

[0218] The experiment was carried out in real vehicle tests. The test scenarios included straight lane departure and curved lane departure, and the vehicle speed range was from 30 km / h to 90 km / h.

[0219] Result: When the coefficient is 5, the correction time is shortened to 1.8 seconds, and 90% of the testers think the torque is comfortable and the correction effect is the best.

[0220] In step S6, the intelligent lane departure warning and correction are implemented by the following modules: multimodal data acquisition module, multimodal data preprocessing module, driver behavior prediction model, risk assessment and multimodal fusion model, adaptive correction module, and multimodal warning feedback module. Each module works together to form a complete closed-loop control system to achieve accurate lane departure warning and correction. The functions, input-output relationships, and data flows of each module are described in detail below.

[0221] Multimodal data acquisition module: Function: Real-time collect and estimate driver behavior data and vehicle state data, and provide input for subsequent analysis. Input: None. Output: Data of 10 feature channels, including:

[0222] Driver characteristic data: Head deflection angle α (degrees, range [-90°, 90°]), line-of-sight coordinates (x, y) (pixels), gesture force F (N), steering wheel angle δ (degrees, range [-540°, 540°]).

[0223] Vehicle state data: Vehicle speed v (m / s, range [0, 40]), yaw rate r (rad / s, range [-0.5, 0.5]), lane departure distance d (m, range [-1.5, 1.5]).

[0224] Environmental data: Lane curvature ρ (m -1 , range [0, 0.1]), road surface adhesion coefficient μ (unitless, range [0, 1]).

[0225] Collection method: Data is collected through devices such as infrared cameras, front-view cameras, pressure sensors, vehicle CAN buses, in-vehicle sensors (such as IMU sensors, weather sensors), etc., at a sampling rate of 30 Hz.

[0226] Data flow: Data is transmitted to the multi-modal data preprocessing module.

[0227] Multi-modal data preprocessing module: Function: Clean, slice, and standardize the collected data to generate tensors suitable for model input. Input: 10-channel raw data provided by the multi-modal data collection module. Output: Standardized input tensors, with dimensions corresponding to 150 time steps (5-second window) and 10 feature channels (α, x, y, F, δ, v, r, d, ρ, μ).

[0228] Processing process: Denoising: Mean filtering is used to eliminate noise. Missing value filling: Missing data is filled in by linear interpolation. Outlier removal: Outliers are removed based on reasonable thresholds (such as vehicle speed exceeding 40 m / s). Temporal slicing: Using a 5-second (150-frame) window with an overlap rate of 50%, updated every 2.5 seconds. Standardization: Normalize the data of each channel to [0, 1] or [-1, 1]. Data flow: The standardized tensors are transmitted to the driver behavior prediction model and the risk assessment and multi-modal fusion model.

[0229] Driver behavior prediction model: Function: Use a Transformer-CNN hybrid model to predict the driver's steering intention.

[0230] Input: Preprocessed tensors Output: Intention probability distribution P = [P no , P left , P right , representing the probabilities of "no intention", "left turn", and "right turn" respectively, and satisfying P no +Pleft +P right = 1.

[0231] Processing process: CNN feature extraction: Extract local spatio-temporal features from the input tensor, with an output dimension of 144×128.

[0232] Transformer encoding: Model temporal dependencies, with an output dimension of 144×512. Output layer: Generate a 3D intent probability distribution through global average pooling and a fully connected layer, ensuring that the sum of probabilities is 1. Data flow: The intent probability distribution P is transmitted to the risk assessment and multi-modal fusion model.

[0233] Functions of the risk assessment and multi-modal fusion model: Based on the single-rail vehicle dynamics model and the graph neural network (GNN), fuse multi-source data and dynamically evaluate the lane departure risk.

[0234] Input: Preprocessed tensor Containing driver feature data, vehicle state data, and environmental data. The intent probability distribution P = [P no , P left , P right provided by the driver behavior prediction module. Output: Risk level R ∈ {0, 1, 2, 3} (no risk, low risk, medium risk, high risk). Adjusted time-to-collision threshold TTC adj (seconds). Processing process: Vehicle dynamics model: Predict the driving trajectory for the next 3 seconds based on the single-rail model and calculate the lane departure distance d(t). Time-to-collision threshold (TTC): Combine the intent probabilities P left , P right to adjust TTC adj . GNN fusion: Construct a heterogeneous graph (including intent nodes, dynamics nodes, environmental nodes, and risk quantification nodes), generate graph-level features H through 2-layer graph convolution, and generate the risk level R through Softmax mapping. Data flow: The risk level R and TTC adj are transmitted to the adaptive correction module and the multi-modal warning feedback module.

[0235] Adaptive correction module: Function: According to the risk level and the time-to-collision threshold, adjust the steering wheel damping and torque through the electric power steering system (EPS) to achieve active trajectory correction. Input: The risk level R and TTC adj , provided by the risk assessment and multi-modal fusion model. Real-time vehicle state data (such as deviation distance d, heading angle ψ), provided by the multi-modal data preprocessing module. Output: Damping adjustment amount ΔD (dimensionless ratio).

[0236] Steering torque M (N·m). Processing procedure: Damping adjustment: Dynamically adjust the steering wheel damping according to ΔD = 0.3×R. Torque adjustment: Calculate the steering torque according to M = 5 / TTC adj , and the direction is determined by the deviation direction. Closed-loop control: Execute the adjustment through EPS, monitor the deviation distance d in real time, and dynamically update ΔD and M until the vehicle returns to the safe trajectory. Data flow: The correction instruction is transmitted to the vehicle EPS system for execution, and at the same time, the vehicle state is fed back to the risk assessment and multi-modal fusion model to form a closed loop.

[0237] Multi-modal warning feedback module: Function: Provide multi-modal warnings through sound, vibration, and HUD (Head-Up Display) according to the risk level to remind the driver. Input: Risk level R, provided by the risk assessment and multi-modal fusion model. Output: Sound feedback: Volume V (dB) and frequency f (Hz). Vibration feedback: Vibration frequency f vib (Hz). HUD feedback: Color and blink frequency. Processing procedure: Parameter adjustment: Dynamically adjust the warning parameters according to R (e.g., when R = 3, volume 80 dB, frequency 1000 Hz, vibration frequency 100 Hz, HUD red blink 2 Hz). Personalized adjustment: Optimize the parameters according to the driver's preferences (such as reducing the volume and enhancing the vibration). Hardware execution: Output the warning signal through the in-vehicle audio, vibration motor, and HUD display.

[0238] Data flow: The warning signal directly acts on the driver, and at the same time, the driver's reaction (such as steering adjustment) is fed back to the system through the multi-modal data acquisition module to form a closed loop.

[0239] Data flow and closed-loop mechanism: Data flow: Multi-modal data acquisition → Data preprocessing → Behavior prediction → Risk assessment → Correction / warning → Driver / vehicle feedback → Data acquisition, forming a complete closed loop.

[0240] Closed-loop mechanism: Correction closed loop: The adaptive correction module adjusts the steering wheel through EPS, monitors the vehicle state (deviation distance d, heading angle ψ) in real time, and dynamically updates the correction parameters until the vehicle returns to the safe trajectory. Warning closed loop: The multi-modal warning feedback module outputs reminders according to the risk level, and the driver's reaction (such as steering adjustment) is captured by the data acquisition module and re-enters the behavior prediction and risk assessment processes to dynamically adjust the warning strategy.

[0241] Correction execution process: The adaptive correction module realizes the adjustment of damping and torque through the Electric Power Steering system (EPS), and assists the vehicle to return to the safe trajectory through the closed-loop control mechanism. The following is the specific execution process:

[0242] (1) Damping adjustment execution: ① Damping coefficient calculation:

[0243] Calculate the damping adjustment amount according to ΔD = 0.3R. For example, when R = 2, ΔD = 0.3×2 = 0.6, that is, the damping coefficient increases by 60%. ② EPS control signal generation: The EPS system has a built-in electronic control unit (ECU) that receives ΔD as an input parameter. The ECU adjusts the damping feedback coefficient of the EPS motor according to ΔD. The damping feedback coefficient is achieved through current control: increasing the current increases the motor resistance, thereby increasing the steering resistance of the steering wheel. ③ Physical adjustment: The EPS motor acts directly on the steering shaft through a gear mechanism, and the adjusted damping coefficient increases the steering resistance of the steering wheel. For example, when ΔD = 0.6, the EPS system increases the basic damping coefficient (such as 0.5 N·m / rad / s) by 60%, that is, adjusts it to 0.8 N·m / rad / s. ④ Driver perception: The increased steering resistance makes the driver feel greater steering difficulty, reminding them to pay attention to the risk of lane departure and at the same time slowing down the steering actions that may cause deviation.

[0244] (2) Steering torque application: ① Torque calculation: Calculate the steering torque according to For example, when TTC adj = 1 second, ② Correction direction determination: Determine the M torque application direction according to the lane departure direction (determined by the deviation distance d and the heading angle ψ provided by the risk assessment module). For example, if the vehicle deviates to the left (d > 0, ψ < 0), the torque application direction is to the right (positive torque); if it deviates to the right (d < 0, ψ > 0), a left torque (negative torque) is applied. ③ EPS execution: The ECU of the EPS system converts the torque M into a motor control signal. The motor adjusts the output torque through current, acting on the steering gear mechanism. For example, if a right torque of 3.33 N·m needs to be applied, the ECU adjusts the motor current to generate the corresponding torque, which is transmitted to the steering shaft through the gear to change the steering wheel angle. ④ Trajectory correction: The applied torque gradually adjusts the heading angle ψ of the vehicle, and the lane departure distance d approaches the lane center line y lane For example, a right torque makes the vehicle turn to the right, reducing the deviation distance d, and finally the vehicle returns to the safe trajectory.

[0245] (3) Closed-loop control: ① Real-time feedback: The EPS system monitors the actual steering angle and the applied torque of the steering wheel in real time through built-in sensors (such as torque sensors and angle sensors). At the same time, the vehicle monitors the lane departure distance d, the heading angle ψ, and the yaw rate r through on-vehicle sensors (such as cameras and radars). ② Dynamic adjustment: If the vehicle has not completely returned to the safe trajectory (for example, |d| > 0.1 m), the risk assessment and multi-modal fusion model updates R and TTC adj at a frequency of 30 Hz, and the adaptive correction module recalculates Δd and M. For example, if d decreases but is still greater than 0.1 m, TTC adjIt may increase, resulting in a decrease in M and a gradual weakening of the correction force to avoid overcorrection. ③ Safe exit: When d≈0 (for example, |d|<0.05m) and ψ≈0, it indicates that the vehicle has returned to the safe trajectory. The EPS system stops applying torque (M = 0), and the damping returns to the base value (ΔD = 0). The closed-loop control ensures a smooth correction process, avoiding sudden changes or overcorrection and enhancing the driving experience.

[0246] The multi-modal warning feedback module provides real-time warnings to the driver in various forms such as sound, vibration, and HUD (Head-Up Display) according to the risk level output by the risk assessment and multi-modal fusion model, reminding the driver to take necessary measures to avoid lane departure. The following is the specific implementation of this module.

[0247] Module function: Input: Risk level R, output by the risk assessment and multi-modal fusion model, with a range of [0, 3] (0: no risk, 1: low risk, 2: medium risk, 3: high risk). Output: Sound feedback: Volume V (unit: decibel, dB) and frequency f (unit: Hz). Vibration feedback: Vibration frequency f vib (unit: Hz). HUD feedback: Color and blinking frequency. Function: Through the multi-modal feedback mechanism, timely remind the driver of potential risks and enhance driving safety. Warning parameter adjustment rules The warning parameters are dynamically adjusted according to the risk level R, and the specific rules are as follows:

[0248] Sound feedback: Different frequency warning sounds are emitted through a two-channel audio system, and the volume and frequency increase with R (R = 1: (60dB, 500Hz), R = 3: (80dB, 1000Hz)). Tactile feedback: The steering wheel vibrates, and the intensity is proportional to R (R = 1: 50Hz, R = 3: 100Hz). Visual feedback: The dashboard displays an arrow indicating the deviation direction, and the color and blinking frequency change with R.

[0249] Personalized adjustment: Driver preference: According to the driver's sensitivity to different feedback forms, the system supports personalized adjustment. For example, if the driver is sensitive to sound, the volume V can be reduced (for example, from 80dB to 70dB), and at the same time, the vibration frequency f vib (for example, from 100Hz to 120Hz) can be increased to maintain the reminder effect.

[0250] Adjustment basis: Collect driver preference data through user tests and optimize the feedback parameters to ensure both reminder effect and comfort.

[0251] Hardware implementation: Auditory feedback: Output through the in-vehicle audio system, with volume and frequency controlled by a digital signal processor (DSP). Tactile feedback: Achieved through vibration motors built into the steering wheel or seat, with frequency controlled by a PWM (pulse width modulation) signal. Visual feedback: Presented through a head-up display (HUD), supporting RGB color switching and flicker frequency adjustment.

[0252] The embodiments described above are only descriptions of the preferred embodiments of the present invention and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An intelligent lane departure warning and correction method based on driver behavior prediction, characterized in that, The method includes: Step S1, collect multi-modal raw data during vehicle driving. The multi-modal raw data includes the driver's head pose angle, driver's line-of-sight coordinates, driver's gesture strength, steering wheel angle, vehicle speed, yaw rate, lane departure distance, lane curvature, and road surface adhesion coefficient. The multi-modal raw data consists of real road data and data generated by a simulator; Step S2, preprocess the collected multi-modal raw data to obtain multi-modal feature vectors; Step S3, generate a sample data set based on the multi-modal feature vectors, construct a Transformer-CNN model, use the sample data set to train the Transformer-CNN model and optimize the trained model to obtain a trained Transformer-CNN model; Step S4, input the sample data set into the trained Transformer-CNN model to obtain a driver behavior prediction result; Step S5, use a monorail vehicle dynamics model to adjust the risk quantification index based on the driver behavior prediction result and driver intention, and fuse the multi-modal feature vectors based on the risk quantification index using a graph neural network to generate a dynamic risk level; Step S6, perform lane departure warning and correction based on the dynamic risk level.

2. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 1, characterized in that In the step S2, the process of preprocessing the collected multi-modal raw data to obtain multi-modal feature vectors specifically includes: Use the method of multi-channel time window sampling to continuously sample the multi-modal raw data at a sampling rate of 30 Hz; Perform data cleaning on the sampled multi-modal raw data and then perform data normalization to obtain the multi-modal feature vectors.

3. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 1, characterized in that In the step S3, the process of obtaining a trained Transformer-CNN model specifically includes: Input the sample data in the sample data set into a CNN model to extract spatial features and obtain a feature map of a preset size; Input the feature map of the preset size into a Transformer to extract temporal features, and optimize the obtained temporal features to obtain a trained Transformer-CNN model.

4. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 1, characterized in that In the step S5, the process of generating a dynamic risk level specifically includes: Calculate the driver intention classification probability based on the driver behavior prediction result; Predict the vehicle driving trajectory based on the driver intention classification probability and driver intention by inputting them into a monorail vehicle dynamics model; Calculate the time to collision (TTC) based on the predicted vehicle driving trajectory, and optimize the time to collision using the driver intention to obtain a risk quantification index, where the risk quantification index includes driver characteristics, vehicle state, and environmental information; Calculate an adjusted time to collision value based on the driver intention and the risk quantification index; Normalize the adjusted time to collision value to obtain a normalized time to collision value; Input the normalized time to collision value into a graph neural network for multi-modal fusion to generate a dynamic risk level.

5. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 4, characterized in that The monorail vehicle dynamics model is: where d is the lateral deviation distance of the vehicle from the lane center, is the time derivative of the lateral deviation distance of the vehicle from the lane center, ψ is the vehicle heading angle, r is the yaw rate, v is the vehicle speed, L is the vehicle wheelbase, is the time derivative of the vehicle heading angle, and δ is the steering wheel angle.

6. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 5, characterized in that The calculation process of the time to collision (TTC) is:

7. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 6, characterized in that Optimizing the simple time to collision using the driver intention to obtain an adjusted time to collision TTC adj The method includes: TTC adj = TTC · (1 + 0.5 · ConsistencyFactor); Among them, the ConsistencyFactor is the consistency factor.

8. The intelligent lane departure warning and correction method based on driver behavior prediction according to claim 7, characterized in that The calculation process of the said consistency factor is as follows: Among them, P left , P right ∈ [0, 1], which are the intention probabilities for the driver to change lanes to the left and to the right respectively.

Images