Remote lane departure warning method and system based on remote driving

Abstract

The present application relates to the field of vehicle safety technology, and provides a remote lane deviation early warning method and system based on remote driving, which comprises the following steps: using a dual-path decoder to guide a fusion network to extract lane line feature vectors, and combining a dynamic modulation Kalman filter to generate a vehicle state parameter set; applying a dual-mode risk quantification framework to determine an early warning level through situational adaptive decision-making and report early warning information messages; performing time sequence alignment on a comprehensive control instruction set, driver state data and environmental context data, and generating control instruction messages through environmental adaptive instruction intention filter optimization; a vehicle terminal calculates a man-machine control fusion weight based on comprehensive risk and driver state, generates bottom layer control signals by fusing a second comprehensive control instruction set and safety correction instructions, and a remote driving cabin executes graded early warning and generates multi-modal feedback signals. The present application cooperates with prospective risk research and adaptive control optimization, and builds a man-machine collaborative early warning and closed-loop control mechanism for remote driving.

Description

A remote lane departure warning method and system based on remote driving Technical Field

[0001] This invention relates to the field of vehicle safety technology, and in particular to a remote lane departure warning method and system based on remote driving. Background Technology

[0002] Lane departure warning systems are a crucial component of modern automotive safety systems. They utilize onboard cameras, radar, and other sensors to monitor the vehicle's trajectory in real time, identifying unintentional deviations and issuing warnings when necessary to assist the driver in correcting their actions. In remote driving scenarios, achieving proactive risk assessment of vehicle deviation, generating adaptive warnings tailored to the specific scenario, and constructing a human-machine collaborative closed-loop control system with real-time response capabilities have become critical challenges that urgently need to be addressed in remote driving safety and intelligent closed-loop control.

[0003] Chinese patent CN115880658B, authorized by patent number CN115880658B, provides a method and system for lane departure warning in nighttime scenarios. The method includes: acquiring images of the road ahead using an onboard camera; detecting feature points of lane lines on both sides of the current driving lane using a Res2Net50-VHA network; tracking lane line feature points in previous and subsequent frames using Kalman filtering; fitting the upper and lower parts of the lane lines using a combination of quadratic polynomials and linear equations; calculating relevant parameters, including road curvature, lane centerline equation, lateral distance of the vehicle relative to the lane centerline, and yaw angle; and determining the vehicle's driving status and executing a lane departure warning strategy based on the combined parameters of the lateral distance of the vehicle to the lane centerline and the yaw angle.

[0004] However, current technology still faces many challenges. When a remotely driven vehicle is traveling on a highway straight road and is about to enter a sharp curve, traditional lane departure warning systems rely solely on the vehicle's current position to assess the risk of deviation in an instant. At this point, since the vehicle has not yet actually deviated, the system struggles to predict and quantify the potential collision risk caused by the abrupt changes in road geometry ahead. If the system fails to issue a warning in advance based on future trajectory predictions, the remote driver will lack sufficient reaction time to cope with network latency, thus failing to take preemptive deceleration and steering maneuvers. This can lead to the vehicle entering the curve at excessive speed, far exceeding safety thresholds, easily causing rollovers or off-road accidents, resulting in damage to the vehicle and cargo. Summary of the Invention

[0005] To achieve the above objectives, the present invention provides a remote lane departure warning method based on remote driving, the specific technical solution of which is as follows:

[0006] Based on the sensor dataset aligned with the spatiotemporal reference, a dual-path decoder-guided fusion network is used to extract lane line feature vectors. A dual-channel parallel Kalman filter dynamically modulated by inertial measurement unit data in the sensor dataset is constructed to perform time-optimal estimation of the lane line feature vectors and generate a set of vehicle state parameters.

[0007] A dual-mode risk quantification framework is applied to integrate and assess the instantaneous deviation risk and potential geometric risk of a vehicle, generate a fusion risk index, and use a scenario-adaptive decision-making mechanism to generate a comprehensive risk score. The comprehensive risk score and dynamic risk threshold are used to determine the warning level, and multi-level decision data are encapsulated to construct and report a warning information message.

[0008] A synchronous frame aggregation mechanism is used to align the first integrated control command set, driver status data and environmental context data in time. An environment adaptive command intent filter is applied to dynamically optimize the first integrated control command set, which is then encapsulated into a control command message, encrypted and uploaded to the cloud management platform, and sent to the target vehicle after passing multi-level serial verification.

[0009] The vehicle terminal invokes the human-machine control fusion mechanism, dynamically calculates the human-machine control fusion weight based on the comprehensive risk score and driver status data, and performs weighted fusion of the second comprehensive control command set and safety correction command to generate the underlying control signal. The remote cockpit executes graded multimodal warnings according to the warning level and generates multimodal feedback signals.

[0010] Furthermore, the method for generating the vehicle state parameter set includes:

[0011] Simultaneously acquire visual image data, lidar point cloud data, and inertial measurement unit data, and sequentially synchronize the data in the time dimension and calibrate it in the spatial dimension to generate a sensor dataset;

[0012] The visual image data and lidar point cloud data in the sensor dataset are uniformly transformed into the bird's-eye view space. The dual-path decoder guides the fusion network to perform feature extraction and reconstruction, generating a lane line probability map. Lane line instance segmentation and polynomial curve fitting are then performed on the lane line probability map to construct lane line feature vectors.

[0013] Using lane line feature vectors as observations, and fusing inertial measurement unit data, a dual-channel parallel Kalman filter is constructed to perform time-optimal estimation of the lane line feature vectors, outputting the posterior state prediction vectors of the left lane line and the right lane line respectively, and calculating the posterior state prediction vector of the center lane line. The lateral offset distance, heading angle deviation, and lane width are also calculated to construct a set of vehicle state parameters.

[0014] Furthermore, the method for generating the lane line probability map includes:

[0015] Distortion correction is performed on the visual image data, and the inverse perspective transformation matrix is ​​used to project it onto the bird's-eye view space to generate a bird's-eye view. The lidar point cloud data is projected onto the bird's-eye view space, and the data is filtered according to the reflection intensity value to select a set of high reflectivity feature points.

[0016] The bird's-eye view is input into the shared encoder in the dual-channel decoder-guided fusion network to extract multi-level visual features;

[0017] The dual-channel decoder-guided fusion network includes a shared encoder and parallel visual and geometric decoders;

[0018] The multi-level visual features are input into the visual decoder to generate a preliminary visual feature map, and the high reflectivity feature point set is input into the geometric decoder to generate a spatial attention map.

[0019] The preliminary visual feature map and the spatial attention map are fused element-wise by the guided fusion unit to generate a reconstructed fused feature map, and a lane line probability map is generated based on the reconstructed fused feature map.

[0020] Furthermore, the lateral offset distance is directly determined by the zero-point lateral offset coefficient of the posterior state prediction vector of the center lane line, and is used to characterize the lateral position of the vehicle center relative to the lane center.

[0021] The heading angle deviation is calculated by the arctangent function of the zero-point heading angle coefficient of the posterior state prediction vector of the center lane line;

[0022] The lane width is calculated by the absolute value of the difference between the zero-point lateral offset coefficient of the left lane posterior state prediction vector and the zero-point lateral offset coefficient of the right lane posterior state prediction vector.

[0023] Furthermore, the method for constructing the early warning information message includes:

[0024] Based on the vehicle state parameter set, the left lane posterior state prediction vector and the right lane posterior state prediction vector, a dual-mode risk quantification framework is used to output the expected lane departure time, future trajectory and geometric conflict potential indices in parallel. The two parallel outputs are weighted and fused to generate a fused risk index.

[0025] Based on the curvature coefficient in the fusion risk index and the posterior state prediction vector of the center lane line, combined with the weather condition coefficient and the driver's steering intention indication, the fusion risk index is dynamically modified according to the context to generate a comprehensive risk score. The graded risk threshold is dynamically generated according to the vehicle speed and the curvature coefficient. The warning level is output by judging the comprehensive risk score and the risk threshold.

[0026] Based on the warning level, and combining decision-making data, quantitative data, and evidence data, a warning information message is encapsulated, encrypted, and reported to the cloud management platform, which then distributes it to the remote cockpit.

[0027] Furthermore, the decision-making layer data includes warning levels and comprehensive risk scores; the quantification layer data includes the expected lane departure time, future trajectory and geometric conflict potential indicators, vehicle state parameter set, and vehicle speed provided by the vehicle bus; the evidence layer data includes real-time video stream data, left lane posterior state prediction vector and right lane posterior state prediction vector, vehicle future trajectory, and the location coordinates of the minimum distance in Euclidean space.

[0028] The real-time video stream data is the original footage captured by the vehicle-mounted camera.

[0029] The future trajectory of the vehicle is based on the vehicle speed provided by the vehicle bus and the vehicle yaw rate in the data of the inertial measurement unit. A rigid body vehicle kinematics model is used to continuously iterate and solve the problem within a preset prediction time to generate the coordinate points that constitute the future trajectory of the vehicle.

[0030] The minimum Euclidean distance is the minimum Euclidean distance between the coordinates of the vehicle's future trajectory and the posterior state prediction vectors of the left lane and the right lane.

[0031] Furthermore, the method for encapsulating and sending the control command message includes:

[0032] The system digitally acquires the first integrated control command set and driver status data, and receives environmental context data. It also uses a synchronous frame aggregation mechanism to generate a synchronous integrated dataset.

[0033] An adaptive instruction intent filter is applied to optimize the first integrated control instruction set in the synchronous integrated dataset in real time, generate a second integrated control instruction set, encapsulate the second integrated control instruction set and the driver status data in the synchronous integrated dataset into a control instruction message, and upload it to the cloud management platform after encryption.

[0034] The received control command message is sequentially verified by authentication, decryption, integrity and compliance. After the verification is successful, the unique identifier in the control command message is used to query and locate the telemetry transmission topic in the message queue, and the control command message is published to the telemetry transmission topic in the message queue, thus completing the distribution from the cloud management platform to the target vehicle.

[0035] Furthermore, the method for generating the underlying control signal includes:

[0036] The on-board terminal decrypts and verifies the received control command messages, and parses them to obtain the second integrated control command set.

[0037] Based on the comprehensive risk score and driver status data, the human-machine control fusion weight is dynamically calculated to determine whether the autonomous intervention of the vehicle AI is activated. If the autonomous intervention is activated, the human-machine control fusion weight is used to dynamically weight and fuse the second comprehensive control instruction set and the safety correction instruction generated by the vehicle AI autonomous intervention unit, and output the fused control instruction set.

[0038] The vehicle motion controller converts the fused control command set into underlying control signals.

[0039] Furthermore, the method for generating the multimodal feedback signal includes: executing a graded warning strategy in the remote cockpit based on the warning level in the warning information message; if the warning level is a level one warning, triggering non-intrusive visual and auditory warnings; if the warning level is a level two warning, triggering visual, auditory, and tactile warnings of preset intensity; and parsing the evidence layer data in the warning information message, overlaying and rendering it on the real-time video stream to generate a multimodal feedback signal.

[0040] A remote lane departure warning system based on remote driving, which is used to implement the aforementioned remote lane departure warning method based on remote driving, includes an on-board environment perception module, a risk decision module, a human-machine co-driving module, and a collaborative execution module;

[0041] The vehicle environment perception module: Based on the sensor dataset aligned with the spatiotemporal reference, it uses a dual-path decoder to guide the fusion network to extract lane line feature vectors, constructs a dual-channel parallel Kalman filter dynamically modulated by the inertial measurement unit data in the sensor dataset, performs time-optimal estimation of the lane line feature vectors, and generates a set of vehicle state parameters.

[0042] The risk decision module is used to apply a dual-mode risk quantification framework to fuse and evaluate the instantaneous deviation risk and potential geometric risk of the vehicle, generate a fused risk index, generate a comprehensive risk score using a scenario-adaptive decision mechanism, determine the warning level by combining the comprehensive risk score and the dynamically generated risk threshold, encapsulate the multi-level decision data, construct and report the warning information message.

[0043] The human-machine co-driving module is used to perform time-series alignment of the first integrated control instruction set, driver status data and environmental context data using a synchronous frame aggregation mechanism, dynamically optimize the first integrated control instruction set using an environment adaptive instruction intent filter, encapsulate it into a control instruction message, upload it to the cloud management platform after encryption, and send it to the target vehicle after passing multi-level serial verification.

[0044] The collaborative execution module is used by the vehicle terminal to invoke the human-machine control fusion mechanism, dynamically calculate the human-machine control fusion weight based on the comprehensive risk score and driver status data, dynamically weight and fuse the second comprehensive control instruction set and safety correction instructions to generate the underlying control signal, and the remote cockpit executes graded multimodal warnings according to the warning level to generate multimodal feedback signals.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] This invention improves upon the technical bottleneck of traditional perception methods, which are prone to both target misidentification and discontinuous state output when facing multiple challenges such as reflections, shadows, and lane line wear, by combining the guidance and fusion of visual features guided by the geometric prior of lidar with the temporal tracking of perception results based on the vehicle's own motion model.

[0047] This invention addresses the technical contradiction of traditional warning methods, which lack foresight and scenario adaptability, leading to untimely warnings when facing high-risk road conditions such as sharp bends or causing unnecessary interference during normal driving. This is achieved by combining forward-looking risk quantification of future road geometry with dynamic adaptive warning thresholds based on real-time vehicle speed, curvature, and driving context.

[0048] This invention uses the road curvature in the remote vehicle driving environment as a dynamic factor to adaptively adjust the filtering parameters of the remote driver's control commands in real time, thus solving the technical contradiction that traditional fixed-parameter filters cannot simultaneously ensure the stability of high-speed straight driving and maintain precise control response in curves.

[0049] This invention uses quantified deviation risk and driver on-loop status as dynamic factors to calculate human-machine control fusion weights in real time, and then performs continuous and smooth weighted fusion of remote control commands and vehicle autonomous safety commands accordingly. This overcomes the problems of abrupt intervention and human-machine conflict caused by traditional remote driving systems that rely on fixed thresholds for rigid control switching. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 is a flowchart illustrating the principle of a remote lane departure warning method based on remote driving according to the present invention.

[0052] Figure 2 is a functional block diagram of a remote lane departure warning system based on remote driving according to the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1:

[0055] Please refer to Figure 1. This embodiment provides a remote lane departure warning method based on remote driving, including:

[0056] S1000, based on sensor datasets aligned with a spatiotemporal reference. The dual-channel decoder guides the fusion network to extract lane line feature vectors. Construct the sensor dataset A dual-channel parallel Kalman filter dynamically modulates the data from the inertial measurement unit (I) for the lane feature vector. Perform time-optimal estimation to generate a set of vehicle state parameters. .

[0057] Specifically, this step aims to build the core of the vehicle terminal's local real-time perception and computing capabilities. By performing in-depth processing and fusion analysis of raw multi-source sensor signals, it continuously provides a high-precision, high-update-rate set of vehicle status parameters for risk assessment on the cloud platform of the remote driving system. This provides data for achieving reliable remote early warning and closed-loop control.

[0058] The original multi-source sensor signals include, but are not limited to, visual image data V collected by the vehicle-mounted camera, lidar point cloud data L collected by the lidar (LiDAR), and motion state data collected by the inertial measurement unit (IMU), i.e., inertial measurement unit data I.

[0059] Further, step S1000 includes:

[0060] Step S1100: Simultaneously acquire visual image data V, LiDAR point cloud data L, and inertial measurement unit data I, and sequentially synchronize the data in the time dimension and calibrate it in the spatial dimension to generate a sensor dataset. .

[0061] Specifically, this step aims to transform the dynamic environmental information captured by each independent sensor during vehicle operation, under its own temporal and spatial references, into a sensor dataset aligned with the spatiotemporal references. This provides a high-quality, highly reliable data source for subsequent steps.

[0062] In the specific implementation process, a heterogeneous data stream reflecting vehicle dynamics and environmental status is simultaneously acquired through an onboard integrated sensor suite. This data mainly includes: visual image data V, LiDAR point cloud data L, and inertial measurement unit (I) data I. Visual image data V is acquired by at least four onboard cameras deployed at the front, rear, left, and right of the vehicle, providing high-resolution images containing rich semantic and textural information such as lane lines and road markings. LiDAR point cloud data L is acquired by a LiDAR installed on the roof or front of the vehicle, providing three-dimensional spatial geometric information of the vehicle's surrounding environment, accurately describing road contours and obstacle positions. Inertial measurement unit (IMU) data I is acquired by the IMU, providing the vehicle's own three-axis acceleration and angular velocity information to characterize the vehicle's instantaneous motion attitude.

[0063] To achieve accurate environmental perception, the vehicle terminal needs to perform strict spatiotemporal alignment on various sensors to obtain a final standardized sensor dataset. The specific processing procedure is as follows:

[0064] Synchronization is achieved in the time dimension by using a unified high-precision master clock source, such as the Pulse Per Second (PPS) signal provided by the Global Navigation Satellite System (GNSS), to assign a globally synchronized timestamp to each frame of data collected by all sensors. This eliminates the asynchronous problems caused by differences in internal clocks and data transmission delays among sensors, ensuring strict alignment of data in the time series.

[0065] In spatial calibration, to unify the spatial observation benchmark of each sensor, a pre-defined extrinsic parameter matrix is ​​required. This extrinsic parameter matrix defines the relative pose relationships between the sensors. The extrinsic parameter matrix is ​​essentially a rotation matrix. Translation vector It is not calculated in real time during vehicle operation, but rather accurately measured once through a standard offline calibration process during the system integration phase and stored as an inherent parameter.

[0066] In particular, for the fusion of visual image data V and LiDAR point cloud data L, a rigid spatial transformation relationship needs to be established between the vehicle-mounted camera coordinate system and the LiDAR coordinate system. The specific process formula is as follows:

[0067] ;

[0068] in, This means that the lidar directly measures and outputs the position coordinates of a point in space in its own three-dimensional coordinate system; It is a 3×3 rotation matrix used to perform a rotation operation to align the position coordinates output in the LiDAR coordinate system with the coordinate axes of the vehicle camera coordinate system. This represents the new position coordinates in the vehicle camera coordinate system after rotation and translation operations, representing the position coordinates directly measured by the lidar. It is a 3×1 translation vector used to perform a second translation operation to align the origin of a point that has been aligned with the coordinate axes after a rotation operation, in order to compensate for the spatial offset between the physical installation positions of the lidar and the camera.

[0069] By performing synchronization in the aforementioned time dimension and calibration in the spatial dimension, a standardized sensor dataset is ultimately generated. All data is aligned under a unified spatiotemporal reference.

[0070] Step S1200, transfer the sensor dataset Visual image data V and LiDAR point cloud data L are uniformly transformed into a bird's-eye view space. A dual-channel decoder guides the fusion network for feature extraction and reconstruction, generating a lane line probability map. For the lane line probability map Lane line instance segmentation and polynomial curve fitting are performed sequentially to construct lane line feature vectors. .

[0071] Specifically, this step aims to utilize the sensor dataset output from step S1100. In this process, lane line feature vectors with high stability and robustness under various complex environments are extracted and constructed. This ensures that remotely driven entities can still achieve accurate lane boundary recognition under conditions of low visibility, inclement weather, or strong light interference.

[0072] In the specific implementation process, the core of this step lies in proposing a Dual-Decoder Guided Fusion Network (DDGF-Net). This network actively guides the learning and reconstruction of visual features during the feature extraction stage using the geometric prior knowledge provided by the LiDAR, fundamentally avoiding the instability problem of accuracy in traditional solutions that rely solely on visual post-processing verification. The specific process is divided into the following three stages:

[0073] The first stage involves unified spatial preprocessing of multimodal data. To achieve cross-modal feature fusion, this stage transforms both visual image data V and LiDAR point cloud data L into a unified bird's-eye-view (BEV) space. Distortion correction is applied to the visual image data V, and it is then projected onto the bird's-eye-view space using an inverse perspective mapping (IPM) matrix to generate the bird's-eye-view image. Simultaneously, the lidar point cloud data L is also projected onto the same bird's-eye view space and filtered according to the reflection intensity value to select the set of high reflectivity feature points that are physically highly likely to correspond to the lane line paint. .

[0074] The second stage involves feature extraction and reconstruction based on the dual-channel decoder-guided fusion network DDGF-Net. DDGF-Net is a customized deep learning semantic segmentation network whose structure includes a shared encoder and two parallel, functionally heterogeneous decoders.

[0075] The shared encoder will display a bird's-eye view. As input, a series of convolution and downsampling operations are used to extract multi-level visual features from low to high order in the image.

[0076] The decoder includes a visual decoder and a geometric decoder. The visual decoder, as the first branch, receives multi-level visual features from the shared encoder output. Through upsampling and convolution operations, it aims to perform semantic understanding from a purely visual perspective, generating a preliminary visual feature map that may contain visual artifacts, such as reflections and shadows that cause uncertainty. The geometric decoder, acting as the second branch, receives the set of high-reflectivity feature points selected by the lidar. As input, these sparse geometric points are transformed into a preliminary visual feature map through lightweight network layers, such as convolutions or transposed convolutions. Dense spatial attention maps of the same size The spatial attention map Locations with lidar points have high weight values, while other locations have lower weight values, forming the spatial distribution basis for physical authenticity.

[0077] Following the dual-branch decoder, the dual-decoder-guided fusion network DDGF-Net performs feature-level active guidance and reconstruction through the guided fusion unit, transforming the spatial attention map... and preliminary visual feature map Perform element-wise weighted fusion to obtain the reconstructed fusion feature map. In this reconstructed fusion feature map In this study, effective visual features supported by geometric constraints are significantly enhanced, while spurious features lacking physical basis are effectively suppressed.

[0078] Finally, based on the reconstructed and fused feature map The output layer generates a high-confidence lane line probability map. .

[0079] The third stage is lane line structure modeling. This stage involves processing the non-parametric lane line probability map output from the previous stages, which is located in pixel space. Abstracted into low-latitude lane line feature vectors containing explicit physical meaning. This is to achieve structured modeling and parametric representation of lane lines. The specific operations are as follows:

[0080] First, lane line instance segmentation is performed. To distinguish multiple independent lane lines coexisting in the image, the lane line probability map is processed. For pixels with a confidence level higher than a preset threshold, a density-based spatial clustering algorithm (DBSCAN) is used. This algorithm can adaptively divide high-density pixels that are connected in space into different clusters, with each cluster corresponding to an independent lane line instance, while effectively filtering out isolated noise pixels.

[0081] Next, parametric curve fitting is performed. After instance segmentation, weighted least squares is used to fit curves for each identified lane line pixel cluster. Considering the continuity and complexity of lane line geometry in real roads, a third-order polynomial equation is preferred for modeling lane lines. The specific formula is as follows:

[0082] ;

[0083] in, The forward distance coordinate is defined in the vehicle's own coordinate system and is an independent variable along the vehicle's direction of travel. Its value is a non-negative real number, used to specify the longitudinal distance between any point in front of the lane line and the vehicle's current reference point. The lateral offset coordinate is a dependent variable defined in the vehicle's own coordinate system and perpendicular to the vehicle's direction of travel. The positive or negative value of the value indicates whether the lane line is located to the left or right of the vehicle's center line, respectively, and is used to characterize the lateral position of the lane line at a specified forward distance coordinate x. A functional model representing the lateral position of a lane line with respect to forward distance, used to achieve a continuous description of the lane line profile; The zero-point lateral offset coefficient is physically defined as the lateral distance relative to the vehicle's longitudinal center axis when the vehicle is currently positioned, i.e., when the forward distance coordinate x=0. It is a key initial parameter used to determine whether the vehicle is currently deviating from the lane. The zero-point heading angle coefficient is physically defined as the angle between the tangent direction of the lane line at the vehicle's current position and the vehicle's direction of travel, used to reflect the relative attitude or orientation of the vehicle and the lane line. It represents the curvature coefficient, the magnitude and sign of which determine whether the road curves to the left or right, and is used to describe the degree of curvature of lane lines near vehicles. This represents the rate of change of curvature coefficient, used to describe the trend of lane curvature change along the road's extension direction.

[0084] Finally, the zero-point lateral offset coefficient Zero-point heading angle coefficient curvature coefficient and the coefficient of curvature change The lane feature vectors that together constitute a single lane line Where T represents the transpose operation.

[0085] For example, taking a scenario where a vehicle is driving on a left-turning road, the vehicle coordinate system has its origin (0,0) at the center of the rear axle of the vehicle, the positive x-axis is defined along the direction of vehicle travel, and the positive y-axis is defined along the left side of the vehicle.

[0086] Assuming lane line instance segmentation has been completed, extract the pixel cluster corresponding to the left lane line, and its coordinate set in the vehicle coordinate system, in meters, specifically represented as follows: For this coordinate set, a pre-defined weighted least squares method is invoked to solve for the optimal coefficient vector in the third-order polynomial equation, i.e., the zero-point lateral offset coefficient. Zero-point heading angle coefficient curvature coefficient and the coefficient of curvature change Among them, the zero-point lateral offset coefficient The physical meaning is that, at the vehicle's current position, i.e., forward distance coordinate x=0, the target lane line is located 1.6 meters to the left of the vehicle's longitudinal centerline; the zero-point heading angle coefficient The physical meaning of curvature coefficient is that, at the vehicle's current position, the tangent direction of the lane line makes a small angle with the vehicle's direction of travel, pointing to the left, indicating that the lane line tends to extend to the left; The physical meaning is that the lane markings in this section curve to the left; the coefficient of curvature change. The physical meaning is that the curvature of the lane line increases with the increase of the forward distance coordinate x, which is consistent with the typical road geometry characteristics of a vehicle entering a transition curve from a straight section.

[0087] Step S1300, using lane line feature vectors Using the observed values, and fusing inertial measurement unit data I, a dual-channel parallel Kalman filter is constructed to analyze the lane line feature vector. Perform time-optimal estimation and output the posterior state prediction vector of the left lane line respectively. And the right lane posterior state prediction vector Based on this, the posterior state prediction vector of the center lane line is calculated. And calculate the lateral offset distance. Heading angle deviation and lane width Construct a set of vehicle state parameters .

[0088] Specifically, this step aims to construct the lane line feature vector based on step S1200. Using the observation benchmark and integrating the inertial measurement unit data I collected in step S1100, a set of vehicle state parameters that is time-continuous, state-stable, and robust under brief interruptions is generated through time-series filtering and dynamic prediction. .

[0089] In practice, the core of this step is to construct a Kalman filter tracking model dynamically modulated by the vehicle's own motion state, i.e., inertial measurement unit data I. This model deeply couples the vehicle's own perception and external environment perception at the state tracking level, realizing the transformation from traditional reactive filtering to active state estimation with short-term predictive capabilities. Specifically, it is divided into the following two stages:

[0090] The first stage involves lane model tracking based on a dynamic model using an inertial measurement unit (IMU). To achieve continuous and stable tracking of lane geometry, this stage employs a dual-channel parallel Kalman filter to perform time-optimal estimation of the feature vectors of the left and right lane lines. Based on the lane feature vectors... Zero-point lateral offset coefficient in the vehicle's own coordinate system Positive and negative values ​​are used to label the left lane line feature vector. Negative values ​​are marked as right lane line feature vectors. Each of these inputs is then fed into its own independent Kalman filter tracking model.

[0091] Taking the left lane as an example, the construction of its Kalman filter tracking model includes the following elements:

[0092] I. Definition of State Vector. Define the state vector of the left lane line at time k. , used to describe the geometry of lane k at that moment.

[0093] II. Observation Model. At time k, the instantaneously sensed feature vector of the left lane line... As the left observation of the Kalman filter The left observation The state vector of the objectively existing left lane line The observation matrix, theoretically linearly transformed to the observation space, is then superimposed with unavoidable random measurement interference from the real environment, i.e., the observation noise vector, to form the actual observation data captured by the sensor system, i.e., the left observation. .

[0094] III. Dynamic State Transition Model. A dynamic state transition model based on vehicle kinematics is proposed, which transforms the left lane line state vector of the previous time step k-1 into a dynamic state transition model. The vehicle yaw rate is obtained from data I obtained by inertial measurement unit. Vehicle speed provided by the vehicle bus The constructed dynamic state transition matrix is ​​calculated, and the uncertainty of the model itself, namely the process noise vector at time k, is taken into account to obtain the predicted vector of the left lane prior state at the current time k. This enables real-time prediction of changes in lane line model parameters caused by vehicle movement at the next moment.

[0095] IV. Filtering and Update Mechanism. The Kalman filter tracking model performs an iterative cycle of prediction and update to update the left lane prior state prediction vector obtained from the dynamic state transition model described above. And the actual left observation value of the visual perception system Perform optimal fusion to output the optimal, time-smoothed posterior state prediction vector of the left lane at time k. .

[0096] Similarly, another Kalman filter synchronously outputs the right lane posterior state prediction vector at time k after time-series smoothing. .

[0097] The second stage involves calculating the vehicle state parameters. This is based on the stable and reliable left lane posterior state prediction vector output from the aforementioned tracking stage. And the right lane posterior state prediction vector The corresponding coefficients of the two are arithmetically averaged to obtain a posterior state prediction vector for the center lane line, representing the geometric center of the lane. And based on this, the lateral offset distance is calculated. Heading angle deviation and lane width Wherein, the lateral offset distance Predicted from the posterior state vector of the center lane line Zero-point lateral offset coefficient Directly determined, used to characterize the lateral position of the vehicle center relative to the lane center; the heading angle deviation Predicted from the posterior state vector of the center lane line Zero-point heading angle coefficient The lane width is calculated using the arctangent function. Predicted from the posterior state vector of the left lane line Zero-point lateral offset coefficient And the right lane posterior state prediction vector Zero-point lateral offset coefficient The absolute value of the difference is calculated.

[0098] Ultimately, based on the lateral offset distance Heading angle deviation and lane width To jointly construct a set of vehicle state parameters .

[0099] S2000 applies a dual-mode risk quantification framework to fuse assessments of instantaneous deviation risk and potential geometric risk of vehicles, generating a fused risk index. A situation-adaptive decision-making mechanism is used to generate a comprehensive risk score. The comprehensive risk score The system uses dynamic risk thresholds to determine and generate early warning levels. By combining multi-level decision-making data and encapsulating it, early warning information messages are constructed and reported. .

[0100] Specifically, this step aims to construct the vehicle state parameter set in step S1300. and the output left lane posterior state prediction vector and the right lane posterior state prediction vector A dual-mode risk quantification and context-adaptive decision-making framework is constructed to proactively assess lane departure risk and generate early warning messages containing visualized diagnostic information. .

[0101] Further, step S2000 includes:

[0102] Step S2100, based on the vehicle state parameter set Left lane posterior state prediction vector and the right lane posterior state prediction vector Through a dual-mode risk quantification framework, the system outputs in parallel the predicted lane departure time (TLC) and future trajectory and geometric conflict potential indicators. The two parallel outputs are weighted and fused to generate a fusion risk index. .

[0103] Specifically, this step comprehensively utilizes the vehicle state parameter set. and the left lane posterior state prediction vector representing the future road geometry. And the right lane posterior state prediction vector By constructing a dual-mode risk quantification framework, instantaneous deviation risk and potential geometric risk are assessed in parallel, and a comprehensive and forward-looking fusion risk indicator is output to characterize deviation risk. .

[0104] In its specific implementation, this dual-mode risk quantification framework calculates risks through the following three stages:

[0105] Phase 1: Instantaneous Deviation Risk Quantification. This phase assesses the imminent short-term deviation risk based on the vehicle's current state and the vehicle's state parameter set. Vehicle speed provided by the vehicle bus The calculation of the estimated lane departure time (TLC) is performed using the following formula:

[0106] ;

[0107] in, This indicates the estimated time of lane departure and is a key indicator for measuring the risk of instantaneous lane departure. The smaller the value, the higher the risk of the vehicle deviating from the lane. The unit is seconds. This represents a very small positive number set to prevent the denominator from being zero, thus ensuring the stability of the calculation.

[0108] The second phase involves quantifying potential geometric risks. To compensate for the insensitivity of Time-of-Care (TLC) prediction of lane departure to changes in road geometry ahead, this phase proactively quantifies potential risks arising from future road morphology, creatively introducing future trajectory and geometric conflict potential indicators. This indicator is calculated through the following sequential operations:

[0109] The first operation is vehicle future trajectory prediction. This prediction operation aims to deduce the most likely path of the vehicle over a period of time based on its current driving dynamics. It employs a vehicle kinematics model, treating the vehicle as a rigid body, and uses the current vehicle speed provided by the onboard bus. The vehicle yaw rate used to quantify steering intention in the inertial measurement unit data I. Within the preset prediction time, with small time steps The process involves continuous iterative solving to generate a series of coordinate points representing the future position of the vehicle, which together form the predicted future trajectory of the vehicle. .

[0110] The second operation involves future trajectory and geometric conflict potential analysis. This operation aims to quantify the proximity of the predicted trajectory to the road boundary ahead and calculate the vehicle's future trajectory. All points on the left lane, and the predicted vector from the posterior state of the left lane line. and the right lane posterior state prediction vector Minimum Euclidean distance between the described future road boundary curves The minimum distance The smaller the value, the more severe the potential conflict between the predicted trajectory and the road boundary.

[0111] The third step is the generation of potential geometric risk indicators. This step aims to transform the aforementioned conflict potential analysis into standardized future trajectories and geometric conflict potential indicators. speed As a risk amplification factor, and the minimum distance It is inversely proportional and can sensitively reflect the potential risks caused by the geometry of the future path. The specific process formula is as follows:

[0112] ;

[0113] The third stage involves the generation of integrated risk indicators. To create a comprehensive metric that uniformly represents both instantaneous lane departure risk and potential geometric risk, this stage will develop a quantitative representation of instantaneous lane departure risk—the reciprocal of the predicted lane departure time (TLC)—and indicators representing potential geometric risk, such as future trajectory and geometric conflict potential. The fusion risk index is obtained by fusion through linear weighted summation. This integration risk indicator By taking into account the combined effects of short-term vehicle dynamic response and the geometric constraints of the road ahead, it provides a more comprehensive and robust risk assessment basis for remote early warning systems than any single risk indicator.

[0114] Step S2200, based on fusion risk indicators And the posterior state prediction vector of the center lane line curvature coefficient Combined with weather condition coefficient and driver's steering intention indication Regarding the aforementioned fusion risk indicators Context-based dynamic adjustments are made to generate a comprehensive risk score. According to vehicle speed and the curvature coefficient Dynamically generate graded risk thresholds by using the comprehensive risk score. The risk threshold is used to determine and an early warning level is output. .

[0115] Specifically, this step aims to quantify the fusion risk indicators from step S2100. Combined with the posterior state prediction vector of the center lane line output in step S1300 Through a context-adaptive decision-making framework, the warning level is converted into one that matches the current driving scenario. This ensures accurate and efficient early warning.

[0116] In its implementation, this decision-making framework integrates numerical risk indicators through the following two stages. Transform into an early warning level that matches the scenario. .

[0117] Phase 1: Context-Enhanced Risk Scoring. To incorporate real-time contextual information into risk assessment, this phase integrates risk indicators. By applying dynamic gain or suppression, a more comprehensive overall risk score can be generated. The specific process formula is as follows:

[0118] ;

[0119] in, This represents the overall risk score, used to reflect the final risk measurement after correction for contextual information. and The preset weighting coefficients are constant parameters that can be calibrated offline using massive amounts of driving data. Used to adjust the magnitude of the risk gain from weather conditions. Used to define the strength of the risk mitigation effect of the steering intention; The weather condition coefficient is a factor that quantifies the severity of the current weather, and is determined in real time by the vehicle-mounted visual perception system through image analysis algorithms. It indicates the driver's turning intention and is a binary variable. It is obtained by real-time monitoring of the turn signal status on the vehicle bus network and is used to suppress annoying false alarms generated during normal lane change operations.

[0120] The second stage involves a tiered decision-making process based on dynamic risk tolerance. This involves obtaining a context-enhanced comprehensive risk score. The core innovation of this stage lies not in simply applying the aforementioned comprehensive risk score. Instead of comparing the warning to a fixed threshold, it compares it to a set of thresholds dynamically generated based on the inherent geometric risks of the road and the vehicle's driving status, thus achieving adaptive adjustment of the warning sensitivity. Specifically, this includes the following operations:

[0121] First, the dynamic warning threshold generation operation. This operation aims to calculate a set of graded warning trigger thresholds in real time based on the inherent risk level of the current driving scenario, i.e., a dynamic low-risk threshold. and a dynamic high-risk threshold The two thresholds are defined by two independent binary functions and calculated using a shared input variable: the vehicle speed provided by the onboard bus. And based on the posterior state prediction vector of the center lane line curvature coefficient The road curvature is obtained. The core characteristic of the two binary functions mentioned above is designed as a dynamic low-risk threshold. or dynamic high-risk threshold With vehicle speed The value decreases monotonically with the increase of road curvature. The value increases monotonically and decreases, and always meets the dynamic high-risk threshold. Greater than the dynamic low-risk threshold This enables the system to have higher early warning sensitivity in high-risk scenarios such as high speeds or curves.

[0122] Second, the generation of warning levels. After obtaining the threshold pairs that match the current scenario, i.e. This operation uses a comprehensive risk score. The warning level is compared with two dynamically generated risk thresholds to ultimately determine and generate a discrete warning level. If the overall risk score is... Not greater than the dynamic low-risk threshold If the risk is deemed acceptable or within a controllable range, the warning level is adjusted accordingly. A value set to 0 indicates no warning; if the overall risk score is... Greater than the dynamic low-risk threshold But less than or equal to the dynamic high-risk threshold If this is detected, it is determined that there is a warning risk that needs to be noticed by the remote driver, and the warning level is [level 1]. A value of 1 represents a Level 1 warning, i.e., a preliminary alert; based on the overall risk score... Greater than the dynamic high-risk threshold If this is detected, a high-risk situation is identified, requiring immediate attention or preparation for intervention from a remote driver; warning level [not specified]. A value of 2 indicates a Level 2 warning, which is considered high-risk.

[0123] Step S2300, based on the warning level By combining decision-making data, quantitative data, and evidence data, a warning information message is encapsulated. For the aforementioned warning information message The data is encrypted and reported to the cloud management platform, which then distributes it to the remote cockpit.

[0124] Specifically, this step aims to adjust the warning level generated in step S2200. The decision-making data, quantitative data, and evidence data generated in the preceding steps are combined and encapsulated into a structured early warning message containing visualized diagnostic information. It transmits the data securely and reliably to remote terminals, not only informing users of risk conclusions but also providing a complete chain of evidence for risk assessment, eliminating the information gap between humans and machines, and improving the efficiency and accuracy of remote intervention.

[0125] The specific implementation process is as follows:

[0126] The first phase involves the aggregation and encapsulation of multi-level early warning information. To ensure the interpretability of early warning information, data from the following three levels will be aggregated and uniformly encapsulated into a structured early warning information message. This refers to decision-making data, quantitative data, and evidence-based data.

[0127] The decision-making level data includes the final decision result, namely the warning level. The comprehensive risk score generated in step S2200 Among them, the warning level The warning interface used for direct command of the remote cockpit provides appropriate levels of sound, light, or tactile feedback; comprehensive risk score. This provides a quantifiable risk level reference for remote drivers.

[0128] The quantization layer data includes key quantification indicators supporting decision-making, namely the predicted lane departure time (TLC) output by S2100, as well as future trajectory and geometric conflict potential indicators. The set of vehicle state parameters generated in step S1300 and vehicle speed provided by the vehicle bus Among these, the estimated time of lane departure (TLC) is used to help assess the urgency of the risk, future trajectory, and geometric conflict potential indicators. Used to help identify the root cause of risk.

[0129] The evidence layer data includes core geometric evidence for visualization rendering at the remote driving end, namely real-time video stream data. The left lane posterior state prediction vector generated in step S1300 And the right lane posterior state prediction vector The future trajectory of the vehicle composed of S2100 minimum distance to Euclidean space The location coordinates of the event provide the necessary data foundation for achieving augmented reality rendering and traceability of the decision-making process in a remote cockpit. This includes real-time video stream data. It is the original footage captured by the vehicle's camera.

[0130] The second stage involves message serialization and secure transmission. The aforementioned multi-layered data is serialized according to a predefined protocol format, such as JSON, to form a structured warning information message. Subsequently, a high-strength encryption algorithm, such as AES-256, was used to encrypt the warning message. Encryption is applied to ensure the confidentiality and integrity of the data during transmission. Finally, the encrypted warning message is transmitted via the vehicle's high-bandwidth, low-latency communication module. The data is reported in real time to the cloud management platform, which then distributes it to the remote cockpit.

[0131] S3000 employs a synchronous frame aggregation mechanism for the first integrated control instruction set. Driver status data and environmental context data Timing alignment is performed, and the first integrated control instruction set is dynamically optimized using an environment-adaptive instruction intent filter. Encapsulated into control command messages The data is encrypted and uploaded to the cloud management platform, and then distributed to the target vehicle after passing multiple levels of serial verification.

[0132] Specifically, this step aims to enable the remote driver to perform physical control actions. Real-time environmental context data transmitted back from the vehicle terminal Fusion, through an environment-adaptive intelligent filtering mechanism, generates and transmits control command messages that are highly matched to the vehicle's current driving scenario. .

[0133] The physical control actions of the remote driver Real-time data is captured by a set of high-fidelity, force-feedback electromechanical control devices deployed in a remote cockpit. These devices highly simulate the driving components of a real vehicle in terms of physical form and operational feedback, including a force-feedback steering wheel for capturing steering intentions, a high-precision pedal assembly for capturing acceleration and deceleration intentions, and a physical shift lever and switch panel for capturing other discrete commands.

[0134] Further, step S3000 includes:

[0135] Step S3100: Digitally acquire the first integrated control instruction set. and driver status data and receive environment context data. A synchronous frame aggregation mechanism is used to generate a synchronous comprehensive dataset. .

[0136] Specifically, this step aims to use a high-frequency synchronous sampling mechanism to time-align the external environment perception of the remotely controlled vehicle with the real-time control intentions of the remote driver, overcoming the effects of network latency and data jitter, and generating a time-consistent, latency-free synchronous comprehensive dataset in the remote cockpit. .

[0137] The specific implementation process includes the following three stages:

[0138] The first stage involves the digital acquisition of the driver's operational intentions. This stage involves the recording of the driver's continuous physical control actions. This is converted into discrete, computer-processable digital signals. The electromechanical controller in the remote cockpit completes this in real time, generating the first integrated control command set. and driver status data Among them, the first integrated control instruction set Built-in sensors, such as high-precision encoders, Hall effect sensors, and microswitches, quantify physical actions like steering wheel rotation, pedal depressing, and lever operation into steering wheel angle data, pedal data, and switch input data, respectively. This driver state data... A hand detection module is constructed by integrating a capacitive or infrared sensor array into the steering wheel grip area to determine in real time at a high frequency whether the driver's hands have left the steering wheel.

[0139] The second phase involves asynchronous reception of the environmental context. The cockpit local processor continuously receives environmental context data relayed back from the vehicle terminal via the cloud platform through an independent communication thread. This data stream contains road geometry information and real-time vehicle pose state perceived by the vehicle image analysis system, which are derived from the perception results of step S1000 and the decision results of step S2000.

[0140] The third stage, synchronous frame aggregation, addresses the millisecond-level asynchrony among the multiple data sources during generation and transmission. This step employs a synchronous frame aggregation mechanism. Using the cockpit's local, high-frequency clock as a unified reference, a snapshot sample is taken of all input data at a fixed sampling time within each clock cycle: capturing the latest first integrated control command set at that instant. Driver status data and the received environmental context data with the timestamp closest to the current moment. And uniformly assign a precise timestamp to the current time k. To form a time-aligned synchronous comprehensive dataset .

[0141] Step S3200: Apply the environment adaptive instruction intent filter to the synchronous synthesis dataset. The first integrated control instruction set in Real-time optimization is performed to generate a second integrated control instruction set. The second integrated control instruction set and the synchronized integrated dataset Driver status data Encapsulated into control command messages The data is then encrypted and uploaded to the cloud management platform.

[0142] Specifically, this step aims to process the synchronized integrated dataset formed in step S3100. Intelligent preprocessing and optimization are performed to generate control command messages that are securely encapsulated and encrypted, reflecting the driver's core intentions and highly matched to the current driving environment of the remote vehicle. .

[0143] In the specific implementation, an environment-adaptive instruction intent filter with dynamically adjustable parameters is used to process the synchronous integrated dataset. The first integrated control instruction set in In particular, the steering wheel angle data is optimized in real time to generate a filtered second integrated control command set. .

[0144] The core feature of the environment adaptive instruction intent filter lies in its key internal filtering parameter, namely the low-pass filter strength. and input dead zone size It will be based on the synchronized comprehensive dataset Context data in Especially the curvature of the road Real-time adaptive adjustment is performed, and its adjustment logic is designed as follows:

[0145] When a vehicle travels on a straight road, i.e., the road curvature If the value is approximately 0, the system will automatically increase the dead zone size. And increase the low-pass filter strength Larger dead zone size It can effectively filter out subtle, unconscious hand tremors caused by the driver; it has a strong low-pass filtering strength. It can smooth out relatively coarse operational inputs, thus ensuring the trajectory stability of the vehicle when traveling in a straight line at high speed.

[0146] When a vehicle travels on a curve, i.e., the road curvature If the value is large, the system will automatically reduce the dead zone size. And reduce the low-pass filter strength Extremely small dead zone size and weaker low-pass filter strength This ensures that every subtle and rapid steering intention made by the driver for cornering is transmitted sensitively and without delay, thereby guaranteeing the vehicle's handling and agility when cornering.

[0147] After the above filter optimization, a safer and smoother second integrated control command set is obtained. Together with the synchronous integrated dataset Driver status data Information such as timestamps and checksums are encapsulated together into a structured control command message. The message is signed and encrypted using an asymmetric encryption algorithm, such as elliptic curve cryptography (ECC), to ensure the unforgeability of the instructions and the confidentiality of the transmission. Then, it is uploaded to the cloud management platform via a highly reliable, low-latency private network, such as 5G Ultra-Reliable Low-Latency Communication (URLLC).

[0148] Step S3300: Receive the control command message. The authentication, decryption, integrity, and compliance checks are performed sequentially. Upon successful verification, the control command message is executed accordingly. The unique identifier in the message queue is used to locate the telemetry transmission topic and send the control command message. The message is published to the telemetry transmission topic in the message queue, completing the distribution from the cloud management platform to the target vehicle.

[0149] Specifically, this step aims to utilize a highly reliable and secure cloud management platform as a trusted data exchange hub between the remote cockpit and the remote vehicle to implement the control command messages encapsulated in step S3200. Precise routing and secure distribution.

[0150] In practice, this step may include the following three stages:

[0151] The first phase involves command reception, decryption, and authentication. The access endpoint of the cloud management platform is responsible for receiving encrypted control command messages uploaded from the remote control cockpit. Based on Transport Layer Security (TLS) and a digital certificate pre-shared with the cockpit, the platform performs rigorous authentication of the command source. After successful authentication, the platform uses the corresponding key to decrypt the message and restore the internal command data.

[0152] The second phase involves verifying the integrity and compliance of the instructions. After decryption, the platform performs multiple verifications on the instruction data. First, it verifies whether the data has been tampered with during transmission using the digital signature or hash checksum contained in the message, ensuring its integrity. Second, it performs compliance checks based on preset business logic and security policies, such as verifying whether the instruction's timestamp is within the allowed delay window to prevent replay attacks or execution of expired instructions.

[0153] The third stage involves target routing and delivery based on the Message Queuing Telemetry Transport (MQTT) protocol. After the command passes integrity and compliance verification, the platform parses the unique identifier of the target vehicle from the command data. Then, it queries the target vehicle's current network session status and its subscribed MQTT topics in the internally maintained device registration and session management service. Finally, the platform publishes the verified and compliant control command as a message to the vehicle's dedicated MQTT topic.

[0154] Ultimately, the vehicle terminal that has established a long connection with the cloud platform and subscribed to the topic will receive the control command in real time through the MQTT agent, thereby completing the closed loop of command transmission from the remote cockpit to the vehicle.

[0155] The S4000 in-vehicle terminal utilizes a human-machine interface fusion mechanism based on a comprehensive risk score. and driver status data Dynamically calculate the human-machine control fusion weights for the second integrated control instruction set. and safety correction instructions Weighted fusion is performed to generate underlying control signals, which are then processed by the remote cockpit based on the warning level. Perform hierarchical multimodal early warning and generate multimodal feedback signals.

[0156] Specifically, this step aims to receive the warning information message generated in step S2300. and the control command message generated in step S3200 By performing human-machine collaborative control at the vehicle end and multimodal feedback at the cockpit end, low-level control signals for vehicle actuators and multimodal feedback signals for remote driver operation perception are generated.

[0157] Further, step S4000 includes:

[0158] Step S4100: The on-board terminal processes the received control command message. Decryption and verification are performed to obtain the second integrated control instruction set. The second integrated control instruction set The human-machine control fusion mechanism input to step S4300 generates a fusion control instruction set. The vehicle motion controller will then transmit the fused control command set. It is converted into low-level control signals.

[0159] Specifically, this step serves as the final entry and exit point for the vehicle's physical actions, aiming to construct a real-time control loop for the onboard terminal and parse control command messages received from the cloud. The human-machine control fusion weights are called in step S4300 to generate a fusion control instruction set. It then transforms the signals into underlying control signals that can be recognized by the underlying actuators, thereby achieving the final execution of the entire perception-decision-control closed loop.

[0160] In practice, this step is executed cyclically at a fixed high frequency on the onboard main processing unit. Within each control cycle, this step is performed sequentially in the following order to ensure a deterministic delay in the reception of the vehicle's response from the command:

[0161] First, the reception and parsing of remote commands. At the beginning of the control cycle, the onboard communication unit sends the latest received control command message... The message is transmitted to the onboard main processing unit. This processing unit processes the control command message. Decryption and integrity verification are performed, and the encapsulated second integrated control command set, representing the remote driver's current control intention within that period, is parsed out. .

[0162] Second, the invocation of human-machine collaborative decision-making. The parsed second integrated control instruction set. It will not be executed directly, but will be immediately passed to the human-machine control fusion mechanism in step S4300 as one of the core inputs. This mechanism will quickly calculate and output the fused control instruction set based on the human-machine control fusion weights within the current control cycle. .

[0163] Third, the encoding and issuance of the final command. At the end of the control cycle, the vehicle's motion controller receives the fused control command set output from step S4300. The motion controller is responsible for integrating the control command set. The high-level instructions defined in the code are converted in real time into one or more sets of low-level control signals that can be recognized and executed by the low-level actuators, and then sent down via the vehicle CAN bus.

[0164] Step S4200: In the remote cockpit, based on the warning information message... The warning level in Implement a tiered early warning strategy; if the aforementioned early warning level... When the warning level is Level 1, non-intrusive visual and auditory warnings are triggered. When a level-two warning is issued, high-intensity visual, auditory, and tactile warnings are triggered, and the warning information message is parsed. The evidence layer data is overlaid and rendered on the real-time video stream to generate multimodal feedback signals.

[0165] Specifically, this step aims to transform the abstract early warning information message... It translates in real time at the remote cockpit into multimodal feedback signals that can be intuitively and unambiguously understood by human operators, enabling a visual explanation and perceptible response to the causes of risks, thereby guiding the driver to take quick and correct intervention actions.

[0166] In practice, the remote cockpit receives warning messages from the cloud. Then, the data is decrypted and verified to ensure communication security and data authenticity. Subsequently, the decision-making data is parsed, and core parameters, including the warning level, are extracted. and comprehensive risk score The system determines the alert level based on the warning level. Trigger tiered alert strategies that match the risk level to create a hierarchical warning experience, as follows:

[0167] When the warning level When the value is 1, it is a Level 1 warning, which is a warning that alerts the driver to potential risks in a non-intrusive manner, triggering visual and auditory warnings. The visual warning is a red "breathing light" effect at a low frequency on the edge of the main display screen in the cockpit; the auditory warning is a low-frequency, medium-volume alert beep from the speaker.

[0168] When the warning level When the value is 2, it is a Level 2 warning, i.e., a high-risk warning. The system strongly and undeniably warns of approaching danger, triggering visual, auditory, and tactile warnings. The visual warning is indicated by the red "breathing light" at the edge of the display screen changing to a high-frequency, high-brightness flashing effect; the auditory warning is indicated by the speaker emitting a faster-frequency, louder emergency alarm sound; and the tactile warning is indicated by activating the vibration motor integrated into the force feedback steering wheel, generating a value related to the overall risk score. Positively correlated, rhythmic, and strong vibrations.

[0169] The cockpit's graphics rendering engine is activated immediately only when a Level 2 warning is triggered, to ensure the interpretability of the risk. This engine parses the warning message. The evidence layer data is then overlaid and rendered on the real-time video stream:

[0170] First, visualization of road geometry. Based on the posterior state prediction vector of the left lane line. And the right lane posterior state prediction vector On both sides of the road in the video footage, two bright virtual lane lines are rendered to help drivers intuitively grasp the true geometry of the road ahead.

[0171] Second, visualization of hazardous trajectories: based on the vehicle's future trajectory. A striking virtual trajectory path is projected forward from the vehicle's position in the video frame, indicating the vehicle's future direction of travel under the current control state.

[0172] Third, visualization of conflict points: based on the minimum distance in Euclidean space. The coordinate data of the location where the collision is about to occur is used to render a highlighted and flashing warning marker at the location where the virtual lane line and virtual trajectory path are about to collide, so as to visually indicate the potential collision risk area.

[0173] Step S4300, based on the comprehensive risk score and driver status data The system dynamically calculates the human-machine control fusion weights to determine whether to activate the autonomous intervention of the onboard AI. If the autonomous intervention is activated, the human-machine control fusion weights are used to apply the second integrated control command set. and safety correction commands generated by the onboard AI autonomous intervention unit Perform dynamic weighted fusion and output a fused control instruction set. .

[0174] Specifically, this step aims to assess the overall risk score generated in step S2200. and the driver status data generated in step S3100 The system intelligently calculates the human-machine control fusion weights and, based on these weights, extracts the remote driver's second integrated control command set from step S4100. Safety correction commands generated by the onboard AI autonomous intervention unit Dynamic weighted fusion is performed to output a fused control command set that maximizes respect for the driver's intentions while ensuring a safety baseline. .

[0175] In the specific implementation process, unlike the traditional passive takeover logic based on fixed time delay triggering, this step adopts a continuous and dynamic human-machine control fusion mechanism, which calculates the human-machine control fusion weight in real time. Its value ranges dynamically within the interval [0, 1], and is determined based on the comprehensive risk score through a preset mapping function. and driver status data Calculation. The mapping function is designed so that the value of the human-machine control fusion weight will vary with the comprehensive risk score. The increase is smooth and gradual; when the driver's state data When the driver's hands are off the steering wheel, the human-machine interface fusion weight is assigned a higher base value or is set to 1. Conversely, when the risk is low and the driver's hands are on the steering wheel, the human-machine interface fusion weight remains at 0.

[0176] The autonomous intervention unit of the vehicle AI is activated if and only if the value of the human-machine control fusion weight is greater than 0. This unit is based on the posterior state prediction vector of the center lane line output in step S1300. Through path-following algorithms such as Lane Keeping Assist (LKA) or Model Predictive Control (MPC), it autonomously calculates safe correction commands to guide the vehicle safely back to the center of the lane or the optimal trajectory. .

[0177] The final fusion control command set issued to step S4100 It is the second integrated control command set for remote drivers. Safety correction commands from in-vehicle AI The result is a dynamically weighted fusion based on the human-machine control fusion weights.

[0178] Once the vehicle's condition returns to safety, the overall risk score... Once the weight of the human-machine control fusion falls below the dynamic warning threshold, it will automatically and smoothly drop back to 0. At this point, the autonomous intervention unit of the vehicle AI will no longer take control, and control will be seamlessly returned to the remote driver. The entire intervention and exit process is seamless and without any abrupt switching.

[0179] Example 2:

[0180] Based on Embodiment 1, this embodiment provides a remote lane departure warning system based on remote driving, as shown in Figure 2, including an in-vehicle environment perception module, a risk decision module, a human-machine co-driving module, and a collaborative execution module.

[0181] The vehicle-mounted environmental perception module is based on a sensor dataset aligned with a spatiotemporal reference. The dual-channel decoder guides the fusion network to extract lane line feature vectors. Construct the sensor dataset A dual-channel parallel Kalman filter dynamically modulates the data from the inertial measurement unit (I) for the lane feature vector. Perform time-optimal estimation to generate a set of vehicle state parameters. ;

[0182] The risk decision-making module is used to apply a dual-mode risk quantification framework to fuse and assess the instantaneous deviation risk and potential geometric risk of a vehicle, and generate a fused risk index. A situation-adaptive decision-making mechanism is used to generate a comprehensive risk score. The comprehensive risk score The warning level is generated by judging the risk thresholds dynamically. By combining multi-level decision-making data and encapsulating it, early warning information messages are constructed and reported. ;

[0183] The human-machine co-driving module is used to employ a synchronous frame aggregation mechanism to process the first integrated control instruction set. Driver status data and environmental context data Timing alignment is performed, and the first integrated control instruction set is dynamically optimized using an environment-adaptive instruction intent filter. Encapsulated into control command messages The encrypted data is uploaded to the cloud management platform and then distributed to the target vehicle after passing multiple levels of serial verification.

[0184] The collaborative execution module is used by the vehicle terminal to invoke the human-machine control fusion mechanism, based on a comprehensive risk score. and driver status data Dynamically calculate the human-machine control fusion weights for the second integrated control instruction set. and safety correction instructions Dynamic weighted fusion is performed to generate underlying control signals, which are then processed by the remote cockpit based on the warning level. Perform hierarchical multimodal early warning and generate multimodal feedback signals.

[0185] The parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0186] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote lane departure warning method based on remote driving, characterized in that, include: Based on a sensor dataset aligned with a spatiotemporal reference, a dual-path decoder-guided fusion network is used to extract lane line feature vectors. A dual-channel parallel Kalman filter, dynamically modulated by inertial measurement unit data from the sensor dataset, is constructed to perform time-optimal estimation of the lane line feature vectors, generating a set of vehicle state parameters. A dual-mode risk quantification framework is applied to fuse and evaluate the instantaneous deviation risk and potential geometric risk of the vehicle, generating a fused risk index. A context-adaptive decision-making mechanism is used to generate a comprehensive risk score. The comprehensive risk score and dynamic risk threshold are used to determine the warning level. The warning information message is constructed and reported by combining multi-level decision data. A synchronous frame aggregation mechanism is used to align the first integrated control command set, driver status data and environmental context data in time. An environment adaptive command intent filter is applied to dynamically optimize the first integrated control command set, which is then encapsulated into a control command message, encrypted and uploaded to the cloud management platform, and sent to the target vehicle after passing multi-level serial verification. The vehicle terminal invokes the human-machine control fusion mechanism, dynamically calculates the human-machine control fusion weight based on the comprehensive risk score and driver status data, and performs weighted fusion of the second comprehensive control command set and safety correction command to generate the underlying control signal. The remote cockpit executes graded multimodal warnings according to the warning level and generates multimodal feedback signals. The method for generating the underlying control signal includes: decrypting and verifying the received control command message at the vehicle terminal, and parsing it to obtain the second integrated control command set; Based on the comprehensive risk score and driver status data, the human-machine control fusion weight is dynamically calculated to determine whether to activate the autonomous intervention of the vehicle AI. If the autonomous intervention is activated, the human-machine control fusion weight is used to dynamically weight and fuse the second comprehensive control instruction set and the safety correction instruction generated by the vehicle AI autonomous intervention unit to output a fused control instruction set. The vehicle motion controller then converts the fused control instruction set into underlying control signals.

2. The remote lane departure warning method based on remote driving according to claim 1, characterized in that, The method for generating the vehicle state parameter set includes: synchronously acquiring visual image data, LiDAR point cloud data, and inertial measurement unit (IMU) data, and sequentially synchronizing the data in the time dimension and calibrating it in the spatial dimension to generate a sensor dataset; uniformly transforming the visual image data and LiDAR point cloud data in the sensor dataset to a bird's-eye view space, using a dual-channel decoder-guided fusion network for guided reconstruction of feature extraction to generate a lane line probability map; sequentially performing lane line instance segmentation and polynomial curve fitting on the lane line probability map to construct lane line feature vectors; using the lane line feature vectors as observations, fusing IMU data, constructing a dual-channel parallel Kalman filter to perform time-optimal estimation of the lane line feature vectors, outputting the left lane line posterior state prediction vector and the right lane line posterior state prediction vector respectively, calculating the center lane line posterior state prediction vector, and solving for lateral offset distance, heading angle deviation, and lane width to construct the vehicle state parameter set.

3. The remote lane departure warning method based on remote driving according to claim 2, characterized in that, The method for generating the lane line probability map includes: performing distortion correction on visual image data and projecting it onto a bird's-eye view space using an inverse perspective transformation matrix to generate a bird's-eye view; projecting LiDAR point cloud data onto the bird's-eye view space and filtering it based on reflection intensity values ​​to select a set of high reflectivity feature points; inputting the bird's-eye view into a shared encoder in a dual-channel decoder-guided fusion network to extract multi-level visual features; the dual-channel decoder-guided fusion network includes a shared encoder and parallel visual and geometric decoders; inputting the multi-level visual features into the visual decoder to generate a preliminary visual feature map, and simultaneously inputting the set of high reflectivity feature points into the geometric decoder to generate a spatial attention map; and using a guided fusion unit to perform element-wise weighted fusion of the preliminary visual feature map and the spatial attention map to generate a reconstructed fusion feature map, and generating a lane line probability map based on the reconstructed fusion feature map.

4. The remote lane departure warning method based on remote driving according to claim 2, characterized in that, The lateral offset distance is directly determined by the zero-point lateral offset coefficient of the posterior state prediction vector of the center lane line, and is used to characterize the lateral position of the vehicle center relative to the lane center; the heading angle deviation is calculated by the arctangent function of the zero-point heading angle coefficient of the posterior state prediction vector of the center lane line; the lane width is calculated by the absolute value of the difference between the zero-point lateral offset coefficient of the posterior state prediction vector of the left lane line and the zero-point lateral offset coefficient of the posterior state prediction vector of the right lane line.

5. A remote lane departure warning method based on remote driving according to claim 1, characterized in that, The method for constructing the warning information message includes: based on the vehicle state parameter set, the left lane line posterior state prediction vector, and the right lane line posterior state prediction vector, using a dual-mode risk quantification framework, parallel outputting the expected lane departure time, future trajectory, and geometric conflict potential indicators; weightedly fusing the two parallel outputs to generate a fused risk indicator; based on the fused risk indicator and the curvature coefficient in the center lane line posterior state prediction vector, combined with weather condition coefficients and driver steering intention indications, dynamically correcting the fused risk indicator to generate a comprehensive risk score; dynamically generating graded risk thresholds based on vehicle speed and the curvature coefficient; determining and outputting a warning level by judging the comprehensive risk score and the risk thresholds; based on the warning level, combining decision-level data, quantification-level data, and evidence-level data, encapsulating the warning information message into a warning information message; encrypting the warning information message; reporting it to the cloud management platform; and having the cloud management platform distribute it to the remote cockpit.

6. The remote lane departure warning method based on remote driving according to claim 5, characterized in that, The decision-making layer data includes warning levels and comprehensive risk scores; the quantification layer data includes predicted lane departure time, future trajectory and geometric conflict potential indicators, vehicle state parameter set, and vehicle speed provided by the vehicle bus; the evidence layer data includes real-time video stream data, left lane posterior state prediction vectors and right lane posterior state prediction vectors, vehicle future trajectory, and the coordinates of the location where the minimum distance in Euclidean space occurs; the real-time video stream data is the original scene captured by the vehicle camera; the vehicle future trajectory is based on the vehicle speed provided by the vehicle bus and the vehicle yaw rate in the inertial measurement unit data, using a rigid body vehicle kinematics model, and continuously iteratively solved within a preset prediction time to generate the coordinate points constituting the vehicle future trajectory; the minimum distance in Euclidean space is the minimum distance in Euclidean space between the coordinate points on the vehicle future trajectory and the left lane posterior state prediction vectors and right lane posterior state prediction vectors.

7. The remote lane departure warning method based on remote driving according to claim 1, characterized in that, The method for encapsulating and sending control command messages includes: digitally acquiring a first integrated control command set and driver status data, receiving environmental context data, and generating a synchronous integrated dataset using a synchronous frame aggregation mechanism; applying an environment-adaptive command intent filter to optimize the first integrated control command set in the synchronous integrated dataset in real time to generate a second integrated control command set; encapsulating the second integrated control command set and the driver status data in the synchronous integrated dataset into a control command message, and uploading it to the cloud management platform after encryption; sequentially performing identity verification, decryption, integrity and compliance verification on the received control command message; after successful verification, querying and locating the message queue telemetry transmission topic based on the unique identifier in the control command message, and publishing the control command message to the message queue telemetry transmission topic to complete the sending from the cloud management platform to the target vehicle.

8. The remote lane departure warning method based on remote driving according to claim 1, characterized in that, The method for generating the multimodal feedback signal includes: executing a graded warning strategy in the remote cockpit based on the warning level in the warning information message; if the warning level is a level one warning, triggering non-intrusive visual and auditory warnings; if the warning level is a level two warning, triggering visual, auditory, and tactile warnings of preset intensity; parsing the evidence layer data in the warning information message; overlaying and rendering the data on the real-time video stream to generate a multimodal feedback signal.

9. A remote lane departure warning system based on remote driving, used to implement the remote lane departure warning method based on remote driving as described in any one of claims 1-8, characterized in that, The system includes an in-vehicle environment perception module, a risk decision-making module, a human-machine co-driving module, and a collaborative execution module. The in-vehicle environment perception module, based on a sensor dataset aligned to a spatiotemporal reference, uses a dual-path decoder-guided fusion network to extract lane line feature vectors. It then constructs a dual-channel parallel Kalman filter dynamically modulated by inertial measurement unit data from the sensor dataset to perform time-optimal estimation of the lane line feature vectors, generating a set of vehicle state parameters. The risk decision-making module applies a dual-mode risk quantification framework to fuse and evaluate the vehicle's instantaneous deviation risk and potential geometric risk, generating a fused risk index. It uses a context-adaptive decision-making mechanism to generate a comprehensive risk score, and uses the comprehensive risk score and dynamically generated risk threshold to determine a warning level. It then encapsulates this information with multi-level decision data, constructs and reports a warning information message. The human-machine co-driving module uses a synchronous frame aggregation mechanism to time-align a first comprehensive control command set, driver state data, and environmental context data. It applies an environment-adaptive command intent filter to dynamically optimize the first comprehensive control command set, encapsulates it into a control command message, encrypts it, uploads it to the cloud management platform, and sends it to the target vehicle after multi-level serial verification. The collaborative execution module is used by the vehicle terminal to invoke the human-machine control fusion mechanism, dynamically calculate the human-machine control fusion weight based on the comprehensive risk score and driver status data, dynamically weight and fuse the second comprehensive control instruction set and safety correction instructions to generate the underlying control signal, and the remote cockpit executes graded multimodal warnings according to the warning level to generate multimodal feedback signals.

Images