AR-fused remote driving vehicle virtual-real interaction hardware-in-the-loop test system

Abstract

The invention discloses an AR-fused remote driving vehicle virtual-real interaction hardware-in-the-loop test system, particularly relates to the technical field of automatic driving test, and is used for solving the problems of inaccurate coupling between a virtual scene and a real vehicle behavior and lack of AR prompt response evaluation. The method comprises the following steps: firstly, constructing a dynamic obstacle intention-driven prediction model based on time series data of a multi-modal sensor, and generating a trajectory probability distribution and risk thermodynamic diagram; then, space-time alignment of the virtual accident scene and the real environment is achieved through a dynamic binding algorithm, and the virtual-real shielding priority of an AR interface is dynamically adjusted; by simulating abnormal disturbance of a vehicle actuator, synchronously collecting control and watching responses of a driver, and extracting obstacle avoidance path deviation degree and takeover timeliness parameters; and finally, separating and compensating virtual and actual residual errors based on a path deviation index, realizing online correction of a virtual scene attitude and a dynamic trajectory, constructing a closed-loop optimization mechanism, and improving the precision and stability of a test system.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving testing technology, and in particular to a hardware-in-the-loop testing system for virtual-reality interaction of remotely driven vehicles integrated with AR. Background Art

[0002] With the rapid development of intelligent connected vehicles and autonomous driving technologies, remote driving, as a key complementary capability, has gradually become an important supplementary means to solve emergency operating conditions, vehicle-road cooperative assistance, and low-speed scene control. In this process, how to ensure the reliability and response timeliness of remote driving decision-making systems in complex dynamic environments has become a core issue in industry research and testing verification. Although traditional simulation tests are repeatable and controllable, it is difficult to fully evaluate system performance when faced with factors such as the uncertainty of the behavior of dynamic traffic participants, the human-computer interaction response of the AR interface, and actuator interference disturbances. Therefore, building a virtual-reality fusion hardware-in-the-loop test platform for remote driving and enhancing the driver's perception and response capabilities through AR technology have become effective ways to improve system stability and safety.

[0003] Existing testing and evaluation methods for remote driving systems primarily rely on one-way simulation processes or data playback mechanisms in closed environments. These processes, based on static obstacle configurations or fixed behavior patterns, are unable to integrate multimodal sensor data in real time to predict dynamic obstacle intentions, resulting in AR prompts lagging behind actual risk evolution. For example, traditional probabilistic graphical models fail to account for inter-vehicle interactions, significantly increasing prediction errors in complex traffic flows. Mainstream methods rely on two-dimensional coordinate transformations for a single sensor and fail to address the three-dimensional spatial synchronization of multiple sensors and the virtual scene, leading to jittery AR overlay positions. Current testing systems often employ one-way fault injection without synchronously overlaying virtual accident scene disturbances, resulting in driver response assessments that are detached from the real human-machine-environment coupled environment. Furthermore, path deviation calculations rely on a single lateral deviation threshold, ignoring the impact of accumulated heading angle errors on obstacle avoidance safety. Existing virtual-real residual compensation methods fail to distinguish between residual sources in the static environment and those in dynamic objects, leading to conflicting correction strategies and further exacerbating the risk of AR display distortion. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a hardware-in-the-loop testing system for virtual-reality interaction of remote-controlled vehicles integrated with AR, which solves the problems of the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a hardware-in-the-loop test system for virtual-reality interaction of remote-controlled vehicles integrated with AR, comprising the following modules: a dynamic obstacle intention analysis module, a virtual-reality scene spatiotemporal coupling module, a multi-level obstacle avoidance decision evaluation module, and a virtual-reality residual adaptive compensation module; the dynamic obstacle intention analysis module is used to construct an intention-driven behavior prediction model of dynamic obstacles based on the time series data of multimodal sensors, and output trajectory probability distribution and dynamic risk area parameters; the virtual-reality scene spatiotemporal coupling module is used to dynamically bind virtual accident scenes with real sensor data based on the dynamic risk area parameters. The multi-level obstacle avoidance decision evaluation module is used to inject abnormal disturbances of vehicle actuators into the fusion scene after spatiotemporal alignment, synchronously collect the driver's response data to AR guidance, and generate obstacle avoidance path deviation index and takeover timeliness parameters; the virtual-real residual adaptive compensation module is used to separate the static characteristics of the environment and the dynamic object residual according to the obstacle avoidance path deviation index, dynamically compensate the position offset of the virtual-real scene through the online correction algorithm, and feed the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization.

[0006] Furthermore, the construction logic of the intention-driven behavior prediction model for dynamic obstacles based on the time series data of multimodal sensors is as follows: the raw data from the camera, millimeter-wave radar, and lidar are synchronized with the timestamp and converted to the multi-sensor coordinate system to extract the motion feature vector of the dynamic obstacle. The motion feature vector includes the speed change trend, heading angle offset, and acceleration fluctuation pattern. Based on the historical trajectory data and real-time motion characteristics, a spatiotemporal behavior probability graph model of the dynamic obstacle is constructed through a Bayesian network to predict its multimodal trajectory distribution in the future motion direction. The trajectory distribution includes the probability weights of straight driving, lane changing, and sudden braking behaviors. Based on the relative motion direction and lane topology between the dynamic obstacle and the target vehicle, the conflict degree between the trajectory prediction result and the expected path of the target vehicle is calculated, and trajectories with a conflict degree exceeding the safety threshold are marked as high-risk events. The actual motion trajectory of the dynamic obstacle collected in real time is compared with the predicted trajectory, and the node weights of the Bayesian network are reversely adjusted using the gradient descent algorithm to optimize the scenario adaptability of the behavior prediction model.

[0007] Furthermore, the trajectory probability distribution and dynamic risk area parameters are output, including the following steps: mapping the multimodal trajectory prediction results to a two-dimensional rasterized map, generating a dynamic risk heat map based on the probability weight, and the grid density of the heat map is positively correlated with the trajectory conflict probability; extracting the geometric boundaries of the high-risk area based on the density gradient change of the heat map, calculating the lateral and longitudinal buffer distances of the safe obstacle avoidance path in combination with the vehicle dynamics constraints, and dynamically adjusting the transparency and rendering level of the AR prompt information according to the risk area boundary and the driver's field of view focus position; and performing residual convergence detection on the predicted trajectory and real-time sensor data through the extended Kalman filter algorithm. If the residual accumulation in the sliding window exceeds the dynamic adjustment threshold, the reconstruction of the Bayesian network model and the update of the heat map are triggered.

[0008] Furthermore, according to the dynamic risk area parameters, the virtual accident scene and the real sensor data are temporally and spatially aligned through a dynamic binding algorithm, including the following steps: according to the geometric boundaries and heat map distribution in the dynamic risk area parameters, key temporal and spatial feature points in the sensor data, including lane line intersections and traffic sign positions, are extracted, and multimodal feature matching is performed with the topological structure of the virtual accident scene, and the initial coordinate system alignment is achieved through an iterative nearest point algorithm; within a preset time window, the temporal and spatial offsets between the real sensor data and the virtual scene are accumulated and calculated, and the posture parameters of the virtual scene are dynamically adjusted through a nonlinear optimization algorithm to compensate for the superposition error caused by communication delay and sensor sampling jitter; the residuals of the virtual and real scenes after feature matching are estimated in real time through the extended Kalman filter algorithm. If the residual exceeds the dynamic adjustment threshold, the local coordinate system fine-tuning of the virtual scene is triggered.

[0009] Furthermore, the display priority of the virtual and real occlusion areas in the AR interface is dynamically adjusted through the trajectory probability distribution of dynamic obstacles, including the following steps: dividing the AR prompt area into multiple rendering levels according to the high-probability trajectory branches of the trajectory probability distribution, and dynamically assigning transparency levels according to the probability values; detecting the spatial overlapping areas of real obstacles and virtual roadblocks in real time, calculating the occlusion priority based on the trajectory probability weights, obtaining the driver's field of view focus coordinates, and dynamically adjusting the rendering position and size of the AR prompt information; when the trajectory probability distribution mutates due to model reconstruction, the current rendering thread is immediately interrupted, and the occlusion priority is reallocated according to the updated probability weights.

[0010] Furthermore, abnormal disturbances of the vehicle actuators are injected into the fusion scene after spatiotemporal alignment, and the driver's response data to AR guidance is synchronously collected to generate obstacle avoidance path deviation indicators and takeover timeliness parameters, including the following steps: simulating abnormal working conditions of the vehicle actuators through the hardware-in-the-loop interface, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, and synchronously superimposing virtual accident scene disturbances; recording the driver's control signals, eye focus trajectory, AR prompt gaze duration, and vehicle status data, including lateral acceleration and yaw angle, in real time; calculating the lateral deviation integral and heading angle offset accumulation of the vehicle's actual trajectory based on the positioning data and the expected obstacle avoidance path; defining the interval from the prompt triggering moment to the driver's first effective control response as the takeover delay, and calculating the takeover success rate by weighting the path deviation indicator.

[0011] Furthermore, the static features of the environment and the residuals of dynamic objects are separated according to the obstacle avoidance path deviation index, and the position offset of the virtual and real scenes is dynamically compensated by an online correction algorithm, including the following steps: extracting the static features of the environment and the dynamic object features from the lidar point cloud data, and distinguishing the residual components of the static features and the dynamic object features according to the spatiotemporal consistency verification algorithm, the static features include the road surface and fixed buildings, and the dynamic object features include pedestrians and vehicles; according to the lateral deviation integral and the heading angle offset accumulation of the obstacle avoidance path deviation index, it is determined that the source of the residual is the static environment coordinate system offset or the dynamic object motion prediction error, and the static residual vector and the dynamic residual vector are generated by classification; for the static residual vector, the global coordinate system offset is compensated by the least squares optimization algorithm based on the sliding window, and for the dynamic residual vector, the trajectory probability distribution parameters of the dynamic obstacle are reversely corrected by the kinematic prediction model; the corrected static and dynamic residual parameters are input into the virtual and real scene rendering engine, and the posture of the virtual accident scene and the motion trajectory of the dynamic object are adjusted in real time.

[0012] Furthermore, the logic of feeding back the residual compensation amount to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: the static residual vector and the dynamic residual vector are normalized and weighted, and a comprehensive residual compensation weight is generated in combination with the real-time scene complexity, wherein the scene complexity includes traffic flow and lighting conditions; the comprehensive residual compensation weight is input into the dynamic binding algorithm of the virtual-real scene spatiotemporal coupling module, and the objective function weight coefficient of the nonlinear optimization algorithm is adjusted to give priority to compensating high-weight residual components; according to the distribution characteristics of the dynamic residual compensation amount, the real-time update of the display priority of the virtual-real occlusion area in the AR interface is triggered, and the convergence of the residual feedback loop is analyzed by the Lyapunov stability criterion. If the system state variables diverge, switch to the backup parameter set and restart the spatiotemporal coupling optimization thread.

[0013] The present invention has the following beneficial effects: (1) The virtual-reality interaction hardware-in-the-loop test system for remote driving vehicles integrated with AR significantly improves the accuracy of dynamic obstacle trajectory prediction and adaptability to multimodal scenarios through the dynamic obstacle intention analysis module's extraction of temporal features of multimodal sensor data and the Bayesian network-driven behavior prediction model. At the same time, based on conflict degree calculation and high-risk event marking, it realizes the dynamic allocation of AR risk prompt weights driven by intention, effectively reducing the risk of decision-making delays caused by sudden obstacle behavior. The virtual-reality scene spatiotemporal coupling module solves the spatiotemporal reference deviation problem between real sensor data and virtual scenes through iterative nearest point algorithm initial alignment, sliding window nonlinear optimization and extended Kalman filter residual correction, ensuring that the virtual-reality superposition position error is less than the industry standard requirement, and gives priority to rendering real obstacle information through the dynamic occlusion priority adjustment mechanism to avoid visual confusion caused by virtual-reality occlusion conflicts in the interface.

[0014] (2) A hardware-in-the-loop test system for virtual-reality interaction of remote-driving vehicles integrated with AR. The multi-level obstacle avoidance decision-making evaluation module injects abnormal disturbances to the vehicle actuators through the hardware-in-the-loop interface and simultaneously superimposes disturbances of virtual accident scenes to build a high-fidelity machine-machine collaborative test environment. The path deviation index generated by the lateral deviation integral and the heading angle accumulation is combined to quantitatively evaluate the driver's response accuracy to AR guidance; the virtual-reality residual adaptive compensation module realizes dynamic closed-loop compensation of virtual-reality scene position offsets through static and dynamic feature residual separation, sliding window least squares optimization and kinematic model inverse correction. Combined with the feedback loop optimization mechanism of the Lyapunov stability criterion, it ensures the robustness and interactive reliability of the system under complex disturbances, and can be directly applied to road testing and certification of intelligent connected vehicles.

[0015] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the hardware-in-the-loop testing system for virtual-reality interaction of remote-controlled vehicles integrated with AR in the present invention. DETAILED DESCRIPTION

[0017] The embodiments of the present application solve the core problems of insufficient coordination between dynamic obstacle intention prediction and AR prompts in real time, low spatiotemporal alignment accuracy of virtual and real scenes, lack of human-machine-environment coupling verification in hardware-in-the-loop testing, and lack of classification compensation in the residual correction mechanism through the integration of AR remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system.

[0018] The overall idea of ​​the solution in the embodiments of this application is as follows: An intention-driven behavior prediction model for dynamic obstacles is constructed using multimodal sensor data (Bayesian network + gradient descent optimization), outputting high-confidence trajectory probability distribution and dynamic risk parameters.

[0019] Based on the dynamic binding algorithm (iterative closest point initial alignment + sliding window nonlinear optimization + extended Kalman filter residual correction), high-precision spatiotemporal alignment of virtual and real scenes is achieved, and the AR interface occlusion priority is dynamically adjusted based on trajectory probability.

[0020] During the hardware-in-the-loop test, vehicle actuator anomalies and virtual accident scenario disturbances are simultaneously injected, and the driver's response behavior is quantified through the path deviation integral and the heading angle accumulation.

[0021] Finally, based on static / dynamic residual separation and classification compensation (sliding window least squares + kinematic model inverse correction), combined with the Lyapunov stability criterion, closed-loop optimization is achieved to ensure the system's AR interaction reliability and test authenticity in complex scenarios.

[0022] See also Figure 1 , the embodiment of the present invention provides a technical solution: a hardware-in-the-loop test system for virtual-reality interaction of remote-driving vehicles integrated with AR, including the following modules: a dynamic obstacle intention analysis module, a virtual-reality scene spatiotemporal coupling module, a multi-level obstacle avoidance decision evaluation module, and a virtual-reality residual adaptive compensation module; the dynamic obstacle intention analysis module is used to construct an intention-driven behavior prediction model of dynamic obstacles based on the time series data of multimodal sensors, and output trajectory probability distribution and dynamic risk area parameters; the virtual-reality scene spatiotemporal coupling module is used to time-bind virtual accident scenes with real sensor data based on the dynamic risk area parameters through a dynamic binding algorithm. The system dynamically adjusts the display priority of the virtual and real occlusion areas in the AR interface through the trajectory probability distribution of dynamic obstacles; the multi-level obstacle avoidance decision evaluation module is used to inject abnormal disturbances of the vehicle actuator into the fusion scene after spatiotemporal alignment, synchronously collect the driver's response data to AR guidance, and generate obstacle avoidance path deviation indicators and takeover timeliness parameters; the virtual and real residual adaptive compensation module is used to separate the static characteristics of the environment and the dynamic object residuals according to the obstacle avoidance path deviation indicators, dynamically compensate for the position offset of the virtual and real scenes through an online correction algorithm, and feed the residual compensation amount back to the virtual and real scene spatiotemporal coupling module for closed-loop optimization.

[0023] In this implementation, the Dynamic Obstacle Intent Parsing Module builds an obstacle behavior prediction model based on continuous time-series data collected by multimodal sensors (including cameras, millimeter-wave radar, and lidar). This model integrates historical behavior and real-time motion features to identify and predict the intentions and future trajectories of dynamic participants in traffic scenarios (such as pedestrians, vehicles, and cyclists). It outputs trajectory distribution information with probability weights and dynamic risk area descriptions. This module provides real-time risk assessment for subsequent AR rendering priority control and obstacle avoidance decisions. The Intent-Driven Behavior Prediction Model combines a Bayesian inference framework with temporal pattern recognition for behavior modeling. Input variables include the obstacle's speed, acceleration, heading angle, and historical trajectory. The model calculates the posterior probability distribution for various behavior modes (such as going straight, changing lanes, and decelerating) using trained prior probabilities and likelihood functions. The trajectory probability distribution is based on the multiple possible trajectories and their corresponding probabilities output by the intention model. For example, for an obstacle, the predicted results are: 60% for lane change left, 30% for going straight, and 10% for lane change right. This distribution is used to assess potential conflict areas and expected dynamic change trends. Dynamic risk area parameters: These include the spatial boundaries of the intersection between the predicted trajectory and the ego-vehicle path (e.g., polygonal envelopes like rectangles and ellipses) and the risk level of this area (typically categorized as high, medium, and low). These parameters are used to define the priority and scope of AR prompt triggering. The virtual-real scene spatiotemporal coupling module: This module implements high-precision spatiotemporal fusion of virtual traffic event scenes (e.g., virtual landslides and simulated construction obstacles) with sensor data from the real road environment. Coordinate alignment and delay compensation algorithms ensure seamless overlay of virtual and real elements in the AR interface, eliminating visual artifacts such as misalignment and drift. The display order of virtual and real objects is adjusted based on obstacle risk levels to prevent user attention from being misdirected. The dynamic binding algorithm: This algorithm dynamically aligns virtual and real information in space and time. It includes the following key processes: Iterative Closest Point (ICP) initial alignment: This algorithm extracts key geometric feature points (e.g., road boundary points and lane intersections) from the virtual and real environments and iteratively optimizes the coordinate transformation matrix to achieve coarse-precision registration. Sliding window nonlinear optimization: This algorithm, taking into account factors such as sensor sampling delay and frame rate variations, performs nonlinear optimization adjustments based on a multi-frame window of historical data to achieve temporal synchronization. Extended Kalman filter residual correction: Combined with dynamic system state prediction, the coupling error is filtered and updated to eliminate cumulative errors and maintain local registration accuracy. Dynamic adjustment of display priority: When virtual scene elements (such as virtual lane closures) overlap with real objects (such as real pedestrians), the system determines the rendering priority based on the risk level of the obstacle and the trajectory prediction weight, ensuring that high-risk real targets are displayed and prompted first. Multi-level obstacle avoidance decision evaluation module: This module simulates actuator-level fault disturbances (such as steering wheel failure and brake lag) in the coupled virtual-reality fusion scene, and guides users to make obstacle avoidance operations in combination with specific AR prompts.At the same time, data on the driver's takeover behavior is collected to evaluate their response efficiency to AR prompts, path execution accuracy, and fault tolerance, forming a multi-dimensional obstacle avoidance decision-making capability evaluation metric system. A hardware-in-the-loop interface (HIL interface) uses a physical simulation platform (such as dSPACE or NI) or a force feedback device to inject fault signals from a virtual environment into the vehicle's actuator control loop in real time. This integrates real hardware and virtual simulation, improving the realism and accuracy of the test. Path deviation metrics quantify the degree to which the vehicle's actual obstacle avoidance path deviates from the intended path. Specifically, these metrics include: lateral deviation integral (the integral of the lateral distance between the vehicle's center of mass and the centerline of the planned path); and heading angle offset accumulation (the integral of the deviation between the vehicle's current heading angle and the tangent of the target path, reflecting directional control error). A takeover timeliness parameter measures the time interval from the triggering of the system's AR prompt to the driver's first active and effective action (such as steering or braking). Dynamically weighted path deviations are combined to produce the "effective takeover success rate." Adaptive Compensation Module for Virtual-Real Residual Errors: This module performs fine-grained classification of position errors generated in the fusion environment, separately addressing virtual-real residual errors between static scenes (such as roads and buildings) and dynamic objects (such as pedestrians and vehicles). Through residual separation, trajectory inverse modeling, and online compensation, the system achieves adaptive closed-loop correction, further improving scene fusion accuracy and stability. Static feature residuals: Systematic deviations caused by errors in lidar point cloud calibration or map coordinate system drift, such as misalignment between AR virtual lane lines and real road markings. Dynamic object residuals: Position prediction deviations of dynamic objects caused by factors such as errors in the behavior prediction model and environmental emergencies, such as a deviation between the predicted walking path of a pedestrian and its actual behavior. Online Correction Algorithm: Sliding Window Least Squares Optimization: For static scene deviations, a least squares problem is constructed by selecting multiple consecutive time frames to fit the optimal coordinate mapping between the real and virtual scenes. Kinematic Model Inverse Correction: Based on the dynamic or Bayesian inference model of the predicted trajectory, the model parameters are inversely optimized using the observed trajectory to achieve closed-loop correction of dynamic residual errors. Lyapunov stability criterion: Use the monotonicity of the Lyapunov function in control theory as a verification criterion for system convergence to ensure that the compensation feedback system does not exhibit unstable behaviors such as oscillation and divergence during closed-loop operation.

[0024] Specifically, the construction logic of the intention-driven behavior prediction model for dynamic obstacles based on the time series data of multimodal sensors is as follows: the raw data of the camera, millimeter-wave radar and lidar are synchronized with the timestamp and converted to the multi-sensor coordinate system to extract the motion feature vector of the dynamic obstacle. The motion feature vector includes the speed change trend, heading angle offset and acceleration fluctuation pattern; based on the historical trajectory data and real-time motion characteristics, a spatiotemporal behavior probability graph model of the dynamic obstacle is constructed through a Bayesian network to predict its multimodal trajectory distribution in the future motion direction. The trajectory distribution includes the probability weights of straight-ahead, lane-changing and sudden braking behaviors; based on the relative motion direction and lane topology between the dynamic obstacle and the target vehicle, the conflict degree between the trajectory prediction result and the expected path of the target vehicle is calculated, and the trajectory with a conflict degree exceeding the safety threshold is marked as a high-risk event; the actual motion trajectory of the dynamic obstacle collected in real time is compared with the predicted trajectory, and the node weights of the Bayesian network are reversely adjusted according to the gradient descent algorithm to optimize the scenario adaptability of the behavior prediction model.

[0025] In this implementation plan, the basic technical process of multimodal sensor data preprocessing and unified modeling is as follows: timestamp synchronization of raw data from cameras, millimeter-wave radars, and lidars; using external parameter calibration matrices to complete coordinate system conversion between sensors and uniformly map them to the vehicle coordinate system or world coordinate system. Interpolation or alignment of multi-source data ensures that all sensor observations fall at the same time. Coordinate system conversion: Through the rotation matrix R and the translation vector t, a rigid transformation from any sensor coordinate system Fi to the target coordinate system F0 is achieved: ; Where: Pi: 3D points collected by sensor i; P0: point cloud data in a unified coordinate system. Constructing dynamic obstacle motion feature vector Feature extraction content: Extract the following real-time motion features from the fused target tracking data to form the input vector required for behavior prediction : : instantaneous speed; :acceleration; : heading angle; : heading offset; : Standard deviation of acceleration fluctuation (indicating the degree of behavioral instability). Eigenvector representation: ; Bayesian network constructs spatiotemporal behavior probability graph model Modeling logic: The obstacle motion behavior is modeled as a set of hidden state nodes (such as "straight ahead", "change lane left", "brake suddenly", etc.), and the Bayesian network is used to represent its state transition relationship and observation probability to generate a spatiotemporal probability graph. Mathematical model structure: Let the state set be , corresponding to the behavior modes (such as going straight, changing lanes, and braking suddenly): The network structure includes: prior probability , conditional probability : Given the feature vector, the probability prediction results of each behavior are obtained through maximum a posteriori inference: ;in: : the prior frequency of the behavior in the historical trajectory; : The likelihood of feature distribution under each behavior (can be modeled using Gaussian distribution); : The final probability output is used to judge the behavior tendency. Multimodal trajectory prediction and risk assessment technology content: Based on state probability , for each type of behavior Generate several future time steps respectively Movement trajectory ; Each trajectory is attached with its corresponding probability weight , forming the trajectory probability distribution set: ; Risk assessment: define the expected path of the vehicle as , and the obstacle trajectory Conduct conflict analysis; calculate the spatial and temporal overlap of the intersection interval , and introduce the trajectory conflict index : ;in: : is the Euclidean distance between trajectories; : is the distance tolerance parameter (usually set to half the vehicle width); if (set risk threshold), it is marked as a high-risk trajectory event. Model adaptive optimization (residual reverse correction) real-time correction: collect the real trajectory of the obstacle ; and predicted trajectory Compare and define the trajectory residual: ;Construct residual loss function , and the node parameters of the Bayesian network (including conditional probability table, prior probability, etc.) Perform gradient descent update: ;in: : Learning rate; parameter update enables the model to gradually adapt to the behavioral characteristics of the current scene and improve prediction accuracy.

[0026] Specifically, the output trajectory probability distribution and dynamic risk area parameters include the following steps: mapping the multimodal trajectory prediction results to a two-dimensional rasterized map, generating a dynamic risk heat map based on the probability weight, and the grid density of the heat map is positively correlated with the trajectory conflict probability; extracting the geometric boundaries of the high-risk area based on the density gradient change of the heat map, calculating the lateral and longitudinal buffer distances of the safe obstacle avoidance path in combination with the vehicle dynamics constraints, and dynamically adjusting the transparency and rendering level of the AR prompt information according to the risk area boundary and the driver's field of view focus position; and performing residual convergence detection on the predicted trajectory and real-time sensor data through the extended Kalman filter algorithm. If the accumulated residual in the sliding window exceeds the dynamic adjustment threshold, the reconstruction of the Bayesian network model and the update of the heat map are triggered.

[0027] In this implementation, the principle of mapping trajectory probability distribution to a two-dimensional grid map is to map the multimodal predicted trajectory results to a two-dimensional plane map with a fixed resolution, and establish a spatial distribution representation based on trajectory probability. Grid mapping method: Assume that the two-dimensional map is divided into a grid set ,in Represents the horizontal and vertical grid index; for each predicted track (including time series point set) and its weight , counting the number of cells passing through each grid The risk value of each grid is weighted and summed by the probability weight: ;in: : Grid Heat map risk value; : Trajectory The probability weight of : indicator function, which is 1 when the trajectory passes through the grid, and 0 otherwise; : Total number of trajectory samples. Description: In the heat map The larger the value, the more likely the location is to have a trajectory conflict. Principle of the technology for extracting dynamic risk area boundaries based on gradient fields: Based on the gradient field changes of the heat map, high-risk dense areas are identified and their geometric boundaries are extracted for subsequent obstacle avoidance judgment. Mathematical processing: Constructing the heat map gradient field: ; Use the gradient modulus to determine the rising boundary of the risk area: ; Set threshold , extract satisfaction The continuous closed area is taken as the geometric boundary of the high-risk area and is recorded as: . Principle of calculating obstacle avoidance buffer distance based on vehicle dynamics: Combine the maximum steering ability, braking performance and dynamic response delay of the target vehicle to calculate the lateral and longitudinal buffer distances required for the vehicle to safely pass through the risk area. Mathematical model: Assume that the maximum safe lateral acceleration of the vehicle is , the current speed is , the delayed reaction time is ;Then the horizontal buffer distance , longitudinal buffer distance Expressed as: ;in: : Maximum longitudinal deceleration capacity; : Includes the total delay of driver recognition + operation response (derived from historical takeover parameter statistics). Step 4: Dynamically adjust the AR rendering technology mechanism based on the driver's field of view focus position: Set the center position of the driver's current gaze area to , obtained through eye tracking device; calculate risk area Center of gravity ; Calculate its field of view deviation angle : ;in: :Camera focal length, used for viewing angle normalization; The smaller the value, the closer it is to the driver's gaze center. , rendering priority according to Dynamic settings: ;in: : Experience parameter adjustment coefficient; makes the prompts closer to the driver's line of sight clearer and more prioritized. Technical principle of residual convergence detection mechanism based on extended Kalman filter: continuously monitor the deviation residual between the predicted trajectory and the sensor measured trajectory to evaluate the stability of the prediction system. Key formula: The state prediction trajectory is , the sensor measured trajectory is ;Define filter residual: ; Sliding window residual accumulation (in is the window length): ;like , then the trajectory model reconstruction and heat map refresh process are triggered. Model adaptive feedback: clear the Bayesian model cache and retrain the trajectory probability distribution ; Update grid heat map , correct the location of high-risk areas.

[0028] Specifically, according to the dynamic risk area parameters, the virtual accident scene and the real sensor data are temporally and spatially aligned through a dynamic binding algorithm, including the following steps: according to the geometric boundaries and heat map distribution in the dynamic risk area parameters, key temporal and spatial feature points in the sensor data, including lane line intersections and traffic sign positions, are extracted, and multimodal feature matching is performed with the topological structure of the virtual accident scene, and the initial coordinate system alignment is achieved through an iterative nearest point algorithm; within a preset time window, the temporal and spatial offsets between the real sensor data and the virtual scene are accumulated and calculated, and the posture parameters of the virtual scene are dynamically adjusted through a nonlinear optimization algorithm to compensate for the superposition error caused by communication delay and sensor sampling jitter; the residuals of the virtual and real scenes after feature matching are estimated in real time through the extended Kalman filter algorithm. If the residual exceeds the dynamic adjustment threshold, the local coordinate system fine-tuning of the virtual scene is triggered.

[0029] In this implementation, multimodal feature extraction and initial registration are performed: key topological feature points (lane intersections, traffic signs, corner buildings) are extracted from the real and virtual scenes, and initial coordinate system alignment is achieved through point cloud registration. The operation process is to select feature point sets such as lane intersections, intersection signs, and cross-sectional outlines from the high-density area of ​​the heat map; assuming that the real scene point set is: ; The corresponding point set in the virtual scene is: ; Use the iterative closest point (ICP) algorithm to obtain the rigid body transformation matrix , so that the matching error is minimized: ;in: : initial rotation matrix; : initial translation vector; : Euclidean norm. Nonlinear optimization dynamic compensation communication and sampling offset technical background: After initial registration, communication delay (such as AR head-mounted display delay) and sensor sampling jitter will cause spatiotemporal drift, requiring online correction of the virtual scene posture. Algorithm processing: In the time window In , record the drift error vector after each frame registration: ;in: : Number of frames in the sliding window; : The pose of the virtual scene in the current frame; : The current frame registration residual. Construct the residual cumulative cost function for nonlinear optimization: ;in: : Rotation fine-tuning vector; : Translation fine-tuning vector; : The Jacobian matrix of the residual with respect to rotation and translation. Solve for the optimal compensation , used to update the virtual scene pose in real time: .in, : historical translation vector, : The adjusted rotation matrix, : history rotation matrix, : The adjusted translation vector. The optimal solution is obtained by setting the residual error objective function: Let the current moment measured sensor point cloud / image feature be , the virtual scene projection feature is , then construct the optimization objective function: ;in: :In the virtual scene 3D feature points; : projection function, simulating camera / sensor imaging model; : Euclidean distance norm; : The number of features involved in alignment. Extended Kalman filter residual estimation and triggering mechanism Technical goal: Real-time estimation of registration residuals, and when the cumulative error exceeds the set threshold, trigger the local coordinate system readjustment process to ensure stable alignment of the virtual scene. State definition: Define the system state vector: ; represents the posture state of the virtual scene at the kth moment (rotation and translation parameters). Filter residual calculation: The observation model is: ;in is a projection function that maps the scene pose to the sensor coordinate system; the residual is defined as: ; The extended Kalman filter state is updated as: ;in is the Kalman gain. Sliding residual statistics: in the window Inner cumulative residual norm squared: ;like , the following operations are triggered: local coordinate system offset correction; restart of nonlinear optimization process; update of heat map spatial weight to adapt to drift correction.

[0030] Specifically, the display priority of the virtual and real occlusion areas in the AR interface is dynamically adjusted through the trajectory probability distribution of dynamic obstacles, including the following steps: dividing the AR prompt area into multiple rendering levels according to the high-probability trajectory branches of the trajectory probability distribution, and dynamically assigning transparency levels according to the probability values; detecting the spatial overlapping areas of real obstacles and virtual roadblocks in real time, calculating the occlusion priority based on the trajectory probability weights, obtaining the driver's field of view focus coordinates, and dynamically adjusting the rendering position and size of the AR prompt information; when the trajectory probability distribution mutates due to model reconstruction, the current rendering thread is immediately interrupted, and the occlusion priority is reallocated according to the updated probability weights.

[0031] In this implementation, a multi-layer AR rendering hierarchy is constructed based on trajectory probability distribution: according to the multimodal trajectory distribution output by the trajectory prediction model, the trajectory branches with higher probability values ​​are extracted, and a hierarchical display structure of the AR prompt area is constructed accordingly, and transparency is mapped by probability weighting. Operation instructions: Assume that the predicted trajectory set output by the system is: ; among them Tracks The corresponding probability of occurrence is ,satisfy: ; According to trajectory probability The size of the track is divided into different rendering levels: Let the rendering level set be ,in Indicates the Layer rendering layer; construct probability layer mapping function: ;in: is the level boundary threshold, satisfying ; Each layer is based on Dynamically calculated transparency : ;in: : Transparency control upper and lower limits; The larger the value, the more opaque the AR prompt (the higher the priority). Real-time evaluation of virtual and real occlusion conflicts and adjustment of display layout: Detect the spatial overlap area of ​​virtual prompt information and real obstacles through geometric calculations, calculate the occlusion priority based on trajectory probability weighting, and dynamically adjust the AR prompt display position and size based on the driver's field of view focus. Mathematical modeling: Define the virtual and real occlusion conflict area : ;in: : Mapping of the virtual prompt area in screen projection coordinates; : The projection mapping of the obstacle prediction position in the current frame. Calculate the occlusion weight in combination with the trajectory probability : ;in: : All and prompt areas There are overlapping sets of trajectories; : occlusion weight factor, Control weight sensitivity; : The function represents the area of ​​the projection area. The focus position of the driver's field of view is , calculate the priority offset direction of the prompt information: ;in: : The current center position of the prompt information; : two-dimensional Euclidean distance; : Position fine-tuning direction vector. Finally update rendering position and size: Position fine-tuning: ; Size adjustment: ;in: : original display size; : adjustment coefficient; : The maximum occlusion conflict amount in the current frame. Rendering thread interruption and occlusion redistribution logic during model reconstruction: When the trajectory probability distribution changes due to the Bayesian behavior model update or residual mutation, resulting in significant fluctuations in trajectory probability (i.e., a group of trajectories with a probability increase of Exceeding the dynamic reconstruction threshold ), the existing AR rendering thread needs to be interrupted immediately and the occlusion layering logic needs to be recalculated. Interruption trigger condition expression: Let the new and old trajectory probabilities be and ,but: ;Interrupt processing content: forcefully terminate the current rendering thread; clear the current frame layer display stack; reallocate: rendering level ; Transparency of each layer ; Occlusion priority ; Prompt position and size .

[0032] Specifically, abnormal disturbances of vehicle actuators are injected into the fusion scene after spatiotemporal alignment, and the driver's response data to AR guidance is synchronously collected to generate obstacle avoidance path deviation indicators and takeover timeliness parameters, including the following steps: simulating abnormal working conditions of vehicle actuators through the hardware-in-the-loop interface, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, and synchronously superimposing virtual accident scene disturbances; recording the driver's control signals, eye focus trajectory, AR prompt gaze duration, and vehicle status data, including lateral acceleration and yaw angle, in real time; calculating the lateral deviation integral and heading angle offset accumulation of the vehicle's actual trajectory based on the positioning data and the expected obstacle avoidance path; the interval time from the AR prompt triggering moment to the driver's first effective control response is defined as the takeover delay, and the takeover success rate is calculated by weighting the path deviation indicator.

[0033] In this implementation, actuator abnormal disturbance injection and virtual-real fusion are collected synchronously: Actuator abnormal disturbance injection simulates the following typical vehicle actuator abnormal working conditions through the hardware-in-the-loop system: Steering system torque mutation: injecting unexpected instantaneous disturbance torque into the vehicle's wire-controlled steering system , causing the actual steering angle to deviate; Braking system response delay: artificially set the braking delay period , compared with normal working conditions; throttle system opening drift: introducing throttle control error , making the actual acceleration behavior inconsistent with the expected; and simultaneously superimposing virtual accident scene disturbances (such as sudden lane changes, static obstacles, blind spot intrusions) to increase the complexity of the test and the degree of environmental realism. Data collection items and their mapping relationships, the system collects in real time and synchronously:

[0034] The obstacle avoidance path deviation index is constructed by combining the expected obstacle avoidance trajectory with the high-precision map The actual trajectory of the vehicle , construct the following two indicators: lateral deviation integral index A measure of the vehicle's lateral stability relative to its desired trajectory: ; : The arc length of the vehicle on the path The lateral offset distance at 、 : represent the starting and ending arc lengths of the path respectively; :The larger the value, the greater the lateral control error of the vehicle, indicating poor obstacle avoidance accuracy. Used to describe the cumulative error of direction control and trajectory target consistency: ; : The yaw angle of the vehicle during actual driving; : The heading angle of the expected trajectory at the corresponding time point; : The time interval during which the disturbance is injected into the vehicle to stabilize; :The larger the value, the worse the vehicle's direction control accuracy, indicating that the driver failed to effectively return to the correct trajectory. Takeover response timeliness evaluation, takeover delay time Defined as: ; : The moment when the AR prompt is generated and enters the driver's field of view; : The time when the driver first triggers a valid action (such as a significant steering wheel angle, brake or accelerator input); :The smaller the value, the faster the takeover response, which improves the safety of the scene. , combining path deviation and response delay to construct a weighted composite evaluation function: ; is an empirical weight coefficient, adjusted according to actual test data; the indicator range is [0,1], and the closer the value is to 1, the faster and more accurately the driver can complete the evasive takeover; it is differentiable and can be used as a gradient feedback source for subsequent driving strategy optimization models.

[0035] Specifically, the static features of the environment and the residuals of dynamic objects are separated according to the obstacle avoidance path deviation index, and the position offset of the virtual and real scenes is dynamically compensated by the online correction algorithm, including the following steps: extracting the static features of the environment and the dynamic object features from the lidar point cloud data, and distinguishing the residual components of the static features and the dynamic object features according to the spatiotemporal consistency verification algorithm, the static features include the road surface and fixed buildings, and the dynamic object features include pedestrians and vehicles; according to the lateral deviation integral and the heading angle offset accumulation of the obstacle avoidance path deviation index, the residual source is determined to be the static environment coordinate system offset or the dynamic object motion prediction error, and the static residual vector and the dynamic residual vector are generated by classification; for the static residual vector, the global coordinate system offset is compensated by the least squares optimization algorithm based on the sliding window, and for the dynamic residual vector, the trajectory probability distribution parameters of the dynamic obstacle are reversely corrected by the kinematic prediction model; the corrected static and dynamic residual parameters are input into the virtual and real scene rendering engine, and the posture of the virtual accident scene and the motion trajectory of the dynamic object are adjusted in real time.

[0036] In this implementation, LiDAR point cloud feature extraction and residual separation are performed. Feature point extraction extracts structural features from LiDAR point cloud data between frames. Voxel Grid Filter is used to reduce noise. Curvature change rate is used to identify edge / plane features and construct a feature set. ;in Indicates the current frame number, For the The three-dimensional coordinates of the points. Spatiotemporal consistency check and residual separation Assume that the feature point of the previous frame is , the current frame point is , then the point pair residual is: ; : Estimated pose transformation between frames (4×4 homogeneous matrix); if , it is considered as a static residual point; otherwise it is considered as a dynamic object point or a sensor error point. The set is divided as follows: Static residual set: Dynamic residual set: The residual source classification judgment based on the path deviation index is combined with the existing obstacle avoidance path lateral deviation integral index. Accumulated error with heading angle , construct the residual source judgment logic function: ; : Error tolerance threshold; if the deviation is small, it is considered to be a scene reference error (static residual); if the deviation is significant, it is determined to be an error caused by dynamic target prediction error. Generate two types of residual vectors: static residual vector: ; Dynamic residual vector: Static residual compensation: Sliding window least squares optimization is used to eliminate the static alignment error between virtual and real scenes, and the position compensation objective function within the sliding window is constructed: ; : The static coordinate translation to be optimized; : Sliding window frame length; :No. The number of residual points of the frame; this optimization can be solved in real time using the Gauss-Newton method or QR decomposition to obtain the position offset compensation vector Finally, the global coordinates of the virtual scene are corrected to: Dynamic residual correction: trajectory probability distribution compensation is to correct the dynamic obstacle prediction offset, and the original predicted trajectory probability distribution parameters Perform residual feedback correction. Model representation: Assume that the multimodal distribution of the trajectory of a dynamic target at the current moment is as follows: ;in: :No. The probability weight of each trajectory branch; : Mean value of trajectory position (two-dimensional coordinate); : covariance matrix; correction method: according to the residual vector Corrected trajectory distribution: ; : Step coefficient, indicating the degree of correction to the current residual; this method can be compared to mean bias compensation for Bayesian prediction models, improving the adaptability and online consistency of dynamic models. The rendering engine updates the virtual and real poses. Finally, the corrected residuals are input to the rendering engine, and two types of corrections are performed: Global scene pose update (for static residuals): ;Virtual dynamic object trajectory update (for dynamic residual): The rendering engine redraws the environment frame and obstacle motion animation of the virtual accident scene in real time to maintain dynamic alignment with the real scene.

[0037] Specifically, the logic of feeding back the residual compensation amount to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: the static residual vector and the dynamic residual vector are normalized and weighted, and the comprehensive residual compensation weight is generated in combination with the real-time scene complexity, where the scene complexity includes traffic flow and lighting conditions; the comprehensive residual compensation weight is input into the dynamic binding algorithm of the virtual-real scene spatiotemporal coupling module, and the objective function weight coefficient of the nonlinear optimization algorithm is adjusted to prioritize compensation of high-weight residual components; based on the distribution characteristics of the dynamic residual compensation amount, the real-time update of the display priority of the virtual-real occlusion area in the AR interface is triggered, and the convergence of the residual feedback loop is analyzed by the Lyapunov stability criterion. If the system state variables diverge, switch to the backup parameter set and restart the spatiotemporal coupling optimization thread.

[0038] In this implementation, the residual normalization weighting is integrated with the scene complexity, and the static and dynamic residual vectors are: Dynamic residual vector: . Normalization processing normalizes the residuals separately: ; : A small constant to prevent division by zero; a residual signal with the same direction but normalized modulus is obtained. The scene complexity factor is constructed to introduce traffic flow and the mean light intensity (normalized), define the scene complexity coefficient : ; : Weight coefficient, satisfying ; :traffic density; :Ambient lighting; : The larger the value, the more complex the scene. The comprehensive residual compensation weight is generated to define the weighting factor: ; Construct weighted residual compensation: ; : Upgrade the dynamic residual vector (fill with zeros) to match the three-dimensional static residual dimension; :Used for the unified residual input of the coupling module. The objective function weight in the dynamic binding algorithm is updated in the virtual real-time space-time binding module. The original nonlinear optimization objective function is: ; : Virtual scene pose parameters (including rotation and translation); : Feature point pair weight coefficient; , : Sensor and virtual feature point pairs; solved by Ceres. Update objective function weights: according to the modulus of weighted residual compensation , dynamically adjust the weights of various point pairs: ; : initial weight; : Residual amplification factor related to point pair categories; corresponding to high residual areas, binding optimization will give higher matching accuracy priority. The priority update of virtual and real occlusion areas guides the adjustment of AR interface display levels based on dynamic residual distribution: Construct occlusion priority function: ; : The angle between the residual direction and the driver's field of view direction; : weight coefficient; : Used to control rendering transparency and Zbuffer priority; display module based on Update the display status of each occluded area to enhance the prompt effect of dynamic objects with large residuals. Convergence analysis of residual feedback closed-loop system (Lyapunov criterion) Define the system state variables Represents the current virtual-real alignment error state and constructs the Lyapunov function: ; : positive definite matrix; if it satisfies , the closed-loop system is stable. If there exists any moment that satisfies: ; Indicates that the system is diverging, and immediately triggers the emergency process: switch to the backup parameter set ; Reset the objective function weight and residual state; Restart the dynamic binding optimization thread.

[0039] In summary, this application has at least the following effects: A hardware-in-the-loop (HIL) test system for virtual-reality interaction in a remotely piloted vehicle, integrated with AR, introduces a dynamic residual separation and weighted compensation mechanism, combined with a nonlinear optimization algorithm and feature-level alignment methods, to effectively suppress virtual scene pose errors caused by sensor drift and communication delay, achieving high-precision coupling between virtual information and the real environment. A dynamic risk heat map is constructed based on Bayesian behavior prediction and probabilistic trajectory distribution. Combined with a dynamic rendering mechanism for the AR interface, this system enables drivers to perceive potential collision risk areas and high-priority obstacle avoidance prompts in real time, enhancing driving safety and proactive decision-making. By injecting abnormal vehicle actuator conditions and collecting driver interaction data, the system quantifies the effectiveness of the driver's response to AR prompts and the timeliness of takeover, forming objective and traceable safety performance evaluation metrics. Combining the driver's field of view focus, the degree of probabilistic obstacle occlusion, and the dynamic characteristics of the environment, the system dynamically adjusts the transparency and display priority of AR prompts, achieving perceptual coordination and cognitive load balancing of information presentation. Lyapunov stability criteria are used to control the convergence of the residual feedback system, maintaining stable operation despite changes in scene complexity or abnormal disturbances, improving the overall system's robustness and self-recovery capabilities. By integrating multimodal sensor data with online learning mechanisms, the system has the ability to quickly model and predict dynamic environmental changes, and can provide stable and efficient auxiliary decision support in typical scenarios such as urban congestion, high-speed traffic, and sudden changes in lighting.

[0040] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may be implemented entirely in hardware, entirely in software, or in a combination of software and hardware. Furthermore, the present invention may be implemented as a computer program product embodied in one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0041] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0042] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0044] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0045] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A hardware-in-the-loop test system for virtual-reality interaction of remotely driven vehicles integrated with AR, characterized by: It includes the following modules: dynamic obstacle intention analysis module, virtual-real scene spatiotemporal coupling module, multi-level obstacle avoidance decision evaluation module, and virtual-real residual adaptive compensation module; The dynamic obstacle intention analysis module is used to build an intention-driven behavior prediction model for dynamic obstacles based on the time series data of the multimodal sensor, and output trajectory probability distribution and dynamic risk area parameters; The virtual-real scene spatiotemporal coupling module is used to align the virtual accident scene with the real sensor data in spatiotemporal order according to the dynamic risk area parameters through a dynamic binding algorithm, and dynamically adjust the display priority of the virtual-real occlusion area in the AR interface according to the trajectory probability distribution of the dynamic obstacle; The multi-level obstacle avoidance decision evaluation module is used to inject abnormal disturbances of vehicle actuators into the fusion scene after spatiotemporal alignment, synchronously collect the driver's response data to AR guidance, and generate obstacle avoidance path deviation indicators and takeover timeliness parameters; The virtual-real residual adaptive compensation module is used to separate the static characteristics of the environment and the dynamic object residuals according to the obstacle avoidance path deviation index, dynamically compensate the position offset of the virtual-real scene through an online correction algorithm, and feed back the residual compensation amount to the virtual-real scene spatiotemporal coupling module for closed-loop optimization.

2. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 1 is characterized by: The construction logic of the intention-driven behavior prediction model for dynamic obstacles based on the time series data of multimodal sensors is as follows: Perform timestamp synchronization and multi-sensor coordinate system conversion on the raw data from the camera, millimeter-wave radar, and lidar to extract the motion feature vectors of dynamic obstacles. The motion feature vectors include speed change trends, heading angle offsets, and acceleration fluctuation patterns. Based on historical trajectory data and real-time motion characteristics, a Bayesian network is used to construct a spatiotemporal behavior probability graph model of dynamic obstacles, predicting their multimodal trajectory distribution in the future direction of movement. The trajectory distribution includes probability weights for straight driving, lane changing, and sudden braking. Based on the relative motion direction of the dynamic obstacle and the target vehicle and the lane topology, the conflict degree between the trajectory prediction result and the target vehicle's expected path is calculated, and trajectories with a conflict degree exceeding the safety threshold are marked as high-risk events; The actual motion trajectory of dynamic obstacles collected in real time is compared with the predicted trajectory, and the node weights of the Bayesian network are reversely adjusted according to the gradient descent algorithm to optimize the scenario adaptability of the behavior prediction model.

3. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 2 is characterized by: Outputting trajectory probability distribution and dynamic risk area parameters includes the following steps: The multimodal trajectory prediction results are mapped to a two-dimensional grid map, and a dynamic risk heat map is generated based on the probability weight. The grid density of the heat map is positively correlated with the probability of trajectory conflict. The geometric boundaries of high-risk areas are extracted based on the density gradient changes of the heat map. The lateral and longitudinal buffer distances of the safe obstacle avoidance path are calculated in combination with vehicle dynamics constraints. The transparency and rendering level of the AR prompt information are dynamically adjusted based on the risk area boundary and the driver's field of view focus position. The extended Kalman filter algorithm is used to detect the residual convergence of the predicted trajectory and real-time sensor data. If the accumulated residual in the sliding window exceeds the dynamically adjusted threshold, the reconstruction of the Bayesian network model and the update of the heat map are triggered.

4. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 3 is characterized by: Based on the dynamic risk area parameters, the virtual accident scene is spatiotemporally aligned with the real sensor data through a dynamic binding algorithm, which includes the following steps: Based on the geometric boundaries and heat map distribution in the dynamic risk area parameters, key spatiotemporal feature points in the sensor data, including lane intersections and traffic sign locations, are extracted. Multimodal feature matching is performed with the topological structure of the virtual accident scene, and initial coordinate system alignment is achieved through an iterative closest point algorithm. Within a preset time window, the temporal and spatial offsets between real sensor data and the virtual scene are accumulated and calculated, and the pose parameters of the virtual scene are dynamically adjusted through a nonlinear optimization algorithm to compensate for the superposition error caused by communication delay and sensor sampling jitter. The extended Kalman filter algorithm is used to estimate the residual of the virtual and real scenes after feature matching in real time. If the residual exceeds the dynamic adjustment threshold, the local coordinate system of the virtual scene is fine-tuned.

5. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 4 is characterized by: Dynamically adjusting the display priority of virtual and real occlusion areas in the AR interface based on the probability distribution of dynamic obstacle trajectories includes the following steps: Based on the high-probability trajectory branches of the trajectory probability distribution, the AR prompt area is divided into multiple rendering levels, and the transparency level is dynamically assigned according to the probability value; Real-time detection of the spatial overlap between real obstacles and virtual roadblocks, calculation of occlusion priority based on trajectory probability weights, obtaining the driver's field of view focus coordinates, and dynamically adjusting the rendering position and size of AR prompt information; When the trajectory probability distribution mutates due to model reconstruction, the current rendering thread is immediately interrupted and the occlusion priority is redistributed according to the updated probability weight.

6. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 5 is characterized by: Inject abnormal disturbances to the vehicle actuators into the spatiotemporally aligned fusion scene, synchronously collect the driver's response data to the AR guidance, and generate obstacle avoidance path deviation indicators and takeover timeliness parameters. The process includes the following steps: The hardware-in-the-loop interface simulates abnormal operating conditions of vehicle actuators, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, while simultaneously superimposing virtual accident scenario disturbances. Real-time recording of the driver's control signals, eye focus trajectory, AR prompt gaze duration, and vehicle status data, including lateral acceleration and yaw angle; Based on the positioning data and the expected obstacle avoidance path, the lateral deviation integral and the heading angle offset accumulation of the vehicle's actual trajectory are calculated; The interval from the moment the AR prompt is triggered to the driver's first effective control response is defined as the takeover delay, and the takeover success rate is calculated by combining the weighted path deviation index.

7. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 6 is characterized by: The method separates the static characteristics of the environment and the residuals of dynamic objects based on the obstacle avoidance path deviation index, and dynamically compensates for the position offset of the virtual and real scenes through an online correction algorithm. The method includes the following steps: Extracting environmental static features and dynamic object features from the LiDAR point cloud data, and distinguishing residual components of the static features and dynamic object features based on a spatiotemporal consistency check algorithm, wherein the static features include road surfaces and fixed buildings, and the dynamic object features include pedestrians and vehicles; According to the lateral deviation integral and heading angle offset accumulation of the obstacle avoidance path deviation index, the residual source is determined to be the static environment coordinate system offset or the dynamic object motion prediction error, and the static residual vector and dynamic residual vector are generated by classification; For the static residual vector, the global coordinate system offset is compensated by the least squares optimization algorithm based on the sliding window. For the dynamic residual vector, the trajectory probability distribution parameters of the dynamic obstacle are reversely corrected through the kinematic prediction model. The corrected static and dynamic residual parameters are input into the virtual-reality scene rendering engine to adjust the posture of the virtual accident scene and the motion trajectory of dynamic objects in real time.

8. The AR-integrated remote driving vehicle virtual-reality interaction hardware-in-the-loop testing system according to claim 7 is characterized by: The logic of feeding back the residual compensation to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: Normalizing and weighting the static residual vector and the dynamic residual vector to generate a comprehensive residual compensation weight based on the real-time scene complexity, including traffic flow and lighting conditions; The comprehensive residual compensation weight is input into the dynamic binding algorithm of the spatiotemporal coupling module of the virtual and real scene, and the objective function weight coefficient of the nonlinear optimization algorithm is adjusted to give priority to compensating the high-weight residual components; Based on the distribution characteristics of the dynamic residual compensation amount, the real-time update of the display priority of the virtual and real occlusion areas in the AR interface is triggered. The convergence of the residual feedback loop is analyzed using the Lyapunov stability criterion. If the system state variables diverge, the backup parameter set is switched and the spatiotemporal coupling optimization thread is restarted.

Images