Augmented reality hardware-in-the-loop test system for remote driving vehicles with ar fusion

Abstract

The application discloses a remote driving vehicle virtual-real interaction hardware-in-the-loop test system fusing AR, and particularly relates to the technical field of automatic driving test, and is used for solving the problems of inaccurate coupling of virtual scene and real vehicle behavior and missing AR prompt response evaluation. First, a dynamic obstacle intention driving prediction model is constructed based on multi-modal sensor time series data to generate a trajectory probability distribution and a risk heat map. Then, a dynamic binding algorithm is used to realize the space-time alignment of virtual accident scenes and real environments, and to dynamically adjust the virtual-real occlusion priority of the AR interface. By simulating the abnormal disturbance of the vehicle actuator, the driver's control and gaze response are synchronously collected, the obstacle avoidance path deviation and takeover timeliness parameters are extracted. Finally, based on the path deviation index, virtual-real residual errors are separated and compensated to realize online correction of the virtual scene posture and dynamic trajectory, and a closed-loop optimization mechanism is constructed to improve the accuracy and stability of the test system.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving testing technology, specifically to a hardware-in-the-loop testing system for remote-driving vehicles that integrates AR for virtual-real interaction. Background Technology

[0002] With the rapid development of intelligent connected vehicles and autonomous driving technologies, remote driving, as a key supplementary capability, is gradually becoming an important means to address sudden takeover situations, vehicle-to-infrastructure (V2I) assistance, and low-speed scenario control. In this process, ensuring the reliability and responsiveness of remote driving decision-making systems in complex dynamic environments has become a core issue in industry research and testing. While traditional simulation testing offers repeatability and controllability, it struggles to comprehensively evaluate system performance when faced with uncertainties in the behavior of dynamic traffic participants, human-machine interaction responses in AR interfaces, and actuator interference. Therefore, constructing a virtual-physical fusion hardware-in-the-loop test platform for remote driving and enhancing driver perception and response capabilities through AR technology has become an effective path to improve system stability and safety.

[0003] Existing testing and evaluation methods in remote driving systems primarily rely on one-way simulation processes or data playback mechanisms in closed environments. Based on static obstacle configurations or fixed behavior patterns, these methods cannot predict dynamic obstacle intentions in real-time by fusing multimodal sensor data, resulting in AR prompts lagging behind actual risk evolution. For example, traditional probabilistic graphical models do not consider inter-vehicle interactions, leading to significantly increased prediction errors in complex traffic flows. Mainstream methods rely on two-dimensional coordinate transformation from a single sensor, failing to address the three-dimensional spatial synchronization problem between multiple sensors and the virtual scene, causing AR overlay position jitter. Current testing systems often employ one-way fault injection, failing to synchronously overlay virtual accident scene disturbances, causing driver response assessments to deviate from the real human-machine-environment coupling environment. Furthermore, path deviation calculations rely on a single lateral deviation threshold, ignoring the impact of accumulated heading angle errors on obstacle avoidance safety. Existing virtual-real residual compensation methods do not distinguish between static environment and dynamic object residual sources, leading to conflicting correction strategies and further amplifying the risk of AR display distortion. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a hardware-in-the-loop testing system for remote-driving vehicles that integrates AR, thus solving the problems mentioned in the background.

[0005] To achieve the above objectives, this invention provides the following technical solution: a hardware-in-the-loop testing system for remote-controlled vehicles integrating AR, comprising the following modules: a dynamic obstacle intent parsing module, a virtual-real scene spatiotemporal coupling module, a multi-level obstacle avoidance decision evaluation module, and a virtual-real residual adaptive compensation module; the dynamic obstacle intent parsing module is used to construct an intent-driven behavior prediction model for dynamic obstacles based on time-series data from multimodal sensors, outputting trajectory probability distribution and dynamic risk area parameters; the virtual-real scene spatiotemporal coupling module is used to, based on the dynamic risk area parameters, use a dynamic binding algorithm to connect the virtual accident scene with real sensor data. The system performs spatiotemporal alignment and dynamically adjusts the display priority of virtual and real occlusion areas in the AR interface based on the trajectory probability distribution of dynamic obstacles. The multi-level obstacle avoidance decision evaluation module injects abnormal disturbances of vehicle actuators into the spatiotemporally aligned fusion scene, synchronously collects driver response data to AR guidance, and generates obstacle avoidance path deviation index and takeover timeliness parameters. The virtual-real residual adaptive compensation module separates static environmental features and dynamic object residuals based on the obstacle avoidance path deviation index, dynamically compensates for the position offset of the virtual and real scenes through an online correction algorithm, and feeds the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization.

[0006] Furthermore, the construction logic of the intention-driven behavior prediction model for dynamic obstacles based on the time-series data of multimodal sensors is as follows: The raw data from cameras, millimeter-wave radar, and lidar are time-stamped and transformed into multi-sensor coordinate systems to extract the motion feature vectors of the dynamic obstacles. These motion feature vectors include velocity change trends, heading angle offsets, and acceleration fluctuation patterns. Based on historical trajectory data and real-time motion characteristics, a spatiotemporal behavior probabilistic graphical model of the dynamic obstacles is constructed using a Bayesian network to predict the multimodal trajectory distribution of their future movement direction. This trajectory distribution includes probability weights for straight-ahead, lane-changing, and emergency braking behaviors. Based on the relative motion direction between the dynamic obstacle and the target vehicle and the lane topology, the conflict degree between the trajectory prediction result and the target vehicle's expected path is calculated. Trajectories with a conflict degree exceeding a safety threshold are marked as high-risk events. The actual motion trajectory of the dynamic obstacle collected in real-time is compared with the predicted trajectory. The node weights of the Bayesian network are adjusted in reverse using the gradient descent algorithm to optimize the scene adaptability of the behavior prediction model.

[0007] Further, the output trajectory probability distribution and dynamic risk area parameters include the following steps: mapping the multimodal trajectory prediction results to a two-dimensional rasterized map, generating a dynamic risk heatmap based on probability weights, wherein the raster density of the heatmap is positively correlated with the trajectory conflict probability; extracting the geometric boundaries of high-risk areas based on the density gradient changes of the heatmap, calculating the lateral and longitudinal buffer distances of the safe obstacle avoidance path in conjunction with vehicle dynamics constraints, and dynamically adjusting the transparency and rendering level of AR prompts based on the risk area boundaries and the driver's field of view focus position; performing residual convergence detection on the predicted trajectory and real-time sensor data using an extended Kalman filter algorithm, and triggering the reconstruction of the Bayesian network model and heatmap update if the cumulative residual within the sliding window exceeds the dynamic adjustment threshold.

[0008] Furthermore, based on the dynamic risk area parameters, a dynamic binding algorithm is used to spatiotemporally align the virtual accident scene with the real sensor data, including the following steps: Based on the geometric boundaries and heatmap distribution in the dynamic risk area parameters, key spatiotemporal feature points are extracted from the sensor data, including lane line intersections and traffic sign positions, and multimodal feature matching is performed with the topology of the virtual accident scene. Initial coordinate system alignment is achieved through an iterative nearest-point algorithm. Within a preset time window, the spatiotemporal offset between the real sensor data and the virtual scene is cumulatively calculated, and the pose parameters of the virtual scene are dynamically adjusted using a nonlinear optimization algorithm to compensate for the superposition error caused by communication delay and sensor sampling jitter. The residual between the virtual and real scenes after feature matching is estimated in real time using an extended Kalman filter algorithm. If the residual exceeds the dynamic adjustment threshold, local coordinate system fine-tuning of the virtual scene is triggered.

[0009] Furthermore, the display priority of virtual and real occlusion areas in the AR interface is dynamically adjusted based on the trajectory probability distribution of dynamic obstacles. This includes the following steps: dividing the AR prompt area into multiple rendering layers according to the high-probability trajectory branches of the trajectory probability distribution, and dynamically assigning transparency levels according to probability values; detecting the spatial overlap area between real obstacles and virtual roadblocks in real time, calculating the occlusion priority based on the trajectory probability weights, obtaining the driver's field of view focus coordinates, and dynamically adjusting the rendering position and size of the AR prompt information; when the trajectory probability distribution changes abruptly due to model reconstruction, immediately interrupting the current rendering thread and redistributing the occlusion priority according to the updated probability weights.

[0010] Furthermore, abnormal disturbances of vehicle actuators are injected into the spatiotemporally aligned fusion scenario, and driver response data to AR guidance is collected synchronously to generate obstacle avoidance path deviation index and takeover timeliness parameters. This includes the following steps: simulating abnormal operating conditions of vehicle actuators through a hardware-in-the-loop interface, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, while simultaneously superimposing virtual accident scenario disturbances; recording the driver's control signals, eye-tracking focus trajectory, AR prompt gaze duration, and vehicle status data, including lateral acceleration and yaw angle, in real time; calculating the lateral deviation integral and yaw angle offset of the vehicle's actual trajectory based on the positioning data and the expected obstacle avoidance path; defining the interval from the prompt triggering time to the driver's first effective control response as the takeover delay, and calculating the takeover success rate by weighting the path deviation index.

[0011] Furthermore, based on the obstacle avoidance path deviation index, the static environmental features and dynamic object residuals are separated, and the positional offset of the virtual and real scenes is dynamically compensated through an online correction algorithm. This includes the following steps: extracting static environmental features and dynamic object features from LiDAR point cloud data; distinguishing the residual components of static features and dynamic object features using a spatiotemporal consistency verification algorithm; the static features include road surfaces and fixed buildings, and the dynamic object features include pedestrians and vehicles; determining whether the residual source is static environmental coordinate system offset or dynamic object motion prediction error based on the lateral deviation integral and heading angle offset accumulation of the obstacle avoidance path deviation index; classifying and generating static residual vectors and dynamic residual vectors; compensating for the global coordinate system offset using a sliding window least squares optimization algorithm for the static residual vector; and correcting the trajectory probability distribution parameters of dynamic obstacles using a kinematic prediction model; and inputting the corrected static and dynamic residual parameters into the virtual and real scene rendering engine to adjust the pose of the virtual accident scene and the motion trajectory of dynamic objects in real time.

[0012] Furthermore, the logic for feeding the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: The static and dynamic residual vectors are normalized and weighted, and a comprehensive residual compensation weight is generated by combining the real-time scene complexity, which includes traffic flow and lighting conditions; the comprehensive residual compensation weight is input into the dynamic binding algorithm of the virtual-real scene spatiotemporal coupling module, adjusting the objective function weight coefficients of the nonlinear optimization algorithm to prioritize compensation of high-weight residual components; based on the distribution characteristics of the dynamic residual compensation amount, the real-time update of the display priority of virtual-real occlusion areas in the AR interface is triggered; the convergence of the residual feedback loop is analyzed using the Lyapunov stability criterion; if the system state variables diverge, the system switches to the backup parameter set and restarts the spatiotemporal coupling optimization thread.

[0013] The present invention has the following beneficial effects:

[0014] (1) The hardware-in-the-loop test system for virtual and real interaction of remote driving vehicles integrating AR significantly improves the accuracy of dynamic obstacle trajectory prediction and multimodal scene adaptability by extracting the temporal features of multimodal sensor data and the behavior prediction model driven by Bayesian network through the dynamic obstacle intent parsing module. At the same time, based on the conflict degree calculation and high-risk event marking, the system realizes the dynamic allocation of AR risk prompt weight driven by intent, effectively reducing the risk of decision lag caused by sudden obstacle behavior of the driver. The virtual and real scene spatiotemporal coupling module solves the problem of spatiotemporal reference deviation between real sensor data and virtual scene through iterative nearest point algorithm initial alignment, sliding window nonlinear optimization and extended Kalman filter residual correction, ensuring that the virtual and real superposition position error is less than the industry standard requirement. It also prioritizes the rendering of real obstacle information through dynamic occlusion priority adjustment mechanism to avoid visual confusion caused by virtual and real occlusion conflict in the interface.

[0015] (2) The hardware-in-the-loop test system for remote driving vehicles integrating AR and virtual reality interaction, the multi-level obstacle avoidance decision evaluation module injects abnormal disturbances of vehicle actuators through the hardware-in-the-loop interface and simultaneously superimposes virtual accident scene disturbances to build a high-fidelity human-machine collaborative test environment. The path deviation index generated by combining the lateral deviation integral and the heading angle accumulation is used to quantitatively evaluate the driver's response accuracy to AR guidance. The virtual-real residual adaptive compensation module realizes dynamic closed-loop compensation of virtual-real scene position offset through static and dynamic feature residual separation, sliding window least squares optimization and kinematic model inverse correction. Combined with the feedback loop optimization mechanism of Lyapunov stability criterion, the system's robustness and interactive reliability under complex disturbances can be directly applied to the road test certification of intelligent connected vehicles.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0017] Figure 1 This is a flowchart of the hardware-in-the-loop test system for remote-driving vehicles that integrates AR, as described in this invention. Detailed Implementation

[0018] This application's embodiments address the core issues of insufficient real-time coordination between dynamic obstacle intent prediction and AR prompts, low spatiotemporal alignment accuracy of virtual and real scenes, lack of human-machine-environment coupling verification in hardware-in-the-loop testing, and lack of classification compensation in residual correction mechanisms through a hardware-in-the-loop testing system that integrates AR for remote driving vehicles.

[0019] The overall concept of the solution in this application embodiment is as follows:

[0020] A predictive model for intention-driven behavior of dynamic obstacles is constructed using multimodal sensor data (Bayesian network + gradient descent optimization), outputting a high-confidence trajectory probability distribution and dynamic risk parameters.

[0021] A high-precision spatiotemporal alignment of virtual and real scenes is achieved based on a dynamic binding algorithm (iterative nearest point initial alignment + sliding window nonlinear optimization + extended Kalman filter residual correction), and the occlusion priority of the AR interface is dynamically adjusted by combining trajectory probability.

[0022] In hardware-in-the-loop testing, vehicle actuator anomalies and virtual accident scenario disturbances are injected synchronously, and driver response behavior is quantified by the path deviation integral and the heading angle accumulation.

[0023] Finally, based on static / dynamic residual separation and classification compensation (sliding window least squares + kinematic model inverse correction), and combined with the Lyapunov stability criterion, closed-loop optimization is achieved to ensure the reliability of AR interaction and the authenticity of testing in complex scenarios.

[0024] Please see Figure 1 This invention provides a technical solution: a hardware-in-the-loop testing system for remote-controlled vehicles integrating AR, comprising the following modules: a dynamic obstacle intent parsing module, a virtual-real scene spatiotemporal coupling module, a multi-level obstacle avoidance decision evaluation module, and a virtual-real residual adaptive compensation module. The dynamic obstacle intent parsing module is used to construct a dynamic obstacle intent-driven behavior prediction model based on time-series data from multimodal sensors, outputting trajectory probability distribution and dynamic risk area parameters. The virtual-real scene spatiotemporal coupling module is used to dynamically bind virtual accident scenes with real sensor data using a dynamic binding algorithm based on the dynamic risk area parameters. The system performs spatial alignment and dynamically adjusts the display priority of virtual and real occlusion areas in the AR interface based on the trajectory probability distribution of dynamic obstacles. The multi-level obstacle avoidance decision evaluation module injects abnormal disturbances of vehicle actuators into the spatiotemporally aligned fusion scene, synchronously collects driver response data to AR guidance, and generates obstacle avoidance path deviation index and takeover timeliness parameters. The virtual-real residual adaptive compensation module separates static environmental features and dynamic object residuals based on the obstacle avoidance path deviation index, dynamically compensates for the position offset of virtual and real scenes through online correction algorithms, and feeds the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization.

[0025] In this implementation scheme, the dynamic obstacle intent parsing module: This module constructs an obstacle behavior prediction model that integrates historical behavior and real-time motion features based on continuous time-series data collected by multimodal sensors (including cameras, millimeter-wave radar, and lidar). It is used to identify and predict the intent and future trajectories of dynamic participants (such as pedestrians, vehicles, and cyclists) in traffic scenarios, and outputs trajectory distribution information containing probability weights and a description of dynamic risk areas. This module provides real-time risk assessment for subsequent AR rendering priority control and obstacle avoidance decisions. The intent-driven behavior prediction model: This is a behavior modeling method combining a Bayesian inference framework and temporal pattern recognition. Input variables include the obstacle's speed, acceleration, heading angle, and historical trajectory. The posterior probability distribution of various behavior patterns (such as going straight, changing lanes, and decelerating) is calculated using prior probabilities and likelihood functions obtained through training. Trajectory probability distribution: Based on multiple possible trajectories output by the intent model and their corresponding probabilities, for example, the prediction result for a certain obstacle is: 60% for changing lanes to the left, 30% for going straight, and 10% for changing lanes to the right. This distribution is used to assess potential conflict areas and expected dynamic change trends. Dynamic Risk Area Parameters: These include the spatial boundaries (e.g., rectangular, elliptical, or other polygonal envelopes) of the intersection area between the predicted trajectory and the vehicle's path, and the risk level of that area (usually divided into high, medium, and low levels). These parameters define the priority and scope of AR prompt triggering. Virtual-Real Scene Spatiotemporal Coupling Module: This module achieves high-precision spatiotemporal fusion of virtual traffic event scenarios (e.g., virtual landslides, simulated construction obstacles) with real road environment sensor data. Through coordinate alignment and delay compensation algorithms, it ensures seamless overlay of virtual elements with real elements in the AR interface, eliminating visual errors such as misalignment and drift. Simultaneously, it adjusts the display order of virtual and real objects based on obstacle risk levels to avoid misleading user attention. Dynamic Binding Algorithm: This algorithm enables dynamic spatial and temporal matching of virtual and real information, including the following key processes: Iterative Closest Point (ICP) Initial Alignment: By extracting key geometric feature points of the virtual and real environments (e.g., road boundary points, lane line intersections), iteratively optimizing the coordinate transformation matrix achieves coarse-precision registration. Sliding Window Nonlinear Optimization: Considering factors such as sensor sampling delay and frame rate differences, it performs nonlinear optimization adjustments based on multi-frame historical data windows to achieve time synchronization. Extended Kalman Filter Residual Correction: Combining dynamic system state prediction, coupling errors are filtered and updated to eliminate accumulated errors and maintain local registration accuracy. Dynamic Display Priority Adjustment: When virtual scene elements (such as virtual lane closures) overlap with real objects (such as real pedestrians), the system determines the rendering priority based on the obstacle's risk level and trajectory prediction weight, ensuring that high-risk real targets are displayed and highlighted first. Multi-level Obstacle Avoidance Decision Evaluation Module: In the coupled virtual-real fusion scene, this module simulates actuator-level fault disturbances (such as steering wheel failure or braking lag) and guides the user to perform obstacle avoidance operations in conjunction with specific AR prompts.Simultaneously, driver takeover behavior data is collected to evaluate their response efficiency to AR prompts, path execution accuracy, and fault tolerance, forming a multi-dimensional obstacle avoidance decision-making capability evaluation index system. Hardware-in-the-Loop (HIL) interface: This refers to the system injecting fault signals from the virtual environment into the vehicle actuator control loop in real time through a physical simulation platform (e.g., dSPACE, NI platform) or force feedback device, achieving the fusion of real hardware and virtual simulation, improving the realism and accuracy of the test. Path deviation index: Used to quantify the degree of deviation between the vehicle's actual obstacle avoidance path and the expected path. Specifically, it includes: Lateral deviation integral: the integral of the lateral distance between the vehicle's center of gravity and the centerline of the planned path; Heading angle offset cumulative amount: the integral of the deviation between the vehicle's current heading angle and the tangent direction of the target path, used to reflect directional control error. Takeover timeliness parameter: Measures the time interval from the system's AR prompt trigger to the driver's first active and effective operation (e.g., steering, braking). Combined with dynamic weighting of path deviation, the "effective takeover success rate" can be obtained. The Virtual-Real Residual Adaptive Compensation Module: This module performs fine-grained classification of positional errors generated in the fusion environment, handling the virtual-real residuals between static scenes (such as roads and buildings) and dynamic objects (such as pedestrians and vehicles). Through residual separation, trajectory reverse modeling, and online compensation, it achieves adaptive closed-loop correction of the system, further improving the accuracy and stability of scene fusion. Static Feature Residuals: Systematic deviations caused by LiDAR point cloud calibration errors or map coordinate system drift, such as misalignment between AR virtual lane lines and real road markings. Dynamic Object Residuals: Deviations in dynamic target position prediction caused by factors such as behavior prediction model errors and sudden environmental events, such as the deviation between predicted pedestrian walking paths and their actual behavior. Online Correction Algorithm: Sliding Window Least Squares Optimization: For static scene deviations, multiple consecutive time frames are selected to construct a least squares problem, fitting the optimal coordinate mapping relationship between the real and virtual scenes. Kinematic Model Reverse Correction: Based on the dynamics or Bayesian inference model of the predicted trajectory, the model parameters are optimized by reverse-engineering the actual observed trajectory, achieving closed-loop correction of dynamic residuals. Lyapunov stability criterion: The monotonicity of the Lyapunov function in control theory is used as the verification standard for system convergence to ensure that the compensated feedback system will not oscillate, diverge or other unstable behaviors in closed-loop operation.

[0026] Specifically, the construction logic of the intention-driven behavior prediction model for dynamic obstacles based on time-series data from multimodal sensors is as follows: The raw data from cameras, millimeter-wave radar, and lidar are time-stamped and transformed into multi-sensor coordinate systems to extract motion feature vectors of the dynamic obstacles. These vectors include velocity change trends, heading angle offsets, and acceleration fluctuation patterns. Based on historical trajectory data and real-time motion features, a spatiotemporal behavior probabilistic graphical model of the dynamic obstacles is constructed using a Bayesian network to predict the multimodal trajectory distribution of their future movement direction. The trajectory distribution includes probability weights for straight-ahead, lane-changing, and emergency braking behaviors. Based on the relative motion direction between the dynamic obstacle and the target vehicle, and the lane topology, the conflict degree between the trajectory prediction result and the target vehicle's expected path is calculated. Trajectories with conflict degrees exceeding a safety threshold are marked as high-risk events. The actual motion trajectory of the dynamic obstacle collected in real-time is compared with the predicted trajectory. The node weights of the Bayesian network are adjusted in reverse using the gradient descent algorithm to optimize the scene adaptability of the behavior prediction model.

[0027] In this implementation plan, the basic technical process for multimodal sensor data preprocessing and unified modeling includes: time-stamping the raw data from cameras, millimeter-wave radar, and lidar; using extrinsic parameter calibration matrices to perform coordinate system transformations between sensors, uniformly mapping them to the vehicle coordinate system or world coordinate system; interpolating or aligning multi-source data to ensure all sensor observations fall at the same time; and performing coordinate system transformation: using a rotation matrix R and a translation vector t to achieve a rigid transformation from any sensor coordinate system Fi to the target coordinate system F0. Where: Pi: 3D points acquired by sensor i; P0: point cloud data in a unified coordinate system. Construction of dynamic obstacle motion feature vector: Feature extraction content: Extract the following real-time motion features from the fused target tracking data to construct the input vector required for behavior prediction. : Instantaneous velocity; Acceleration; : heading angle; : Heading offset; The standard deviation of acceleration variation (indicating the degree of behavioral instability). Eigenvector representation: The modeling logic for constructing a spatiotemporal behavioral probabilistic graphical model using Bayesian networks is as follows: Obstacle movement behavior is modeled as a set of hidden state nodes (e.g., "go straight," "change lanes left," "sudden braking," etc.), and a Bayesian network is used to represent its state transition relationships and observation probabilities, generating a spatiotemporal probability graph. The mathematical model structure is as follows: Let the set of states be... These correspond to different behavior patterns (such as going straight, changing lanes, and sudden braking). The network structure includes: prior probabilities. conditional probability Given the feature vectors, the predicted probability of each behavior is inferred through maximum a posteriori (MAP). ;in: The prior frequency of this behavior in the historical trajectory; : Likelihood of feature distributions for each behavior (can be modeled using Gaussian distribution); The final probability output used to determine behavioral tendencies. Multimodal trajectory prediction and risk assessment technology content: based on state probability. For each type of behavior Generate several future time steps respectively movement trajectory Each trajectory is assigned a corresponding probability weight. This constitutes the trajectory probability distribution set: Risk assessment: Define the desired path of the autonomous vehicle as... , and obstacle trajectory Perform conflict analysis; calculate the spatiotemporal overlap of the intersection intervals. And introduce a trajectory conflict index : ;in: : represents the Euclidean distance between trajectories; : This is the distance tolerance parameter (usually set to half the vehicle width); if (Any event meeting a set risk threshold) is marked as a high-risk trajectory event. Model adaptive optimization (residual inverse correction) real-time correction: collects the actual trajectories of obstacles. ; and predicted trajectory Compare and define the trajectory residual: Construct the residual loss function And the node parameters of the Bayesian network (Including conditional probability tables, prior probabilities, etc.) Perform gradient descent updates: ;in: Learning rate; parameter updates enable the model to gradually adapt to the behavioral characteristics of the current scene, thereby improving prediction accuracy.

[0028] Specifically, the output trajectory probability distribution and dynamic risk area parameters include the following steps: mapping the multimodal trajectory prediction results to a two-dimensional rasterized map, generating a dynamic risk heatmap based on probability weights, where the raster density of the heatmap is positively correlated with the trajectory conflict probability; extracting the geometric boundaries of high-risk areas based on the density gradient changes in the heatmap, calculating the lateral and longitudinal buffer distances of the safe obstacle avoidance path in conjunction with vehicle dynamics constraints, and dynamically adjusting the transparency and rendering level of AR prompts based on the risk area boundaries and the driver's field of view focus position; and performing residual convergence detection on the predicted trajectory and real-time sensor data using an extended Kalman filter algorithm. If the cumulative residual within the sliding window exceeds the dynamic adjustment threshold, the reconstruction of the Bayesian network model and the update of the heatmap are triggered.

[0029] In this implementation scheme, the principle of mapping trajectory probability distribution to a two-dimensional raster map is as follows: Multimodal predicted trajectory results are mapped to a fixed-resolution two-dimensional planar map, establishing a spatial distribution representation based on trajectory probability. The raster mapping method involves dividing the two-dimensional map into a set of raster cells. ,in Represents the horizontal and vertical raster indexes; for each predicted trajectory (Including time series point sets) and their weights Count the number of grid cells it passes through. The frequency; the risk value of each grid is summed using probability weights: ;in: : grid Heatmap risk values; Trajectory The probability weights; : Indicator function, 1 when the trajectory crosses the grid, 0 otherwise; Total number of trajectory samples. Note: In the heatmap... A larger value indicates a higher likelihood of trajectory collision at that location. The principle of dynamic risk area boundary extraction technology based on gradient field: Based on the gradient field changes in the heatmap, high-risk dense areas are identified, and their geometric boundaries are extracted for subsequent obstacle avoidance judgment. Mathematical processing: Constructing the heatmap gradient field: Determine the ascending boundary of the risk region using the gradient magnitude: Set a threshold Extract to satisfy The continuous closed area is used as the geometric boundary of the high-risk area, denoted as: The principle of obstacle avoidance buffer distance calculation based on vehicle dynamics is explained as follows: Combining the target vehicle's maximum steering capability, braking performance, and dynamic response delay, the lateral and longitudinal buffer distances required for the vehicle to safely pass through a risk area are calculated. Mathematical model: Assume the vehicle's maximum safe lateral acceleration is... The current speed is The delayed reaction time is Then the lateral buffer distance Longitudinal buffer distance Expressed as: ;in: Maximum longitudinal deceleration capability; This includes the total delay in driver recognition and operational response (based on historical takeover parameter statistics). Step 4: Dynamically adjust the AR rendering technology mechanism based on the driver's field of view focus position: Let the center position of the driver's current gaze area be... Obtained through eye-tracking devices; risk areas calculated. Center of gravity ;Calculate its field of view deviation angle : ;in: Camera focal length, used for viewpoint normalization; Smaller values ​​indicate closer proximity to the driver's gaze center. AR rendering adjustment logic: AR cue transparency. Rendering priority according to Dynamic settings: ;in: : Empirical parameter tuning coefficients; making cues closer to the driver's line of sight clearer and given higher priority. The principle of the residual convergence detection mechanism based on extended Kalman filtering: continuously monitoring the residual deviation between the predicted trajectory and the sensor-measured trajectory to evaluate the stability of the prediction system. Key formula: The state-predicted trajectory is... The actual trajectory measured by the sensor is Define the filter residual: ; Sliding window residual accumulation (in (for window length) ;like This triggers the trajectory model reconstruction and heatmap refresh process. Model adaptive feedback: Clears the Bayesian model cache and retrains the trajectory probability distribution. Update the grid heatmap The location of high-risk areas has been revised.

[0030] Specifically, based on dynamic risk area parameters, a dynamic binding algorithm is used to spatiotemporally align the virtual accident scene with real sensor data, including the following steps: Based on the geometric boundaries and heatmap distribution in the dynamic risk area parameters, key spatiotemporal feature points are extracted from the sensor data, including lane line intersections and traffic sign locations. These points are then matched with the topology of the virtual accident scene using multimodal feature matching, and initial coordinate system alignment is achieved through an iterative nearest-point algorithm. Within a preset time window, the spatiotemporal offset between the real sensor data and the virtual scene is cumulatively calculated, and the pose parameters of the virtual scene are dynamically adjusted using a nonlinear optimization algorithm to compensate for superposition errors caused by communication delays and sensor sampling jitter. Finally, the residual between the virtual and real scenes after feature matching is estimated in real time using an extended Kalman filter algorithm. If the residual exceeds the dynamic adjustment threshold, local coordinate system fine-tuning of the virtual scene is triggered.

[0031] In this implementation scheme, multimodal feature extraction and initial registration are performed by extracting key topological feature points (lane intersections, traffic signs, and corner buildings) from both the real and virtual scenes, and then aligning the coordinate systems initially through point cloud registration. The operation process involves selecting feature point sets such as lane intersections, intersection signs, and cross-sectional contours from high-density areas of the heatmap; assuming the real scene point set is: The corresponding point set in the virtual scene is: The rigid body transformation matrix is ​​obtained using the Iterative Closest Point (ICP) algorithm. This minimizes the matching error. ;in: Initial rotation matrix; Initial translation vector; Euclidean norm. Background of nonlinear optimization dynamic compensation communication and sampling offset technology: After initial registration, communication latency (such as AR head-mounted display latency) and sensor sampling jitter can cause spatiotemporal drift, requiring online correction of the virtual scene pose. Algorithm processing: within the time window... Within, record the drift error vector after registration for each frame: ;in: : Number of frames within the sliding window; The pose of the virtual scene in the current frame; The current frame registration residual. A residual cumulative cost function is constructed for nonlinear optimization: ;in: : Rotation fine-tuning vector; Translation fine-tuning vector; Jacobian matrix of the residuals with respect to rotation and translation. Solve for the optimal compensation amount. Used for real-time updates of virtual scene pose: .in, Historical translation vector Adjusted rotation matrix Historical rotation matrix : Adjusted translation vector. Solution method for optimal solution: Residual error objective function setting: Let the measured sensor point cloud / image features at the current time be... Virtual scene projection features are Then, construct the optimization objective function: ;in: In the virtual scene Three-dimensional feature points; Projection function, simulating camera / sensor imaging model; Euclidean distance norm; Number of features involved in alignment. Extended Kalman Filter Residual Estimation and Triggering Mechanism Technical Objective: To estimate the registration residual in real time and trigger a local coordinate system readjustment process when the accumulated error exceeds a set threshold, ensuring stable alignment of the virtual scene. State Definition: Define the system state vector: ; represents the pose state (rotation and translation parameters) of the virtual scene at time k. Filter residual calculation: The observation model is: ;in The projection function maps the scene pose to the sensor coordinate system; the residual is defined as: The extended Kalman filter state is updated as follows: ;in Kalman gain. Sliding residual statistics: within a window. Inner cumulative residual norm squared: ;like If this occurs, the following operations are triggered: local coordinate system offset correction; restarting the nonlinear optimization process; updating the heatmap spatial weights to adapt to the drift correction.

[0032] Specifically, the display priority of virtual and real occlusion areas in the AR interface is dynamically adjusted based on the trajectory probability distribution of dynamic obstacles. This includes the following steps: dividing the AR prompt area into multiple rendering layers according to the high-probability trajectory branches of the trajectory probability distribution, and dynamically assigning transparency levels according to probability values; detecting the spatial overlap area between real obstacles and virtual roadblocks in real time, calculating the occlusion priority based on the trajectory probability weights, obtaining the driver's field of view focus coordinates, and dynamically adjusting the rendering position and size of the AR prompt information; when the trajectory probability distribution changes abruptly due to model reconstruction, immediately interrupting the current rendering thread and redistributing the occlusion priority according to the updated probability weights.

[0033] In this implementation scheme, a multi-layered AR rendering hierarchy is constructed based on trajectory probability distribution: According to the multimodal trajectory distribution output by the trajectory prediction model, trajectory branches with higher probability values ​​are extracted, and a layered display structure for the corresponding AR prompt area is constructed. Transparency is then mapped using probability weighting. Instructions: Assume the system outputs the following predicted trajectory set: Among them, the first Trajectory The corresponding probability of occurrence is ,satisfy: Based on trajectory probability The size of the trajectory will be assigned to different rendering levels: Let the set of rendering levels be... ,in Indicates the first Render layers; construct probability layer mapping function: ;in: The hierarchical boundary threshold satisfies ; each floor according to Dynamically calculate transparency : ;in: Transparency controls upper and lower limits; The larger the size, the less transparent the AR prompt (higher priority). Real-time assessment of virtual-real occlusion conflicts and adjustment of display layout: Geometric calculations detect the spatial overlap between virtual prompts and real obstacles, combined with trajectory probability weighting to calculate occlusion priority, and dynamically adjust the AR prompt display position and size based on the driver's field of view focus. Mathematical modeling: Defining virtual-real occlusion conflict areas. : ;in: : The mapping of the virtual prompt area in screen projection coordinates; The projection mapping of the predicted obstacle position onto the current frame. Occlusion weights are calculated by combining the trajectory probability. : ;in: All and the prompt area A set of overlapping trajectories; Occlusion weight factor Controlling weight sensitivity; The function represents the area of ​​the projected region. The driver's field of view focal position is... Calculate the preferred offset direction of the prompt message: ;in: : Displays the current center location; : Two-dimensional Euclidean distance; : Fine-tune the position direction vector. Final update of render position and size: Position fine-tuning: Size adjustment: ;in: Original display size; : Adjustment coefficient; : Maximum occlusion conflict in the current frame. Explanation of rendering thread interruption and occlusion redistribution logic during model reconstruction: When the trajectory probability distribution changes due to Bayesian behavioral model updates or residual mutations, causing significant fluctuations in trajectory probabilities (i.e., a set of trajectories experiences a probability increase...). Exceeding the dynamic reconstruction threshold This requires immediately interrupting the existing AR rendering thread and recalculating the occlusion layering logic. The interruption trigger condition expression is as follows: Let the probabilities of the new and old trajectories be respectively... and ,but: Interruption handling includes: forcibly terminating the current rendering thread; clearing the current frame's layer display stack; and reallocating the rendering layer. Transparency of each layer Occlusion priority ; Prompt for location and size .

[0034] Specifically, abnormal disturbances in vehicle actuators are injected into the spatiotemporally aligned fusion scene, and driver response data to AR guidance is collected synchronously to generate obstacle avoidance path deviation index and takeover timeliness parameters. This includes the following steps: simulating abnormal operating conditions of vehicle actuators through a hardware-in-the-loop interface, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, while simultaneously superimposing virtual accident scene disturbances; recording the driver's control signals, eye-tracking focus trajectory, AR prompt gaze duration, and vehicle status data, including lateral acceleration and yaw angle, in real time; calculating the lateral deviation integral and yaw angle offset of the vehicle's actual trajectory based on the positioning data and the expected obstacle avoidance path; defining the interval from the AR prompt triggering time to the driver's first effective control response as the takeover delay, and calculating the takeover success rate by weighting it in conjunction with the path deviation index.

[0035] In this implementation plan, actuator abnormal disturbance injection and virtual-real fusion are collected simultaneously: Actuator abnormal disturbance injection is simulated through a hardware-in-the-loop system to simulate the following typical abnormal operating conditions of vehicle actuators: sudden change in steering system torque: an undesired instantaneous disturbance torque is injected into the vehicle's steer-by-wire system. This causes actual steering angle deviation; braking system response delay: manually set braking delay period. Compared with normal operating conditions; throttle system opening drift: introducing throttle control error. This causes the actual acceleration behavior to deviate from the expected behavior; and simultaneously overlays virtual accident scenario disturbances (such as sudden lane changes, static obstacles, and blind spot intrusions) to increase test complexity and environmental realism. Data collection items and their mapping relationships are collected synchronously by the system in real time:

[0036]

[0037] The obstacle avoidance path deviation index is constructed by combining it with high-precision maps to predict the obstacle avoidance trajectory. With respect to the actual trajectory of the vehicle The following two indicators are constructed: lateral deviation integral indicator. Used to measure the lateral stability of a vehicle relative to a desired trajectory: ; Vehicle's path arc length Lateral offset distance at the location; , : Represents the starting and ending arc lengths of the path, respectively; A larger value indicates a greater lateral control error, suggesting poorer obstacle avoidance accuracy. (Cumulative heading angle offset) Used to describe the cumulative error that describes the consistency between directional control and the trajectory target: ; : The yaw angle of the vehicle during actual driving; : The heading angle at the time point corresponding to the expected trajectory; The time interval during which the disturbance is injected into the vehicle until it stabilizes; A larger value indicates poorer vehicle steering control precision, suggesting the driver failed to effectively correct the trajectory. The assessment includes the timeliness of the takeover response and the takeover delay time. Defined as: ; AR prompts are generated and enter the driver's field of vision at the moment; The moment when the driver first triggers a valid operation (such as a noticeable steering wheel angle, braking, or accelerator input). A smaller value indicates a faster takeover response, improving scenario security. Takeover success rate metric construction: Takeover success rate. By combining path deviation and response delay, a weighted composite evaluation function is constructed: ; It is an empirical weighting coefficient, adjusted according to actual test data; the index range is [0,1], the closer the value is to 1, the faster and more accurately the driver can complete the risk avoidance take-off; it has differentiability and can be used as a gradient feedback source for subsequent driving strategy optimization models.

[0038] Specifically, based on the obstacle avoidance path deviation index, the static environmental features and dynamic object residuals are separated. An online correction algorithm dynamically compensates for the positional offset of the virtual and real scenes, including the following steps: Extracting static environmental features and dynamic object features from LiDAR point cloud data; differentiating the residual components of static and dynamic features using a spatiotemporal consistency verification algorithm; static features include road surfaces and fixed buildings, while dynamic object features include pedestrians and vehicles; determining whether the residual source is static environmental coordinate system offset or dynamic object motion prediction error based on the lateral deviation integral and heading angle offset accumulation of the obstacle avoidance path deviation index, and generating static and dynamic residual vectors; for the static residual vector, compensating for global coordinate system offset using a sliding window least squares optimization algorithm; for the dynamic residual vector, correcting the trajectory probability distribution parameters of dynamic obstacles using a kinematic prediction model; and inputting the corrected static and dynamic residual parameters into the virtual-real scene rendering engine to adjust the pose of the virtual accident scene and the motion trajectory of dynamic objects in real time.

[0039] In this implementation scheme, lidar point cloud feature extraction and residual separation are performed. Feature point extraction extracts structural features from lidar inter-frame point cloud data: noise is reduced using voxel grid filtering; edge / plane features are identified using the rate of curvature change, and a feature set is constructed. ;in Indicates the current frame number. For the first The three-dimensional coordinates of each point. Spatiotemporal consistency verification and residual separation. Let the feature points of the previous frame be... The current frame point is Then the point-to-point residual is: ; Inter-frame estimated pose transformation (4×4 homogeneous matrix); if If the condition is met, it is considered a static residual point; otherwise, it is considered a dynamic object point or a sensing error point. The set is divided as follows: Static residual set: Dynamic residual set: The residual source classification and determination based on the path deviation index, combined with the existing obstacle avoidance path lateral deviation integral index, is used. Cumulative error with heading angle Construct a logic function to determine the source of residuals: ; Error tolerance threshold; if the deviation is small, it is considered to be the scene baseline error (static residual); if the deviation is significant, it is determined to be the error caused by dynamic target prediction error. Two types of residual vectors are generated: static residual vector: Dynamic residual vector: Static residual compensation: Sliding window least squares optimization is used to eliminate static alignment errors between virtual and real scenes. A position compensation objective function within the sliding window is constructed as follows: ; The static coordinate translation amount to be optimized; : Sliding window frame length; :No. The number of residual points in the frame; this optimization can be achieved in real time using the Gauss-Newton method or QR decomposition to obtain the position offset compensation vector. Ultimately, the global coordinates of the virtual scene were corrected to: Dynamic residual correction: Trajectory probability distribution compensation corrects the dynamic obstacle prediction offset by adjusting the original predicted trajectory probability distribution parameters. Residual feedback correction is performed. Model representation: Suppose the multimodal distribution of the trajectory of a dynamic target at the current moment is as follows: ;in: :No. The probability weights of each trajectory branch; Mean position of trajectory (two-dimensional coordinates); : Covariance matrix; Correction method: based on residual vector Corrected trajectory distribution: ; The step size coefficient represents the degree of correction to the current residual; this method is analogous to mean bias compensation for Bayesian prediction models, improving the adaptability and online consistency of dynamic models. Finally, the corrected residuals are input into the rendering engine to perform two types of corrections: global scene pose update (for static residuals). Virtual dynamic object trajectory update (for dynamic residuals): The rendering engine uses this information to redraw the environment framework and obstacle movement animations of the virtual accident scene in real time, maintaining dynamic alignment with the real scene.

[0040] Specifically, the logic for feeding the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: The static and dynamic residual vectors are normalized and weighted, and a comprehensive residual compensation weight is generated by combining the real-time scene complexity, which includes traffic flow and lighting conditions. The comprehensive residual compensation weight is input into the dynamic binding algorithm of the virtual-real scene spatiotemporal coupling module, and the weight coefficients of the objective function of the nonlinear optimization algorithm are adjusted to prioritize the compensation of high-weight residual components. Based on the distribution characteristics of the dynamic residual compensation amount, the real-time update of the display priority of virtual-real occlusion areas in the AR interface is triggered. The convergence of the residual feedback loop is analyzed using the Lyapunov stability criterion. If the system state variables diverge, the system switches to the backup parameter set and restarts the spatiotemporal coupling optimization thread.

[0041] In this implementation scheme, residual normalization weighting is fused with scene complexity, resulting in static and dynamic residual vectors. The static residual vector is: Dynamic residual vector: Normalization is performed on the residuals separately: ; Small constants are used to prevent division by zero; residual signals with consistent direction but normalized magnitude are obtained. Traffic flow is introduced into the construction of the scene complexity factor. and mean light intensity (Normalization processing) Define the scene complexity coefficient. : ; Weighting coefficients, satisfying ; Traffic density; Ambient lighting; A larger value indicates a more complex scenario. The weighting factor is defined for generating the comprehensive residual compensation weights. Constructing the weighted residual compensation amount: ; Upsizing (padding with zeros) the dynamic residual vector to match the dimension of the three-dimensional static residual; : Used for unified residual input in the coupling module. The objective function weight update in the dynamic binding algorithm is performed in the virtual real-time space-time binding module. The original nonlinear optimization objective function is: ; Virtual scene pose parameters (including rotation and translation); : Feature point pair weight coefficients; , Sensor and virtual feature point pairs; solved using Ceres. Objective function weights updated: based on the magnitude of the weighted residual compensation. Dynamically adjust the weights of various point pairs: ; Initial weights; : Residual amplification factor related to the point category; for high residual regions, binding optimization will give higher priority to matching accuracy. Priority update of virtual and real occlusion regions guides the adjustment of AR interface display hierarchy based on dynamic residual distribution: Constructing an occlusion priority function: ; : The angle between the residual direction and the driver's field of view; Weighting coefficient; Used to control rendering transparency and Zbuffer priority; the display module depends on... Update the display status of each occluded area to enhance the prompting effect for dynamic objects with large residuals. Convergence analysis of the residual feedback closed-loop system (Lyapunov criterion): Define system state variables. To represent the current virtual-real alignment error state, construct the Lyapunov function: ; : Positive definite matrix; if it satisfies The system is closed-loop stable. If at any given moment: This indicates a system divergence, immediately triggering the emergency procedure: switch to the backup parameter set. ; Reset the objective function weights and residual state; Restart the dynamic binding optimization thread.

[0042] In summary, this application has at least the following effects:

[0043] This AR-integrated hardware-in-the-loop testing system for remote-driving vehicles effectively suppresses virtual scene pose errors caused by sensor drift and communication delays by introducing dynamic residual separation and weighted compensation mechanisms, combined with nonlinear optimization algorithms and feature-level alignment methods. This achieves high-precision coupling between virtual information and the real environment. A dynamic risk heatmap is constructed based on Bayesian behavior prediction and probability trajectory distribution, and combined with a dynamic rendering mechanism for the AR interface, enabling drivers to perceive potential collision risk areas and high-priority obstacle avoidance prompts in real time, enhancing driving safety and proactive decision-making capabilities. By injecting abnormal operating conditions of vehicle actuators and collecting driver interaction data, the system can quantify the effectiveness and timeliness of driver responses to AR prompts, forming objective and traceable safety performance evaluation indicators. Combining the driver's field of view focus, the probability of obstacle occlusion, and dynamic environmental characteristics, the system can dynamically adjust the transparency and display priority of AR prompts, achieving perceptual coordination and cognitive load balance in information expression. The Lyapunov stability criterion is used for convergent control of the residual feedback system, ensuring stable operation even under changes in scene complexity or abnormal disturbances, improving the overall system's robustness and self-recovery capabilities. By integrating multimodal sensor data with an online learning mechanism, the system has the ability to quickly model and predict dynamic environmental changes, and can provide stable and efficient auxiliary decision support in typical scenarios such as urban congestion, highway traffic, and sudden changes in lighting.

[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0049] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A hardware-in-the-loop testing system for remote-driving vehicles integrating AR, characterized in that, It includes the following modules: dynamic obstacle intent parsing module, virtual-real scene spatiotemporal coupling module, multi-level obstacle avoidance decision evaluation module, and virtual-real residual adaptive compensation module; The dynamic obstacle intent parsing module is used to construct an intent-driven behavior prediction model for dynamic obstacles based on the time-series data of multimodal sensors, and output the trajectory probability distribution and dynamic risk area parameters. The virtual-real scene spatiotemporal coupling module is used to spatiotemporally align the virtual accident scene with real sensor data according to the dynamic risk area parameters and through a dynamic binding algorithm, and dynamically adjust the display priority of the virtual-real occlusion area in the AR interface through the trajectory probability distribution of dynamic obstacles. The dynamic binding algorithm includes iterative nearest-point initial alignment, sliding window nonlinear optimization, and extended Kalman filter residual correction; The multi-level obstacle avoidance decision evaluation module is used to inject abnormal disturbances of vehicle actuators into the spatiotemporally aligned fusion scene, synchronously collect driver response data to AR guidance, and generate obstacle avoidance path deviation index and takeover timeliness parameters. The virtual-real residual adaptive compensation module is used to separate the static features of the environment and the residuals of dynamic objects according to the obstacle avoidance path deviation index, dynamically compensate the position offset of the virtual-real scene through an online correction algorithm, and feed the residual compensation amount back to the virtual-real scene spatiotemporal coupling module for closed-loop optimization. The construction logic for building an intention-driven behavior prediction model for dynamic obstacles based on time-series data from multimodal sensors is as follows: The raw data from the camera, millimeter-wave radar, and lidar are time-stamped and transformed into a multi-sensor coordinate system to extract the motion feature vectors of dynamic obstacles. The motion feature vectors include velocity change trends, heading angle offsets, and acceleration fluctuation patterns. Based on historical trajectory data and real-time motion characteristics, a spatiotemporal behavior probabilistic graphical model of dynamic obstacles is constructed using a Bayesian network to predict the multimodal trajectory distribution of its future motion direction. The trajectory distribution includes probability weights for straight-ahead, lane-changing, and emergency braking behaviors. Based on the relative motion direction of the dynamic obstacle and the target vehicle and the lane topology, the conflict degree between the trajectory prediction result and the expected path of the target vehicle is calculated, and the trajectory with the conflict degree exceeding the safety threshold is marked as a high-risk event. The actual movement trajectory of the dynamic obstacle collected in real time is compared with the predicted trajectory. The node weights of the Bayesian network are adjusted in reverse according to the gradient descent algorithm to optimize the scene adaptability of the behavior prediction model.

2. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 1, characterized in that: Outputting the trajectory probability distribution and dynamic risk area parameters includes the following steps: The multimodal trajectory prediction results are mapped to a two-dimensional rasterized map, and a dynamic risk heat map is generated according to the probability weights. The raster density of the heat map is positively correlated with the trajectory conflict probability. The geometric boundaries of high-risk areas are extracted based on the density gradient changes in the heat map. The lateral and longitudinal buffer distances of the safe obstacle avoidance path are calculated in combination with vehicle dynamics constraints. The transparency and rendering level of the AR prompt information are dynamically adjusted based on the boundary of the risk area and the driver's field of view focus position. The extended Kalman filter algorithm is used to detect the residual convergence of the predicted trajectory and real-time sensor data. If the cumulative residual within the sliding window exceeds the dynamic adjustment threshold, the reconstruction of the Bayesian network model and the update of the heat map are triggered.

3. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 2, characterized in that: Based on dynamic risk area parameters, a dynamic binding algorithm is used to spatiotemporally align virtual accident scenarios with real sensor data, including the following steps: Based on the geometric boundaries and heat map distribution in the dynamic risk area parameters, key spatiotemporal feature points in the sensor data are extracted, including lane line intersections and traffic sign locations. Multimodal feature matching is performed with the topology of the virtual accident scenario, and the initial coordinate system is aligned through an iterative nearest point algorithm. Within a preset time window, the spatiotemporal offset between real sensor data and virtual scene is accumulated and calculated. The pose parameters of the virtual scene are dynamically adjusted through a nonlinear optimization algorithm to compensate for the superposition error caused by communication delay and sensor sampling jitter. The residuals of the virtual and real scenes after feature matching are estimated in real time by using the extended Kalman filter algorithm. If the residual exceeds the dynamic adjustment threshold, the local coordinate system of the virtual scene is fine-tuned.

4. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 3, characterized in that: Dynamically adjusting the display priority of virtual and real occlusion areas in the AR interface based on the trajectory probability distribution of dynamic obstacles includes the following steps: Based on the high-probability trajectory branches in the trajectory probability distribution, the AR prompt area is divided into multiple rendering layers, and the transparency level is dynamically assigned according to the probability value. Real-time detection of the spatial overlap area between real obstacles and virtual roadblocks; calculation of occlusion priority based on trajectory probability weights; acquisition of the driver's field of view focus coordinates; and dynamic adjustment of the rendering position and size of AR prompt information. When the trajectory probability distribution changes abruptly due to model reconstruction, the current rendering thread is immediately interrupted, and the occlusion priority is redistributed according to the updated probability weights.

5. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 4, characterized in that: Injecting abnormal disturbances from vehicle actuators into the spatiotemporally aligned fusion scene, and simultaneously collecting driver response data to AR guidance, generates obstacle avoidance path deviation indices and takeover timeliness parameters, including the following steps: Abnormal operating conditions of vehicle actuators are simulated through hardware-in-the-loop interface, including sudden changes in steering system torque, brake pedal response delay, and throttle opening drift, and virtual accident scenario disturbances are superimposed simultaneously. It records the driver's control signals, eye-tracking focus trajectory, AR cue gaze duration, and vehicle status data in real time, including lateral acceleration and yaw angle; Based on the positioning data and the expected obstacle avoidance path, calculate the integral of the lateral deviation and the cumulative amount of the heading angle offset of the vehicle's actual trajectory. The time interval between the AR prompt triggering moment and the driver's first effective control response is defined as the takeover delay, and the takeover success rate is calculated by weighting it together with the path deviation index.

6. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 5, characterized in that: Based on the obstacle avoidance path deviation index, the static features of the environment and the residuals of dynamic objects are separated. The positional offset of the virtual and real scenes is dynamically compensated through an online correction algorithm, including the following steps: The static environmental features and dynamic object features are extracted from the lidar point cloud data. The residual components of the static features and dynamic object features are distinguished according to the spatiotemporal consistency verification algorithm. The static features include road surfaces and fixed buildings, and the dynamic object features include pedestrians and vehicles. Based on the lateral deviation integral and heading angle offset cumulative amount of the obstacle avoidance path deviation index, the source of residual is determined to be either static environmental coordinate system offset or dynamic object motion prediction error, and static residual vector and dynamic residual vector are generated accordingly. For static residual vectors, the global coordinate system offset is compensated by the least squares optimization algorithm based on the sliding window; for dynamic residual vectors, the trajectory probability distribution parameters of dynamic obstacles are corrected in reverse by the kinematic prediction model. The corrected static and dynamic residual parameters are input into the virtual scene rendering engine to adjust the pose of the virtual accident scene and the motion trajectory of dynamic objects in real time.

7. The hardware-in-the-loop testing system for remote-driving vehicles integrating AR as described in claim 6, characterized in that: The logic for feeding back the residual compensation amount to the virtual-real scene spatiotemporal coupling module for closed-loop optimization is as follows: The static residual vector and the dynamic residual vector are normalized and weighted, and a comprehensive residual compensation weight is generated by combining the real-time scene complexity, which includes traffic flow and lighting conditions. The comprehensive residual compensation weight is input into the dynamic binding algorithm of the virtual-real scene spatiotemporal coupling module, and the objective function weight coefficient of the nonlinear optimization algorithm is adjusted to prioritize the compensation of high-weight residual components. Based on the distribution characteristics of the dynamic residual compensation, the real-time update of the display priority of the virtual and real occlusion areas in the AR interface is triggered. The convergence of the residual feedback loop is analyzed by Lyapunov stability criterion. If the system state variables diverge, the backup parameter set is switched and the spatiotemporal coupling optimization thread is restarted.

Images