Full-link interface-based virtual-real combined digital twin method and system for autonomous driving

Abstract

This invention discloses a digital twin method and system for autonomous driving that integrates virtual and real environments based on a full-link interface. The invention includes: converting multi-source perception data from autonomous vehicles in a real physical environment into a real object state dataset; establishing a standardized state mapping between the real system state space and the virtual simulation system state space; converting the real object state dataset into virtual environment state data executable by the simulation engine; constructing a virtual environment; performing virtual perception based on the constructed virtual environment: performing sensor-level sampling on the virtual environment state data to generate virtual raw sensor data, forming a virtual perception result data stream; and verifying the virtual perception result data stream through the twin verification loop. This invention solves the problem of simulation engines being unable to dynamically generate objects during runtime, achieving high-fidelity, low-latency reproduction of real-world traffic flow in a simulation environment.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent driving technology, specifically to an autonomous driving virtual-real combined digital twin method and system, electronic device, and storage medium based on a full-link interface. Background Technology

[0002] As autonomous driving technology evolves from modular to end-to-end architectures, the confidence level of simulation testing has become a key factor in verifying the safety of algorithm models. Existing simulation testing mainly falls into two categories: fully virtual simulation and data playback-based schemes. Traditional playback simulation involves directly injecting recorded real-world data into the autonomous driving system. This approach has many limitations under current end-to-end autonomous driving schemes and cannot perform closed-loop control. Further methods involve extracting vehicle trajectories based on the OpenScenario standard format from the recorded data and reproducing them in simulation software.

[0003] However, traditional technologies have the following drawbacks: 1) Static scene elements, unable to close the loop: In traditional OpenScenario scenes, the participants are often fixed after loading. Simulation engines (such as aiSim, VTD, etc.) usually do not support dynamically creating or destroying a large number of 3D models during simulation runtime. This results in the simulation environment being unable to render the object in real time when the real vehicle perceives a newly appearing obstacle.

[0004] 2) Disconnect between sensor data and simulation environment: Existing playback technologies are mostly based on simple bounding box projection rather than realistic 3D model rendering. This means that the sensor models in the simulation (such as virtual cameras and virtual LiDAR) cannot detect these playback objects, making it impossible to conduct closed-loop testing "based on raw sensor data".

[0005] 3) Insufficient spatiotemporal synchronization accuracy: The acquisition frequency of real vehicle data (such as the perception frame rate of 10Hz-30Hz) often does not match the physical step size of the simulation engine (such as 100Hz), and there is network transmission delay. Direct driving will cause the simulation screen to jitter or clipping. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a method, system, and electronic device for autonomous driving that combines virtual and real elements based on a full-link interface. This method enables the reproduction of real traffic flow in a simulation environment through a full-link control interface, and solves the problem of limited dynamic object generation in simulation engines during runtime.

[0007] To address the aforementioned problems, the first aspect of this invention discloses a method for creating a virtual-real integrated digital twin for autonomous driving based on a full-link interface, comprising the following steps: Transform multi-source perception data of autonomous vehicles in real physical environments into real object state datasets; Establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system; Based on the standardized state mapping, the real object state dataset is transformed into virtual environment state data that can be executed by the simulation engine. Constructing a virtual environment: Constructing a simulation entity pool, pre-generating multiple simulation entity instances in a hidden state, establishing a mapping relationship between real object states and simulation entity instances based on object identification information in the real object state dataset, binding the simulation entity instances to the corresponding real object states based on the mapping relationship, and updating the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. Virtual perception is performed based on the constructed virtual environment: sensor-level sampling is performed on the virtual environment state data to generate virtual raw sensor data, forming a virtual perception result data stream; A digital twin verification loop is constructed, and the virtual perception result data stream is verified through the digital twin verification loop.

[0008] Optional, also includes: Based on the timestamp alignment mechanism, when the simulation step frequency is higher than the sensing data frequency, intermediate state data is generated by interpolation or dead reckoning, and the intermediate state data is input into the simulation system.

[0009] Optionally, the dynamic scheduling includes: When a real target identifier in the sensing object list is bound to a simulated entity instance, the corresponding simulated entity instance is activated and its pose information is synchronized. When the real target identifier no longer appears in the list of perceived objects, the binding relationship between the simulated entity instance and the real target identifier is released, and the simulated entity instance is restored to a hidden state.

[0010] Optionally, the step of converting the multi-source perception data of autonomous vehicles in the real physical environment into a real object state dataset includes: Multi-source sensor data from real vehicles is collected, and the BEVFormer algorithm is used to convert real-time multi-channel images into a 3D perception object list. The 3D perception object list is then standardized and encapsulated to generate a perception object data stream in a unified format.

[0011] Optional, also includes: A communication gateway and ring buffer structure are constructed. Asynchronous data writing is achieved through an independent receiving thread, and synchronous data reading is achieved based on the simulation timestamp through an independent reading thread. Simulation input data frames are constructed through the nearest time frame matching mechanism.

[0012] Optionally, constructing the simulation entity pool includes: Based on historical real object state time series data, the traffic flow change prediction model is used to predict the trend of traffic target quantity change within a future preset time window, and the prediction result of traffic flow scale is obtained. Based on the prediction results and the real-time perceived object list, the virtual objects in the object pool are hierarchically scheduled and bound and mapped; the prediction results are used to plan the capacity of the object pool and pre-allocate simulation entity resources in advance. The object pool is a multi-level virtual object pool, which includes simulation entity pools of different levels. Different levels of simulation entity pools correspond to object pool levels with different model complexity levels.

[0013] Optionally, the twin verification loop verifies the virtual perception result data stream, including: A consistency comparison analysis is performed between the virtual perception result data stream and the real object state dataset, and deviation correction parameters are generated. These deviation correction parameters are used to correct the position parameters, attitude parameters, or model parameters of the virtual simulation entity instance.

[0014] A second aspect of this invention discloses an autonomous driving virtual-real hybrid digital twin system based on a full-link interface, comprising: Data unit, used to transform multi-source perception data of autonomous vehicles in real physical environments into a real object state dataset; Standard cells are used to establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system. The conversion unit is used to convert the real object state dataset into virtual environment state data that can be executed by the simulation engine according to the standardized state mapping. The simulation unit is used to construct a virtual environment: it constructs a simulation entity pool, pre-generates multiple simulation entity instances in a hidden state, establishes a mapping relationship between the real object state and the simulation entity instances based on the object identification information in the real object state dataset, binds the simulation entity instances to the corresponding real object states based on the mapping relationship, and updates the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. The sensing unit is used to perform virtual sensing based on the constructed virtual environment: it performs sensor-level sampling on the virtual environment state data, generates virtual raw sensor data, and forms a virtual sensing result data stream; The verification unit is used to construct a digital twin verification loop and verify the virtual perception result data stream through the twin verification loop.

[0015] A third aspect of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the autonomous driving virtual-real integration digital twin method based on a full-link interface disclosed in the first aspect of the present invention.

[0016] The fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the autonomous driving virtual-real combined digital twin method based on a full-link interface disclosed in the first aspect of the present invention.

[0017] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: 1) This invention breaks through the "static runtime" limitation of simulation engines and realizes dynamic traffic flow rendering. Through the innovative design of the "preloaded object pool + dynamic mapping" mechanism, this invention cleverly avoids the underlying limitations of simulation engines. The system pre-sets sufficient dormant resources during the initialization phase and "wakes up" and drives objects in real time through ID mapping during runtime. This means that no matter how complex the real road conditions are (such as suddenly merging traffic or crossing pedestrians), the simulation environment can perform 3D rendering in real time without delay, which greatly improves the flexibility and coverage of the virtual and real scene combination. It solves the problem that traditional simulation engines (such as aiSim and VTD simulation engines) usually require all traffic participants to be predefined before scene loading and do not support the dynamic creation or destruction of a large number of 3D models during simulation runtime, resulting in the inability to reproduce sudden or variable traffic conditions in the real world. 2) This invention eliminates 2D-3D conversion errors and improves the spatial fidelity of virtual-real mapping. By introducing the BEVFormer bird's-eye view perception model at the vehicle end, feature extraction and inference are performed directly in a unified 3D space, outputting accurate 3D coordinates, dimensions, and orientation. This eliminates the depth estimation error caused by 2D projection from the source, and the positional accuracy can reach the centimeter level. At the same time, the simulation end drives a real 3D model with physical properties, rather than a simple bounding box. This makes the physical characteristics of the simulation environment, such as light occlusion and radar cross section, highly consistent with the real world. This provides a high-fidelity environment for sensor-level closed-loop testing and solves the problem that traditional technologies are mostly based on simple perspective projection of 2D bounding boxes, which lack depth information, resulting in serious positional drift of virtual objects in the simulation space and the inability to present real physical properties (such as occlusion and shadows).

[0018] 3) This invention solves the spatiotemporal jitter of multi-frequency heterogeneous data, ensuring simulation smoothness. By establishing a ring buffer and combining it with dead reckoning interpolation algorithm, it can automatically fill data gaps between high-frequency steps in the simulation and smooth network jitter. This not only ensures visual smoothness but also guarantees the continuity of data collected by virtual sensors on the time axis, avoiding algorithm misjudgment caused by frame loss. It solves the problem of mismatch between the real vehicle-end sensing frequency (usually 10-30Hz) and the simulation engine step size (usually 100Hz) and the random delay in network transmission, which can cause "stuttering" or "teleportation" in the simulated vehicle movement and disrupt physical continuity when directly driven.

[0019] 4) This invention has high scalability and compatibility. By defining a standardized object list (ObjectList) as an intermediate layer protocol, the vehicle side is responsible for "standardizing" heterogeneous sensor data, and the simulation side only recognizes the standard data interface. This decoupled architecture means that the simulation side does not need to care whether the data acquisition vehicle uses LiDAR or a pure vision solution, nor does it need to care about the internal version of the perception algorithm. This greatly reduces the system maintenance cost and facilitates the subsequent connection of autonomous driving test vehicles with different configurations. It solves the problem that traditional technologies often hardcode the CAN signals of specific vehicle models directly into the simulator, which requires rewriting a large amount of interface code when changing the data acquisition vehicle or upgrading the sensor configuration. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an embodiment of the present invention of a method for autonomous driving combining virtual and real digital twins based on a full-link interface; Figure 2 This is a schematic diagram of the structure of an autonomous driving virtual-real integrated digital twin system based on a full-link interface, provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation

[0021] This specific embodiment is merely an explanation of the embodiments of the present invention and is not intended to limit the embodiments of the present invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but as long as they are within the scope of the claims of the embodiments of the present invention, they are protected by patent law.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of the present invention.

[0023] The term "comprising" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0024] In embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] Example 1 Please refer to Figure 1-3 As shown, an autonomous driving method combining virtual and real elements based on a full-link interface is described. Figure 1 As shown, it includes the following steps: Step S110: Convert the multi-source perception data of the autonomous vehicle in the real physical environment into a real object state dataset; In this embodiment, heterogeneous raw sensing data (i.e., multi-source sensing data) collected by multiple sensors in the real world is transformed into a unified, structured, computable, mappable, and simulation system-driving state data representation form, forming a real object state dataset.

[0026] Specifically, the multi-source sensing data may include camera image data, lidar point cloud data, millimeter-wave radar target data, ultrasonic distance data, high-precision map data, GNSS / IMU pose data, and vehicle CAN bus data, etc.

[0027] In practical implementation, the multi-source sensing data is timestamped and aligned, the coordinate systems of different sensors are mapped to the vehicle coordinate system, and then uniformly mapped to the global coordinate system. Target extraction and association are performed based on deep learning perception models and fusion algorithms. For example, image detection models, point cloud detection models, radar target models, etc., can be used to form multimodal fused target data. Then, a target entity state structure is constructed for each perceived target, and a unified state vector is established for each target, thereby outputting a real object state dataset.

[0028] Specifically, this includes constructing a real system state modeling structure, performing time synchronization, coordinate alignment, and multi-source fusion processing on the multi-source sensor perception data of autonomous vehicles in real physical environments, and transforming heterogeneous raw perception data into a unified and structured real object state data expression form through target detection, target classification, multi-target tracking, and state space modeling, generating a real object state dataset containing object identification information, spatial pose, motion state parameters, and semantic attribute information.

[0029] Step S110 specifically includes: Multi-source sensor data from real vehicles are collected, and the BEVFormer algorithm is used to convert real-time multi-channel images into a 3D perception object list. The 3D perception object list is then standardized and encapsulated to generate a unified format perception object data stream, i.e., a real object state dataset.

[0030] Step S120: Establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system; In this embodiment, the objects for establishing standardized state mapping may include spatial state variables, motion state variables, semantic state variables, scale state variables, and time state variables.

[0031] Specifically, this may include establishing a unified transformation relationship between multiple coordinate systems, defining a one-to-one correspondence between real system state variables and simulation system state variables, unifying the state data structure expression format, achieving consistency in object categories, behavioral semantics, and traffic rule semantics, and outputting a mapping rule model.

[0032] In practice, a standardized state space mapping model is constructed, and a unified system of state expression rules is established between the state space of the real system and the state space of the virtual simulation system. Through coordinate system mapping, state variable mapping, state structure mapping and semantic space mapping, the structured equivalent transformation and consistency alignment between the state data model of the real object and the state data model of the virtual world are realized.

[0033] Step S130: Based on the standardized state mapping, the real object state dataset is transformed into virtual environment state data that can be executed by the simulation engine; In this step, based on the standardized state space mapping model, the real object state dataset is subjected to state space transformation processing. Through coordinate space mapping, state variable reconstruction and data structure encapsulation, the real system state data is transformed into a virtual environment state data structure that conforms to the simulation engine interface specification, thereby generating a virtual environment state dataset that the simulation engine can directly parse and execute.

[0034] Step S140: Construct a virtual environment: Construct a simulation entity pool, pre-generate multiple simulation entity instances in a hidden state, establish a mapping relationship between the real object state and the simulation entity instances based on the object identification information in the real object state dataset, bind the simulation entity instances to the corresponding real object states based on the mapping relationship, and update the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. In this step, during the simulation initialization phase, multiple types of simulation entity instances are pre-generated, including but not limited to: vehicle entities, pedestrian entities, non-motorized vehicle entities, and obstacle entities.

[0035] For example, all simulation entity instances are initialized as follows: Spatial location: hidden region coordinates; Physical state: inactive; Rendering state: invisible; Lifecycle state: Idle.

[0036] Parse object identification information (including ID information) from real object state data and use it as a cross-system index key. Establish a mapping relationship between data states and simulated entities, binding simulated entity instances to corresponding real object states. Specific binding content may include: ID binding, lifecycle binding, state synchronization binding, control binding, etc. After state-space mapping transformation, the spatial pose states in the real system state data model are directly written into the pose control interface of the simulated entity instance to achieve state-driven control of the simulated entity's spatial state.

[0037] In specific implementation, the process includes: constructing a simulation entity pool; pre-generating multiple types of simulation entity instances and setting them to a hidden state during the simulation system initialization phase; establishing a mapping relationship between real object states and simulation entity instances based on object identification information in the real object state dataset during the simulation run phase; binding simulation entity instances with corresponding real object states; and, based on the state space mapping results, directly writing the object space pose in the real system state data model into the pose control interface of the simulation entity instance through a pose state injection mechanism to achieve consistency and synchronization between the simulation entity space state and the real object space state.

[0038] Furthermore, when the state of the real object disappears from the dataset, the binding relationship between the corresponding simulation entity instance and the corresponding real object state is released, and the simulation entity instance is recycled to the hidden state area.

[0039] Step S150: Perform virtual perception based on the constructed virtual environment: perform sensor-level sampling on the virtual environment state data to generate virtual raw sensor data and form a virtual perception result data stream; In this step, the virtual environment state data may include the spatial state of objects, the structural state of the scene, and environmental state parameters. The simulated entity instance environment includes the entity's geometric model, material parameters, reflection properties, and dynamic behavior state. The virtual sensor model library may include camera head models, point cloud detection models, and radar target models.

[0040] In this step, based on the completed virtual environment state data model and simulation entity instance environment, a virtual sensor model is constructed, and sensor mechanism-level sampling simulation is performed on the virtual environment state data. Through geometric imaging modeling, physical mechanism modeling and sensor error modeling, virtual raw sensor data with the same format as real sensor data is generated, and a virtual sensing result data stream is formed to drive the algorithm system for closed-loop verification and performance evaluation.

[0041] Step S160: Construct a digital twin verification loop and verify the virtual perception result data stream through the twin verification loop.

[0042] In practical implementation, the digital twin verification loop may include: a virtual perception injection interface, an algorithm execution module, a virtual control feedback mechanism, a virtual system state evolution module, and a multi-source alignment verification mechanism.

[0043] Specifically, the virtual perception injection interface inputs the virtual perception result data stream into the algorithm system, replacing the input from real sensors, ensuring complete consistency in data format, time synchronization, and data protocol. Then, through a virtual control feedback mechanism, the algorithm output is applied to the virtual system, and updated via the virtual system state evolution module: the main vehicle state, traffic body behavior state, scene evolution state, and environmental variable state, etc.

[0044] In practical implementation, the multi-source alignment verification mechanism includes multi-layer alignment verification, which may include: state alignment: real object state vs. virtual object state; trajectory alignment: real trajectory vs. virtual trajectory; perception alignment: real perception result vs. virtual perception result; decision alignment: real system decision vs. virtual system decision.

[0045] In this step, a digital twin verification loop is constructed. By injecting the virtual perception result data stream into the autonomous driving algorithm system, the algorithm system is driven to perform perception fusion, decision planning and control strategy generation, and the algorithm output results are fed back to the virtual environment, so that the virtual world state continues to evolve. At the same time, the consistency between the real system state and the virtual system state is verified through a multi-level alignment mechanism, thus forming a system-level digital twin closed-loop verification structure of "virtual perception - algorithm decision - virtual control - virtual environment evolution - re-perception".

[0046] Optionally, the verification of the virtual perception result data stream via a twin verification loop includes: Step S1601: Perform a consistency comparison analysis between the virtual perception result data stream and the real object state dataset, and generate deviation correction parameters. The deviation correction parameters are used to correct the position parameters, attitude parameters, or model parameters of the simulated entity instance.

[0047] In this step, by performing a multi-dimensional consistency comparison analysis between the virtual perception result data stream and the real object state dataset, the deviation relationship between the virtual and real systems in terms of spatial state, motion state and structural parameters is quantified, and a deviation correction parameter set is generated. This set is used to dynamically correct the position parameters, attitude parameters or model parameters of the simulated entity instances in the virtual simulation system, thereby achieving continuous alignment of the virtual and real system state spaces. By continuously comparing the virtual and real states and dynamically correcting the simulation model, the virtual system continuously approaches the state evolution trajectory of the real system.

[0048] Optionally, the method of the present invention further includes: Step S1401: During the simulation operation phase, the coordinate forced anchoring method is adopted to directly map the positioning attitude data of the real vehicle to the position and attitude parameters of the simulation master vehicle. The simulation master vehicle adopts a multi-source constraint fusion anchoring mechanism, which integrates various positioning and environmental constraint information to uniformly anchor and control the coordinates of the simulation master vehicle.

[0049] Optionally, the method of the present invention further includes: Step S1402: When the simulation step size frequency is higher than the sensing data frequency, intermediate state data is generated based on the timestamp alignment mechanism through interpolation or dead reckoning, and the intermediate state data is input into the simulation system.

[0050] Specifically, including: Step S14021: Obtain the simulation timestamp of the current simulation frame, align and match it with the timestamp of the sensing data frame on the time axis, and determine the nearest neighbor sensing data frame and the frame intervals before and after it; Step S14022: Compare the operating frequency of the simulation system with the data update frequency of the sensing system to determine whether there is a frequency mismatch; Step S14023: Determine whether there are two consecutive frames of sensing state data in the current matching interval for interpolation calculation to generate intermediate state data; in the case of missing or discontinuous sensing data frames, predict the target state by dead reckoning; if there is no next frame of data, use dead reckoning to predict the state and obtain intermediate state data.

[0051] For example, if there are consecutive sensing frames, interpolation calculations are performed on state parameters such as position, velocity, and attitude to generate intermediate states corresponding to the simulation timestamps. If only a single frame of sensing data exists or a sensing frame is lost, state extrapolation prediction is performed based on the target's current velocity, acceleration, and heading information.

[0052] Step S14024: Organize the generated intermediate state data into a standard state structure for use as the state-driven input of the simulation system; Step S14025: Perform continuity and physical rationality checks on intermediate state data to avoid abrupt state changes and non-physical jumps.

[0053] In this embodiment, during the simulation operation phase, the simulation timeline and the sensing data timeline are synchronized and matched based on a timestamp alignment mechanism. When the simulation step size frequency is higher than the sensing data update frequency, the time frame interval is first matched and determined. If there are adjacent sensing data frames, intermediate state data is generated through state interpolation. If the sensing data frames are missing or discontinuous, the target state is predicted and modeled through dead reckoning. The generated intermediate state data is structured, encapsulated, and its continuity and physical rationality are verified before being injected into the simulation system as a state-driven input. This achieves continuous state synchronization between the virtual and real systems under multi-frequency asynchronous conditions, enabling the simulation system to obtain continuous and smooth state input under high-frequency operation conditions.

[0054] Optionally, the method of the present invention further includes: Step S1403: During the simulation run phase, dynamic scheduling is performed, which includes: When the object identification information in the real object state dataset is bound to the simulation entity instance in the object pool, the corresponding simulation entity instance is activated and its pose information is synchronized. When the object identification information no longer appears in the real object state dataset table, the binding relationship between the simulation entity instance and the object identification information is released, and the simulation entity instance is restored to the hidden state.

[0055] Optionally, the method of the present invention further includes: Step S170: Construct a communication gateway and a ring buffer structure, implement asynchronous data writing through an independent receiving thread, implement synchronous data reading based on the simulation timestamp through an independent reading thread, and construct the simulation input data frame through the nearest time frame matching mechanism.

[0056] Specifically, this includes: establishing a circular buffer of a preset length at the simulation end to buffer data packets from the vehicle end or the sensing system; and writing the latest received data packets into the circular buffer in timestamp order through an independent data receiving thread. An independent data reading thread retrieves the two nearest frames of data to the current simulation timestamp from the circular buffer; a simulation input data frame is constructed based on the two nearest frames of data to drive the state update of the simulation system.

[0057] Specifically, a pre-defined circular buffer structure is constructed at the simulation end to store time-series data frames. The circular buffer employs a cyclic overwrite mechanism; when the buffer capacity reaches its limit, the earliest written data is overwritten. Data packets received by the communication gateway are parsed and processed by an independent data receiving thread and then written into the circular buffer, stored in order of data timestamp. Specifically, state alignment, time interpolation, or state prediction processing is performed based on the two nearest frames of data to generate input state data frames that can be directly used by the simulation system, thereby achieving decoupling and synchronization between the sensing data stream and the simulation system's runtime sequence.

[0058] Optionally, constructing the simulation entity pool includes: Step 14000: Based on historical real object state time series data, predict the trend of traffic target quantity change within a future preset time window through a traffic flow change prediction model to obtain the prediction result of traffic flow scale. In this step, the historical real object state time series dataset may include: timestamps, number of objects, type distribution (vehicles / pedestrians / non-motorized vehicles), speed distribution, and other data.

[0059] In practical implementation, a traffic flow change prediction model is constructed to predict future time windows. This model can employ an extended Kalman Filter, with prediction windows such as 3 seconds, 5 seconds, and 10 seconds. For example, it can predict local density changes and regional traffic flow aggregation trends.

[0060] Step 14001: The prediction results are used to plan the capacity of the object pool and pre-allocate simulation entity resources in advance. The object pool is a multi-level virtual object pool, which includes simulation entity pools of different levels. Different levels of simulation entity pools correspond to object pool levels with different model complexity levels.

[0061] For example, if the predicted maximum number of targets is 60 and the safety redundancy factor is 1.3, then the object pool capacity planning can be 78 simulated entity instances.

[0062] For example, a multi-level virtual object pool may include Level-1, Level-2, and Level-3 object pools, which can correspond to high-precision models, medium-precision models, and low-precision models, respectively.

[0063] Specifically, the prediction results are used to plan the capacity of the object pool and pre-allocate simulation entity resources in advance. Based on the prediction results and the real object state dataset, the simulation entity instances in the object pool are scheduled and bound in a hierarchical manner. When the real object state attributes are different, simulation entity instances of different complexity levels are selected for mapping and binding.

[0064] Specifically, real object status attributes may include distance, target type, risk level, visibility level, and other attributes.

[0065] For example, the object pool levels of high-precision model, medium-precision model, and low-precision model correspond to mapping and binding of near-range targets, medium-range targets, and far-range targets, respectively.

[0066] For example, vehicle A, at a distance of 12m (close range), is a high-risk, high-precision model with a simulation entity pool of Level-1.

[0067] The following is a specific embodiment for illustration: The system used in this invention may specifically include a vehicle-side perception and data processing subsystem and a simulation-side synchronous control and rendering subsystem.

[0068] The vehicle-side perception data processing subsystem is used to collect sensor data from real vehicles, extract environmental targets using a deep learning perception model, and convert them into a standardized perception object list, which is then sent to the simulation terminal. The simulation-end synchronous control and rendering subsystem adopts the "preloaded object pool" technology to pre-generate an excessive number of dormant virtual objects during the simulation initialization phase. During the simulation operation phase, a dynamic mapping relationship between real perception IDs and virtual object indexes is established through the full-link control interface. Virtual objects are scheduled from the object pool and moved to their corresponding real-world coordinates, thereby enabling the simulation engine to achieve real-time rendering of variable traffic flows even when it does not support dynamic instantiation.

[0069] Specifically, the vehicle-side perception and data processing subsystem includes: a multimodal sensor acquisition module and a deep learning perception inference module, a data standardization and encapsulation module, and a data transmission and flow control module.

[0070] The multimodal sensor acquisition module functions as follows: it collects real-time information on the vehicle's own status and the external environment. Its detailed configuration is as follows: GNSS / IMU integrated navigation unit: outputs the vehicle's latitude and longitude, altitude, and Roll / Pitch / Yaw attitude angles in the world coordinate system, with a frequency of not less than 100Hz.

[0071] Visual sensor array: A surround-view camera array covering the front, rear, left and right views of the vehicle, collecting image data that is triggered synchronously.

[0072] LiDAR: Collects 3D point cloud data of the environment to supplement depth information.

[0073] The deep learning perception inference module has the following functions: It utilizes an in-vehicle computing platform (such as NVIDIA Orin) to run real-time perception algorithms, extract environmental targets, and directly convert multiple images into a 3D object list. The specific implementation is as follows: Algorithm implementation (taking BEVFormer model as an example): Feature extraction: Input the multi-path loop image into the backbone network of ResNet residual neural network or Swin Transformer model to extract multi-scale feature maps.

[0074] Spatial Cross-Attention: This mechanism uses pre-defined BEV (Bird's EyeView) algorithm queries to interact with image features, projecting 2D pixels onto a 3D BEV plane using camera intrinsic and extrinsic parameters to generate a unified feature view.

[0075] Temporal fusion: Introducing historical frame features and enhancing the stability of occluded and moving target detection through a temporal self-attention mechanism.

[0076] BEV spatial transformation: Utilizing the spatial cross-attention mechanism, 2D image features are projected onto a unified 3D bird's-eye view (BEV) feature space.

[0077] Target decoding: The 3D attributes of the target are directly regressed through the detection head, including: center point coordinates (x,y,z), three-dimensional dimensions (l,w,h), rotation angle (θ), velocity vector (vx,vy), and class confidence.

[0078] Output: Generates a Local Perception List containing all traffic participants.

[0079] The data standardization and encapsulation module functions as follows: it uniformly converts heterogeneous sensing results into a universal transmission protocol defined in this invention. The Object List data structure is defined as follows: { "frame_id": 1024, "timestamp": 1679000000.123, "ego_state": { "lat": ..., "lon": ..., "heading": ...}, "objects": [ { "id": 1, / / Real-world tracking ID "type": "vehicle", / / or pedestrian, cyclist "position": { "x": 10.5, "y": -3.2, "z": 0.0}, / / Relative Ego coordinates "dimension": { "length": 4.5, "width": 1.8, "height": 1.5}, "orientation": 0.15, / / Heading angle relative to Ego "velocity": { "vx": 10.0, "vy": 0.2}, "confidence": 0.95 }, ... ] } Processing logic: Perform coordinate system transformation (from sensor coordinate system to vehicle coordinate system) and filter low-confidence targets.

[0080] The data transmission and flow control module has the following functions: it transmits data to the simulation terminal via a 4G / 5G network or a local area network (Lab environment).

[0081] Implementation: ZeroMQ or ROS2 DDS middleware is used, QoS policy configuration is supported, and priority is given to ensuring the arrival of the latest data (expired frames are discarded).

[0082] The simulation-end synchronous control and rendering subsystem specifically includes: a communication gateway, a circular buffer, and a full-link scene controller. This simulation-end synchronous control and rendering subsystem runs on a workstation and carries simulation engines such as aiSim / VTD and the control program developed in this invention.

[0083] The communication gateway and the ring buffer (Receiver & Buffer) have the following functions: receiving data from the vehicle and resolving the problem of uneven frame rates caused by network jitter.

[0084] The specific design is as follows: A circular buffer of length N (e.g., N=50) is established. The receiving thread is responsible for writing the latest data packet. The reading thread searches the buffer for the two closest frames of data, Tprev and Tnext, based on the simulation timestamp.

[0085] The full-link scenario controller has the following functions: it takes over the main control of the scenario by using the APIs provided by the simulation engine (such as ExternalScenario Callback).

[0086] The end-to-end scenario controller specifically includes the following three key lifecycle callbacks: ① Initialization Phase – Object Pool Construction The logic is as follows: a. Read the configuration file to determine the maximum supported traffic flow density (e.g., 50 vehicles, 20 pedestrians).

[0087] b. Call the GetActorTemplates module to obtain asset models (such as Sedan, Truck, SUV, Pedestrian_Male asset models, etc.) from the simulation library.

[0088] c. Build an object pool: Using the PrepareActor interface, pre-generate 50 vehicle instances and 20 pedestrian instances in an area not visible in the scene (such as below the ground plane at Z = -100 meters).

[0089] d. Create a PoolStatusMap table to record the occupancy status (Free / Busy) and the currently bound real ID of each instance.

[0090] ② Start Phase – Alignment of the Coordinate Origin The logic is as follows: a. Wait for the first frame of vehicle-side data to arrive.

[0091] b. Analyze the GNSS coordinates of the real vehicle and set them as the initial position of the Ego coordinate system in the simulation scene to complete the "virtual and real world coordinate anchoring".

[0092] ③ Drive / Step Phase – Dynamic Mapping and Rendering The logic is as follows (executed per frame): a. Ego Synchronization: Obtain the interpolated Ego pose and force the simulation master vehicle position to be set through Coordinate Forcing.

[0093] b. Context object mapping: a) Input steps: Input the Lreal real-world perception list; b) Update steps: Iterate through the objects in the Lreal real-sense list. If the ID of an object is already bound to the simulation object Osim in the PoolStatusMap, update the position and orientation of the simulation object Osim to the real target position, and set the Active property to True.

[0094] c) Allocation steps: If there is a new ID in the Lreal real-sense list and there are free objects in the object pool, then extract a simulation object Osim from the object pool, bind the ID, move it to the specified position, and set the Active property to True.

[0095] d) Recycling steps: Traverse the PoolStatusMap status table. If a bound ID no longer appears in the Lreal real perception list (target lost or out of sight), unbind the simulation object Osim, reset its position back to the hidden area (Z=-100), and set the Active property to False.

[0096] c. Physical rendering: The simulation engine automatically calculates lighting, shadows, and radar reflection characteristics based on the updated Actor position.

[0097] In the above environment object mapping, pre-loading (Pool Construction) is performed first: during simulation loading, pre-generated virtual objects are uniformly placed in the hidden area outside the view frustum, and their physical state is set to the Idle inactive state; when a new real perception ID is received, an idle object is extracted from the object pool, and it is instantly teleported to the real target position through coordinate transformation. The model is scaled according to the perception size, and the physical rendering attribute is enabled to achieve dynamic activation; when the real perception ID is lost in N consecutive frames of data, the ID binding of the object is unbound, its coordinates are reset back to the hidden area, and it is restored to the inactive state, waiting for the next allocation, thus achieving automatic recycling.

[0098] Specifically, the working principle and process are as follows: The workflow of this invention relies on a simulation platform and is designed with the following key lifecycle callback mechanism, the specific steps of which are as follows: Step 1: Scene initialization, complete object pool construction At the moment the simulation scene loads, the system does not rely on external data and first builds a "resource library": Template loading: Load 3D models of different sizes (cars, SUVs, trucks, pedestrians, etc.) from the simulation 3D material library.

[0099] Object pool instantiation: Pre-create N (e.g., N=50) virtual Actors in a dormant state.

[0100] State preset: Initialize these Actor coordinates in hidden locations outside the map (e.g., Z= (100m), and disable its physics engine usage, marking it as Idle inactive, waiting for the perception data to activate.

[0101] Step 2: Real-time perception, reasoning, and coding on the vehicle side During actual vehicle operation, the system performs high-frequency environmental modeling and multi-source input: the vehicle-mounted camera captures the current surround view image.

[0102] (1) BEV feature transformation: The 6-channel 2D image features are projected onto the 3D BEV feature map through a perception algorithm (such as the BEVFormer model).

[0103] (2) Target extraction: The precise 3D coordinates and dimensions of surrounding obstacles are regressed on the BEV map through a query mechanism.

[0104] (3) Encoding and encapsulation: The identified vehicles and pedestrians are encapsulated into a list of Standard_Object_List objects with timestamps and sent to the simulation terminal via 5G / private network.

[0105] Step 3: Simulation start-up, main vehicle position and attitude anchoring After the simulation terminal receives the first frame of vehicle-side data: (1) Coordinate system transformation: Convert the vehicle-side GNSS coordinates into local Cartesian coordinates within the simulation scene.

[0106] (2) Forced alignment of the main vehicle: Through the StartEvent interface, the main body of Ego in the simulation system is directly "landed" to a geographical location that is completely consistent with the real vehicle.

[0107] Step 4: Real-time execution and dynamic mapping loop During the simulation, the system executes the following closed-loop logic at each step: (1) Self-driving: Read the attitude data of the real self-driving vehicle and directly update the ActorState of the simulated master vehicle to ensure that the virtual and real space positions are perfectly matched.

[0108] (2) Dynamic activation of the perceived target: ① Search and Match: Iterate through the Object List passed from step two. For example, if the real target with ID 101 appears in the current frame, the system retrieves an Idle virtual model from the object pool in step one.

[0109] ②State synchronization: The virtual model is instantly moved from the "hidden position" to the 3D coordinate position calculated by the perception algorithm, and the length, width and height of the model are adjusted according to the perceived size. At the same time, its state is switched to Active state.

[0110] ③ Model Retrieval: If the perception algorithm loses the target with ID 101 in a subsequent frame (the target leaves the field of view), the system automatically moves the virtual model back to the hidden coordinates and resets it to the Idle state for subsequent use.

[0111] Step 5: Physical Rendering and Sensor Resimulation (1) Light and shadow rendering: Since the virtual model has been driven to the correct position by the perception algorithm, the simulation engine performs physical rendering based on the current lighting and occlusion relationship of the environment.

[0112] (2) Virtual sensor capture: The virtual camera and virtual radar on the simulated host vehicle scan these virtual objects generated by the "perception algorithm", generating a raw sensor data stream that combines the virtual and real.

[0113] Algorithm closed loop: The generated virtual sensor data is input again into the autonomous driving algorithm under test to achieve true virtual-real combined closed loop verification.

[0114] In a specific embodiment, the present invention utilizes a dynamic activation mapping mechanism based on an object pool to overcome the shortcomings of simulation engines that do not support runtime dynamic instantiation. During the simulation initialization phase, an excessive number of "dormant" objects are preloaded. At runtime, an ID mapping algorithm dynamically "parasites" real-world perceived targets onto dormant objects, enabling real-time rendering of variable-volume traffic flows. Simultaneously, a standardized perception link is integrated into the BEV perception model: This defines in detail the conversion path from multi-sensor raw data to a structured Object List, and utilizes advanced algorithms such as the BEVFormer model to directly output 3D spatial coordinates, eliminating errors in projecting 2D perception onto 3D space.

[0115] Furthermore, a multi-level spatiotemporal alignment and smoothing algorithm was adopted, and a timestamp-based ring buffer and interpolation / dead reckoning algorithm were designed to effectively resist vehicle-cloud-device communication jitter and ensure the motion continuity of the simulation object.

[0116] This invention solves the problem of the simulation engine being unable to dynamically generate objects during runtime by using a standardized perception data interface and an innovative simulation-end control strategy, thereby achieving high-fidelity, low-latency reproduction of real-world traffic flow in the simulation environment.

[0117] Example 2 This invention discloses an autonomous driving virtual-real hybrid digital twin system based on a full-link interface, such as... Figure 2 As shown, Figure 2 It is an autonomous driving virtual-physical integrated digital twin system based on a full-link interface, including: Data unit 210 is used to convert multi-source perception data of autonomous vehicles in real physical environments into a real object state dataset. Standard unit 220 is used to establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system; The conversion unit 230 is used to convert the real object state dataset into virtual environment state data that can be executed by the simulation engine according to the standardized state mapping. Simulation unit 240 is used to construct a virtual environment: construct a simulation entity pool, pre-generate multiple simulation entity instances in a hidden state, establish a mapping relationship between the real object state and the simulation entity instances based on the object identification information in the real object state dataset, bind the simulation entity instances to the corresponding real object states based on the mapping relationship, and update the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. The sensing unit 250 is used to perform virtual sensing based on the constructed virtual environment: performing sensor-level sampling on the virtual environment state data, generating virtual raw sensor data, and forming a virtual sensing result data stream; The verification unit 260 is used to construct a digital twin verification loop and verify the virtual perception result data stream through the twin verification loop.

[0118] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. For example... Figure 3 As shown, the electronic device may include: Memory 310 storing executable program code; Processor 320 coupled to memory 310; The processor 320 calls the executable program code stored in the memory 310 to execute some or all of the steps in the autonomous driving virtual-real combination digital twin method based on the full-link interface in Embodiment 1.

[0119] This invention discloses a computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps in a fully-linked interface-based virtual-real digital twin method for autonomous driving as described in Embodiment 1.

[0120] This invention also discloses a computer program product, wherein when the computer program product is run on a computer, the computer executes some or all of the steps in the autonomous driving virtual-real combined digital twin method based on a full-link interface in Embodiment 1.

[0121] This invention also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer executes some or all of the steps in the autonomous driving virtual-real combination digital twin method based on a full-link interface in Embodiment 1.

[0122] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0124] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0126] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0127] Those skilled in the art will understand that some or all of the steps in the various methods of the embodiments described can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0128] The foregoing has provided a detailed description of a method, apparatus, electronic device, and storage medium for autonomous driving based on a full-link interface, integrating virtual and real digital twins. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for creating a virtual-real integrated digital twin for autonomous driving based on a full-link interface, characterized in that, It includes the following: Transform multi-source perception data of autonomous vehicles in real physical environments into real object state datasets; Establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system; Based on the standardized state mapping, the real object state dataset is transformed into virtual environment state data that can be executed by the simulation engine. Constructing a virtual environment: Constructing a simulation entity pool, pre-generating multiple simulation entity instances in a hidden state, establishing a mapping relationship between real object states and simulation entity instances based on object identification information in the real object state dataset, binding the simulation entity instances to the corresponding real object states based on the mapping relationship, and updating the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. Virtual perception is performed based on the constructed virtual environment: sensor-level sampling is performed on the virtual environment state data to generate virtual raw sensor data, forming a virtual perception result data stream; A digital twin verification loop is constructed, and the virtual perception result data stream is verified through the digital twin verification loop.

2. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, Also includes: Based on the timestamp alignment mechanism, when the simulation step frequency is higher than the sensing data frequency, intermediate state data is generated by interpolation or dead reckoning, and the intermediate state data is input into the simulation system.

3. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, The dynamic scheduling includes: When a real target identifier in the sensing object list is bound to a simulated entity instance, the corresponding simulated entity instance is activated and its pose information is synchronized. When the real target identifier no longer appears in the list of perceived objects, the binding relationship between the simulated entity instance and the real target identifier is released, and the simulated entity instance is restored to a hidden state.

4. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, The process of converting multi-source perception data from autonomous vehicles in real physical environments into a real object state dataset includes: Multi-source sensor data from real vehicles is collected, and the BEVFormer algorithm is used to convert real-time multi-channel images into a 3D perception object list. The 3D perception object list is then standardized and encapsulated to generate a perception object data stream in a unified format.

5. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, Also includes: A communication gateway and ring buffer structure are constructed. Asynchronous data writing is achieved through an independent receiving thread, and synchronous data reading is achieved based on the simulation timestamp through an independent reading thread. Simulation input data frames are constructed through the nearest time frame matching mechanism.

6. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, The construction of the simulation entity pool includes: Based on historical real object state time series data, the traffic flow change prediction model is used to predict the trend of traffic target quantity change within a future preset time window, and the prediction result of traffic flow scale is obtained. Based on the prediction results and the real-time perceived object list, the virtual objects in the object pool are hierarchically scheduled and bound and mapped; the prediction results are used to plan the capacity of the object pool and pre-allocate simulation entity resources in advance. The object pool is a multi-level virtual object pool, which includes simulation entity pools of different levels. Different levels of simulation entity pools correspond to object pool levels with different model complexity levels.

7. The autonomous driving virtual-real hybrid digital twin method based on a full-link interface according to claim 1, characterized in that, The twin verification loop verifies the virtual perception result data stream, including: A consistency comparison analysis is performed between the virtual perception result data stream and the real object state dataset, and deviation correction parameters are generated. These deviation correction parameters are used to correct the position parameters, attitude parameters, or model parameters of the virtual simulation entity instance.

8. A fully integrated virtual-real digital twin system for autonomous driving based on a full-link interface, characterized in that, It includes: Data unit, used to transform multi-source perception data of autonomous vehicles in real physical environments into a real object state dataset; Standard cells are used to establish a standardized state mapping between the state space of the real system and the state space of the virtual simulation system. The conversion unit is used to convert the real object state dataset into virtual environment state data that can be executed by the simulation engine according to the standardized state mapping. The simulation unit is used to construct a virtual environment: it constructs a simulation entity pool, pre-generates multiple simulation entity instances in a hidden state, establishes a mapping relationship between the real object state and the simulation entity instances based on the object identification information in the real object state dataset, binds the simulation entity instances to the corresponding real object states based on the mapping relationship, and updates the pose state of the simulation entity instances to the spatial pose of their corresponding real objects through a pose state injection mechanism. The sensing unit is used to perform virtual sensing based on the constructed virtual environment: it performs sensor-level sampling on the virtual environment state data, generates virtual raw sensor data, and forms a virtual sensing result data stream; The verification unit is used to construct a digital twin verification loop and verify the virtual perception result data stream through the twin verification loop.

9. An electronic device, characterized in that, It includes: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the autonomous driving virtual-real combination digital twin method based on the full-link interface as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, wherein the computer program causes a computer to execute the autonomous driving virtual-real combined digital twin method based on a full-link interface as described in any one of claims 1-7.

Images