A vehicle-road cooperation data set generation method and system based on high-fidelity digital twinning

Abstract

The application discloses a kind of based on high fidelity digital twinning vehicle-road cooperation data set generation method and system, belong to automatic driving and vehicle-road cooperation technical field.The method is first based on unmanned aerial vehicle photogrammetry and is imported CARLA to construct digital twinning environment map, and is obtained robust spatial calibration matrix by multiple point cloud registration and mean fusion of simulation point cloud and real point cloud, realize scene high-precision spatial alignment;Then, based on real roadside observation feature, establish traffic participant generation rule, generate heterogeneous traffic participant model group;Then according to the real traffic flow space-time distribution, establish control rule and expand parameters, generate diversified basic traffic flow and dynamic background;On this basis, identify conflict sensitive space-time region, generate jam, rush, occlusion and other long tail interaction scenarios by directional generation of restricted behavior reconstruction;Finally, in the scene that has completed spatial alignment, configure multi-modal virtual sensor array, synchronously collect data and directly generate labeled information under unified coordinate frame, and construct vehicle-road cooperation multi-modal data set.The application improves the spatial consistency of simulation data and real scene, and enhances the long tail sample coverage capability.

Description

Technical Field

[0001] This invention relates to autonomous driving and vehicle-road cooperative technologies, and more specifically, to a method and system for generating vehicle-road cooperative datasets based on high-fidelity digital twins. Background Technology

[0002] With the development of autonomous driving and vehicle-road cooperative perception technologies, datasets based on roadside multi-sensor cooperative observation have become an important foundation for vehicle target detection, trajectory prediction, multi-target tracking, and cooperative perception model training. Compared to single-vehicle perspective data, roadside perspectives can cover a larger intersection area and provide more stable global traffic observation information. Therefore, constructing high-quality, multimodal, and scalable vehicle-road cooperative datasets around real intersections has become an important research direction in related fields.

[0003] In existing methods, one approach directly acquires images, point clouds, and trajectory information from real roadside sensors, with annotation done manually or semi-automatically. Another approach relies on digital twin simulation platforms or 3D scene reconstruction techniques to generate synthetic data. While the former can obtain realistic observation results, it suffers from high acquisition and annotation costs, difficulty in acquiring long-tail scenes, and complex multi-sensor time synchronization. The latter, while capable of generating large amounts of data at a lower cost, often faces issues such as inconsistencies in coordinate systems between simulated and real scenes, distorted object distribution, and limited interactive behaviors. This leads to spatial mismatch and distribution shifts in the generated dataset when jointly trained with real data.

[0004] Especially in intersection-level vehicle-road cooperative scenarios, digital twin maps obtained using UAV photogrammetry or other reconstruction methods, when imported into simulation platforms, often exhibit scale, translation, and rotation biases due to differences in the definition of the simulated world coordinate system and the real world coordinate system. Relying solely on the default coordinates during model import, 2D map scale conversion, or single manual point selection calibration makes it susceptible to the influence of map import coordinate definitions, manual point selection errors, and local geometric differences in the point cloud. This makes it difficult to ensure that the map point cloud in CARLA and the point cloud collected at the real intersection are stably consistent within the same spatial framework, further affecting the accuracy of subsequent traffic participant generation, traffic flow construction, virtual sensor deployment, and ground truth label output.

[0005] Furthermore, existing simulation data generation methods, in terms of traffic participant modeling, often tend to randomly deploy data from a pre-set asset library, failing to fully reflect the frequency distribution, visibility scale distribution, and occlusion relationship distribution of real intersections from a roadside perspective. This results in insufficient coverage of low-frequency categories, anomalous configurations, and target samples under occlusion conditions. Regarding traffic flow generation, many solutions can only reproduce relatively conventional car-following and passage processes, lacking the ability to extend and configure entry timing, speed changes, path combinations, weather, lighting, and time conditions under the constraints of real traffic patterns, making it difficult to obtain basic traffic flow samples that are both realistic and diverse.

[0006] In complex intersection environments, what truly impacts the robustness of vehicle-road cooperative perception models is often not ordinary traffic flow scenarios, but rather long-tail interaction scenarios such as cutting in, overtaking, sudden occlusion, delayed avoidance, and intersection interference. Existing datasets, whether generated through real-world collection or conventional simulation methods, struggle to stably and controllably obtain samples of these scenarios. Furthermore, during the simulation data output stage, if the front-end scenario lacks unified spatial alignment, additional coordinate corrections are required for the collected data and annotation information, complicating the process and potentially introducing new errors. Therefore, there is an urgent need for a vehicle-road cooperative dataset generation method and system that can first achieve high-precision alignment between the digital twin scenario and the real intersection space, then generate heterogeneous traffic participants, diverse basic traffic flows, and long-tail interaction scenarios within a unified coordinate framework, and directly output multimodal data and annotation information. Summary of the Invention

[0007] The purpose of this invention is to address the problems existing in current vehicle-road cooperative dataset generation methods, such as inconsistencies between the spatial coordinates of simulated and real-world scenarios, distorted distribution of traffic participants, insufficient coverage of long-tail interaction scenarios, and the difficulty in directly outputting multimodal data and annotation information within a unified coordinate framework. This invention provides a method and system for generating vehicle-road cooperative datasets based on high-fidelity digital twins. By first performing high-precision spatial alignment between the digital twin scenario and the real intersection scenario, and then generating traffic participants, basic traffic flow, and long-tail interaction scenarios within a unified coordinate framework and simultaneously collecting multimodal data, a multimodal dataset more suitable for training vehicle-road cooperative perception models can be constructed.

[0008] The technical solution disclosed in this invention is as follows: A method for generating vehicle-road cooperative datasets based on high-fidelity digital twins, comprising the following steps:

[0009] Step 1: Construct a digital twin environment map at the intersection level based on UAV multi-source photogrammetric data. After cleaning, scaling, and road network drawing, import the digital twin environment map into the CARLA simulation world. Addressing the spatial discrepancy between the simulation map and the real map due to different coordinate definition methods, collect point clouds from the simulation map and the real intersection in CARLA. Import the simulation map point clouds and the real intersection point clouds into a point cloud registration tool. Select at least three pairs of non-collinear, identical feature points from each of the two point clouds. Calculate the rotation-translation matrix from the simulation map point cloud to the real intersection point cloud based on these identical feature points, and use this matrix as the reference. A spatial calibration matrix is ​​generated by performing multiple point selection and registration operations on the corresponding feature points to obtain multiple candidate rotation-translation matrices. These candidate rotation-translation matrices are then subjected to mean fusion and residual verification to obtain a robust spatial calibration matrix. The pose of the digital twin environment map in CARLA is adjusted using this robust spatial calibration matrix to achieve a unified mapping between the simulated world coordinate system and the real world coordinate system. An initial road network mesh is constructed using a road network generation tool, and the initial road network mesh is adaptively fine-tuned based on micro-terrain undulation data from the real scene to generate a three-dimensional lane-level road network with topological consistency and physical realism, thus completing the construction of a high-fidelity digital twin base.

[0010] Step 2: Based on the high-fidelity digital twin base and spatial calibration matrix constructed in Step 1, the traffic participants observed by the roadside sensors at the real intersection are mapped into the digital twin scene. The frequency of occurrence, visibility scale, and occlusion relationship of various traffic participants from the roadside perspective are statistically analyzed to establish traffic participant generation rules. According to the traffic participant generation rules, vehicles, pedestrians, non-motorized vehicles, and road-related targets are combined and generated. Low-frequency categories and abnormal configurations are enhanced to generate a heterogeneous traffic participant model group that conforms to the roadside observation characteristics of the target intersection.

[0011] Step 3: Based on the spatial calibration matrix obtained in Step 1 and the heterogeneous traffic participant model group generated in Step 2, the spatiotemporal distribution information of traffic flow observed by roadside sensors at the real intersection is mapped to the digital twin scene. Based on this, the entry sequence, passage path, speed change, relative spacing, and conventional following and yielding control rules for traffic participants are established. Under the constraints of the control rules, the deployment density, entry sequence, running speed, and path combination of the heterogeneous traffic participant model group are extended to generate a basic traffic flow that conforms to the roadside observation patterns of the target intersection and has diversity. At the same time, combined with the weather conditions, lighting conditions, and time slice characteristics observed at the real intersection, the weather parameters, lighting parameters, and time parameters in the digital twin scene are extended and configured to generate a dynamic traffic background with diverse environmental conditions.

[0012] Step 4: During the basic traffic flow evolution generated in Step 3, identify conflict-sensitive spatiotemporal regions within the target intersection, and perform restricted behavior reconstruction on traffic participants within these regions. The restricted behavior reconstruction includes adjusting the entry sequence, traffic priority relationship, local path and speed changes of traffic participants, and strengthening the control of local occlusion relationships. Under the premise of satisfying road topology constraints and kinematic constraints, one or more long-tail interaction scenarios among cutting in, overtaking, occlusion emergence, delay avoidance and cross-interference are generated in a targeted manner.

[0013] Step 5: Based on the digital twin scene after spatial alignment in Step 1 and the parameters of the roadside sensors at the real intersection, configure a multimodal virtual sensor array in the digital twin scene; according to the position parameters, attitude parameters, and intrinsic parameters of the real roadside sensors, set the virtual sensor array accordingly; perform multi-view, multimodal synchronous data acquisition on the basic traffic flow scene generated in Step 3 and the effective long-tail interaction scene generated in Step 4, and directly generate the target's coordinate information, point cloud information, and annotation information under the unified coordinate framework to construct a vehicle-road cooperative multimodal dataset.

[0014] Further, the determination of the spatial calibration matrix in step one specifically includes: using the digital twin environment map imported into the CARLA simulation world as the simulation map, and collecting the simulation map point cloud corresponding to the simulation map; obtaining the real intersection point cloud corresponding to the target real intersection; importing the simulation map point cloud and the real intersection point cloud into the same point cloud registration tool, and selecting at least three pairs of non-collinear homonymous feature points in each of the two sets of point clouds; and determining the rotation-translation matrix composed of rotation matrix and translation vector based on the three-dimensional correspondence between the homonymous feature points. The rotation-translation matrix is ​​used to characterize the spatial mapping relationship between the CARLA simulation world coordinate system and the real world coordinate system, and serves as the spatial calibration matrix.

[0015] Furthermore, the corresponding feature points include at least one of the following: curb intersections, lane line intersections, stop line endpoints, traffic island corners, road ancillary facility corners, or building boundary corners, which can be stably identified in both the simulated map point cloud and the real intersection point cloud; each registration selects at least three pairs of corresponding feature points that are not collinear and are distributed in different areas of the target intersection, so as to simultaneously constrain translational deviation, rotational deviation, and scale deviation.

[0016] Furthermore, the spatial calibration matrix is ​​obtained through multiple point registrations: multiple sets of three or more pairs of non-collinear, same-name feature points are repeatedly selected in the point cloud registration tool, and multiple candidate rotation-translation matrices are calculated respectively; the registration residuals corresponding to each candidate rotation-translation matrix are statistically analyzed, and abnormal candidate rotation-translation matrices with residuals greater than a preset threshold are removed; the mean of the translation components in the retained candidate rotation-translation matrices is calculated, and the rotation components are subjected to mean fusion and normalization processing to obtain the final spatial calibration matrix.

[0017] Furthermore, adjusting the digital twin environment map in CARLA using the final spatial calibration matrix includes: transforming the overall position and orientation of the digital twin environment map in CARLA according to the final spatial calibration matrix, so that the map point cloud generated in CARLA coincides or nearly coincides with the point cloud collected from the real intersection under a unified coordinate framework; and after the adjustment is completed, the CARLA map point cloud is collected again, and an error check is performed between it and the point cloud collected from the real intersection. When the average distance error of the corresponding feature area of ​​the point cloud meets the preset threshold, the spatial alignment is confirmed to be complete.

[0018] Furthermore, before importing the digital twin environment map into the CARLA simulation world, the process includes: using 3D modeling software to crop, denoise, organize coordinates, and lightweight the model of the real intersection map reconstructed by the UAV; using RoadRunner or other parametric road network generation tools to draw the corresponding road network based on the real intersection's road boundaries, lane lines, stop lines, curbs, and traffic channelization structures; and importing the processed map model and road network file into CARLA to reproduce the digital twin scene of the target real intersection.

[0019] Furthermore, after spatial alignment is completed, the point cloud data subsequently collected by virtual LiDAR or other virtual sensors in CARLA is under the same spatial reference frame as the point cloud collected at the real intersection; the same spatial reference frame is used to support the subsequent generation of traffic participants, the spatiotemporal distribution mapping of traffic flow, the deployment of virtual sensors, and the direct generation of target labeling information.

[0020] Furthermore, the establishment of traffic participant generation rules in step two specifically includes: based on the spatial calibration matrix obtained in step one, uniformly mapping the traffic participants observed by the roadside sensors at the real intersection to the digital twin scene coordinate system; statistically analyzing the frequency of occurrence, visible scale, and occlusion relationship of various traffic participants from the roadside perspective under the unified coordinate system, and establishing traffic participant generation rules based on the statistical results.

[0021] Furthermore, the traffic participant generation rules are used to constrain the category selection, visibility scale configuration, and occlusion relationship configuration of traffic participants in the digital twin scenario, so that the generated traffic participants are consistent with the real observed features of the target intersection from the roadside perspective.

[0022] Furthermore, the enhancement of low-frequency categories and abnormal configurations specifically includes: under the premise of maintaining the overall distribution constraints of roadside observations at the target intersection, targeted compensation is performed to generate traffic participant categories with occurrence frequencies below a preset threshold and traffic participant configurations that deviate from the conventional roadside view morphology distribution, so as to improve the coverage of long-tail traffic participant samples in the digital twin scenario.

[0023] Furthermore, the establishment of traffic flow spatiotemporal distribution information in step three specifically includes: based on the spatial calibration matrix obtained in step one, uniformly mapping the movement trajectories of traffic participants observed by roadside sensors at real intersections to the digital twin scene coordinate system; performing statistical analysis on the mapped movement trajectories of traffic participants to obtain the entry time sequence distribution, travel path distribution, speed change distribution, and relative spacing distribution of various traffic participants; and establishing traffic flow control rules based on the entry time sequence distribution, travel path distribution, speed change distribution, and relative spacing distribution.

[0024] Furthermore, the traffic flow control rules are used to constrain the entry order, travel path, speed changes, car-following relationship, and yielding relationship of the heterogeneous traffic participant model group in the digital twin scenario. Under the constraints of the traffic flow control rules, the parameters of the deployment density, entry sequence, running speed, and path combination of traffic participants are extended to generate diversified basic traffic flows that conform to the roadside observation patterns of the target intersection.

[0025] Furthermore, the environmental configuration specifically includes: determining the value range of weather parameters, illumination parameters, and time parameters in the digital twin scene based on the weather conditions, illumination conditions, and time slice characteristics observed at the real intersection, and expanding the configuration within the value range to generate dynamic traffic backgrounds under different weather conditions, illumination conditions, and time periods.

[0026] Furthermore, the identification of conflict-sensitive spatiotemporal areas in step four specifically includes: based on the basic traffic flow generated in step three, jointly analyzing the running trajectories, arrival times, relative distances, and relative speeds of different traffic participants within the target intersection, determining potential trajectory intersection areas and the degree of temporal overlap of traffic participants within the potential trajectory intersection areas, and identifying areas with a temporal overlap degree higher than a preset threshold as conflict-sensitive spatiotemporal areas.

[0027] Furthermore, the restricted behavior reconstruction includes adjusting the entry timing, traffic priority relationship, local path, speed change and local occlusion relationship of traffic participants in the conflict-sensitive spatiotemporal area, so as to make controlled changes in the temporal relationship, spatial relationship and occlusion relationship between traffic participants to form a long-tail interaction process.

[0028] Furthermore, the long-tail interaction scenarios include one or more of the following: cutting in line, overtaking, delayed avoidance, sudden occlusion, and cross-interference scenarios; the restricted behavior reconstruction is performed under the premise of satisfying road topology constraints and traffic participant kinematic constraints to ensure the physical feasibility of the generated scenarios.

[0029] Furthermore, the restricted behavior reconstruction specifically includes: constructing multiple candidate reconstruction schemes for target traffic participants located in conflict-sensitive spatiotemporal areas, wherein the candidate reconstruction schemes include at least an entry timing adjustment scheme, a traffic priority relationship adjustment scheme, a local path offset scheme, a speed change adjustment scheme, and a local occlusion enhancement scheme; calculating the impact of each candidate reconstruction scheme on the degree of temporal overlap, relative proximity, and occlusion enhancement, and selecting the candidate reconstruction scheme that increases the degree of conflict exposure as the execution scheme, provided that road topology constraints and traffic participant kinematic constraints are satisfied.

[0030] Furthermore, the local occlusion enhancement scheme includes: adjusting the relative positional relationship, relative temporal relationship, or relative speed relationship between the target traffic participant and the occluded traffic participant, so that the visible area of ​​the target traffic participant is reduced, the visible duration is shortened, or the occlusion period is extended from the roadside perspective, thereby forming scenarios of sudden occlusion, rushing after occlusion, or delayed avoidance after occlusion.

[0031] Furthermore, the degree of conflict exposure is jointly determined based on the degree of temporal compression between traffic participants before and after the implementation of the candidate reconstruction scheme, the degree of change in minimum relative distance, and the degree of reduction in visible area from the roadside perspective. The degree of temporal compression characterizes the reduction in the time interval between different traffic participants entering the potential trajectory intersection area; the degree of change in minimum relative distance characterizes the change in the closest approach of different traffic participants within the conflict-sensitive spatiotemporal region; and the degree of reduction in visible area characterizes the reduction in the visible outline range or the shortening of the visible duration of the target traffic participant from the roadside perspective. Under the premise of satisfying road topology constraints and traffic participant kinematic constraints, the candidate reconstruction scheme that increases the degree of temporal compression, decreases the minimum relative distance, and increases the degree of visible area reduction is selected as the execution scheme.

[0032] Furthermore, the screening of candidate reconstruction schemes also includes a feasibility verification of the operational status of the reconstructed traffic participants. The feasibility verification includes at least lane boundary constraint verification, speed continuity verification, acceleration change constraint verification, and steering continuity verification. Candidate reconstruction schemes that do not meet the feasibility verification conditions are eliminated.

[0033] Furthermore, the triggering conditions for the occlusion sudden occurrence scenario include: the visible area of ​​the target traffic participant from the roadside view is lower than a preset threshold, the relative temporal interval between the target traffic participant and the occluding traffic participant is lower than a preset threshold, and the time when the target traffic participant enters the potential trajectory intersection area after the occlusion is removed meets a preset conflict window condition; when the above triggering conditions are met, the entry timing adjustment or speed change adjustment is performed on the target traffic participant to form the occlusion sudden occurrence scenario.

[0034] Furthermore, the generation of the obstruction-induced rushing scenario and the obstruction-induced delayed avoidance scenario specifically includes: maintaining the potential conflict relationship between the target traffic participant and other traffic participants during the continuous period when the target traffic participant is obstructed; after the obstruction is lifted, reconstructing the passage order and speed of the target traffic participant or related traffic participants according to preset priority relationship adjustment rules and speed change rules, so that the target traffic participant forms an interactive process of rushing into the conflict area or delaying avoidance after the obstruction is lifted.

[0035] Furthermore, the generation of the cross-interference scenario specifically includes: for multiple traffic participants with potential trajectory intersection relationships, based on their temporal relationship and relative path relationship when entering the conflict-sensitive spatiotemporal region, adjusting the entry time, passage order or local path of at least one traffic participant, so that multiple traffic participants form an interactive process of temporal compression and spatial overlap enhancement in the conflict-sensitive spatiotemporal region, thereby generating a cross-interference scenario.

[0036] Furthermore, the screening of the cross-interference scenarios includes: jointly evaluating the temporal compression degree, minimum relative distance change degree, and occlusion enhancement degree of multiple traffic participants in the conflict-sensitive spatiotemporal region after reconstruction; when the joint evaluation result is higher than a preset threshold and meets the road topology constraints and kinematic constraints, the corresponding scenario is determined as an effective long-tail interaction scenario.

[0037] Furthermore, the effective long-tail interaction scenarios, along with the corresponding traffic participant trajectory information, occlusion relationship information, conflict-sensitive spatiotemporal area information, and scene type identification information, are stored and used as a scene sample set for subsequent multimodal perception data collection and ground truth label generation.

[0038] Furthermore, the configuration of the multimodal virtual sensor array in step five specifically includes: in the digital twin scene after spatial alignment is completed in step one, the virtual camera, virtual LiDAR, and virtual depth sensor are configured accordingly based on the installation position, orientation parameters, and internal parameters of the roadside sensors at the real intersection.

[0039] Furthermore, the multimodal virtual sensor array is synchronously triggered under a unified simulation clock to synchronously collect the basic traffic flow scene generated in step three and the effective long-tail interaction scene generated in step four, so as to obtain time-corresponding image data, point cloud data and depth data.

[0040] Furthermore, the image data, point cloud data, depth data, and target annotation information are directly generated within the unified coordinate framework after spatial alignment in step one. The target annotation information includes at least the target location, target size, target category, and target trajectory information.

[0041] Furthermore, the perception data and target annotation information in the vehicle-road cooperative multimodal dataset maintain a correspondence within the same coordinate frame, thus eliminating the need for secondary coordinate correction of the collected perception data and target annotation information.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] I. To address the shortcomings of default coordinate import, 2D scale conversion, or single manual point selection calibration in ensuring stable point cloud consistency, this invention imports the simulated map point cloud collected in CARLA and the point cloud collected at the real intersection into a point cloud registration tool. By selecting three or more pairs of non-collinear feature points with the same name from the two sets of point clouds multiple times, multiple rotation-translation matrices are obtained. The mean fusion and residual verification of the multiple rotation-translation matrices are then performed to obtain a robust spatial calibration matrix. This matrix is ​​used to adjust the pose of the digital twin environment map in CARLA, achieving high-precision spatial alignment between the digital twin scene and the real intersection scene, and reducing spatial mismatch caused by coordinate inconsistency.

[0044] Second, addressing the issue that real traffic participants exhibit distinct distribution characteristics in terms of frequency of occurrence, visibility scale, and occlusion relationships from a roadside perspective, this invention maps traffic participants observed by roadside sensors at real intersections into a digital twin scene based on a spatial calibration matrix. It then statistically analyzes their observation characteristics from a roadside perspective to establish traffic participant generation rules. Based on this, vehicles, pedestrians, non-motorized vehicles, and road-related targets are combined and generated, and low-frequency categories and anomalous configurations are enhanced. This results in a heterogeneous traffic participant model group that conforms to the roadside observation characteristics of real intersections. By establishing traffic participant generation rules based on the roadside observation characteristics of real intersections, the coverage of low-frequency categories and anomalous configuration samples can be improved, mitigating the homogenization problem of traffic participants. This makes the generated heterogeneous traffic participant model group more consistent with the distribution characteristics of real intersections, enhancing the authenticity of the dataset.

[0045] Third, this invention establishes entry sequence, travel path, speed change, relative spacing, and conventional yielding control rules for traffic participants based on the spatiotemporal distribution information of traffic flow observed by roadside sensors at real intersections. Under the constraints of these control rules, the parameters of traffic participant deployment density, entry sequence, operating speed, and path combination are expanded. Simultaneously, by combining weather conditions, lighting conditions, and time slice characteristics observed at real intersections, the weather parameters, lighting parameters, and time parameters in the digital twin scenario are expanded and configured, thereby generating basic traffic flow and dynamic traffic background that conform to the observation patterns of the target intersection and have diversity, significantly improving the diversity of the dataset.

[0046] Fourth, by identifying conflict-sensitive spatiotemporal regions and performing restricted behavior reconstruction, it is possible to generate long-tail interaction scenarios such as cutting in line, overtaking, and sudden occlusion, which effectively makes up for the problem of sparse long-tail samples in real collected data and improves the coverage of complex boundary samples.

[0047] Fifth, by directly configuring virtual sensor arrays and outputting annotation information in a spatially aligned digital twin scenario, the complexity of post-processing can be reduced and the consistency between multimodal data and annotation information can be improved. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the overall process of a method for generating vehicle-road cooperative datasets based on high-fidelity digital twins, provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram illustrating the principle of RT matrix acquisition and map pose adjustment based on the registration of three points between simulated map point cloud and real intersection point cloud, as provided in an embodiment of the present invention.

[0050] Figure 3 This is a diagram illustrating vehicle types.

[0051] Figure 4 This is a weather forecast diagram.

[0052] Figure 5 This is a schematic diagram of a traffic scenario. Detailed Implementation

[0053] The technical solution of the present invention will be further described in detail below with reference to specific embodiments. Those skilled in the art should understand that the following embodiments are only for explaining the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0054] Example 1

[0055] This embodiment uses a typical urban intersection as an example to illustrate a method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins. The intersection includes motor vehicle lanes, non-motor vehicle lanes, pedestrian crossings, traffic lights, curb boundaries, traffic signs, and road ancillary facilities. First, a digital twin environment map corresponding to the real intersection is constructed based on multi-source photogrammetric data from unmanned aerial vehicles (UAVs). Then, the spatial calibration matrix is ​​obtained, traffic participant generation is performed, basic traffic flow and environment expansion are completed, long-tail interaction scene generation is performed, and multimodal synchronous acquisition and annotation output are completed, ultimately forming a vehicle-road cooperative multimodal dataset.

[0056] Step 1: Digital Twin Environment Map Construction and RT Matrix Calculation

[0057] In this embodiment, firstly, a drone is used to conduct multi-flight, multi-view photogrammetry of the target intersection. Preferably, a DJI drone is used to acquire overhead images, oblique images, and control point measurement information covering the entire intersection area. Based on the multi-source photogrammetry data, 3D reconstruction is performed to obtain an initial digital twin environment map. The initial digital twin environment map includes at least an intersection surface model, a point cloud model, and road structure information. Subsequently, Blender is used to crop, denoise, lightweight the model, and organize the coordinates of the initial digital twin environment map. RoadRunner is then used to draw the corresponding road network based on the road boundaries, lane lines, stop lines, and curb structures of the actual intersection. The processed map and road network are then imported into CARLA to reproduce the actual target intersection.

[0058] Because the internal coordinate definition of the digital twin environment map imported into CARLA has scale, position, and orientation deviations compared to the real-world coordinate system of the actual intersection, this embodiment further calculates the rotation-translation matrix, i.e., the RT matrix, used for map pose adjustment. Specifically, point cloud acquisition is performed on the imported digital twin environment map in CARLA to obtain a simulated map point cloud; simultaneously, the point cloud of the target real intersection is acquired. The simulated map point cloud and the real intersection point cloud are imported into CloudCompare together, and at least three pairs of non-collinear homonymous feature points are selected from each of the two point clouds. The homonymous feature points are preferably curb intersections, lane line intersections, stop line endpoints, traffic island corners, or corners of road ancillary facilities.

[0059] In CloudCompare, 3D rigid registration is performed based on selected corresponding feature points to obtain the RT matrix from the CARLA simulation map point cloud to the real intersection point cloud. To reduce the impact of single manual point selection errors on the registration results, this embodiment repeats the point selection and registration multiple times, selecting three or more pairs of non-collinear corresponding feature points from different regions each time, obtaining multiple candidate RT matrices. Then, the registration residuals corresponding to each candidate RT matrix are statistically analyzed, candidate RT matrices with abnormal residuals are eliminated, and the translation components of the retained candidate RT matrices are averaged, while the rotation components are averaged and normalized to obtain the final RT matrix. The final RT matrix is ​​used to adjust the overall position and orientation of the digital twin environment map in CARLA, so that the re-acquired CARLA map point cloud and the real intersection point cloud are consistent within the corresponding road boundaries, lane lines, and intersection structure areas. Through the above method, the digital twin environment map and the real intersection can achieve robust spatial alignment within a unified coordinate framework.

[0060] After completing spatial alignment, the initial road network grid is further adjusted by combining the parameterized road network generated by RoadRunner with the micro-terrain undulation data of the real scene. This ensures that the lane-level road network in CARLA is consistent with the real intersection in terms of topological relationships, planar geometry, and elevation changes, thereby completing the construction of a high-fidelity digital twin base.

[0061] Step 2: Establishing Traffic Participant Generation Rules and Generating Heterogeneous Traffic Participant Model Clusters

[0062] After spatial alignment is completed in step one, traffic participants observed by roadside sensors at the real intersection are mapped to the digital twin scene coordinate system based on the spatial calibration matrix. The frequency of occurrence, visibility scale, and occlusion relationships of various traffic participants from the roadside view are then statistically analyzed to establish traffic participant generation rules. These rules constrain the selection of traffic participant categories, visibility scale configuration, and occlusion relationship configuration in the digital twin scene.

[0063] Based on the aforementioned traffic participant generation rules, vehicles, pedestrians, non-motorized vehicles, and roadside targets are generated in combination, and low-frequency categories and abnormal configurations are enhanced. Specifically, low-frequency categories are preferably traffic participant categories whose occurrence frequency is below a preset threshold; abnormal configurations are preferably traffic participant configurations that deviate from the conventional roadside view morphology distribution. While maintaining the overall distribution constraints of roadside observations at the target intersection, the generation ratio of low-frequency categories and abnormal configurations in the digital twin scenario is increased to form a heterogeneous traffic participant model group that conforms to the roadside observation characteristics of the target intersection.

[0064] In this embodiment, the simulation asset library was expanded to enhance the diversity of traffic participants. Based on corresponding experimental data, the object categories, including vehicles, trucks, buses, vans, motorcycles, bicycles, and pedestrians, were all expanded. Figure 3 The number of other assets, such as Car, Truck, Van, and Bus, has increased significantly compared to the default asset pool, thus providing a richer sample of participants for subsequent basic traffic flow construction and long-tail interaction scenario generation.

[0065] Step 3: Generation of Basic Traffic Flow and Dynamic Traffic Background

[0066] Based on the spatial calibration matrix obtained in step one and the heterogeneous traffic participant model group generated in step two, the spatiotemporal distribution information of traffic flow observed by roadside sensors at real intersections is mapped to the digital twin scenario, and the entry order, passage path, speed change, relative spacing, and conventional following and yielding control rules of traffic participants are established accordingly.

[0067] like Figure 4 Under the constraints of the control rules, the deployment density, entry timing, operating speed, and path combinations of the heterogeneous traffic participant model group are extended to generate a diverse basic traffic flow that conforms to the roadside observation patterns of the target intersection. Unlike simply replicating the original traffic flow at a real intersection, this embodiment extends the parameters under the constraints of real observation patterns to form multiple traffic states ranging from low to high traffic load. In this way, different basic traffic flow samples, such as free flow, steady flow, and locally congested flow, can be obtained.

[0068] Simultaneously, by combining weather conditions, lighting conditions, and time-slice characteristics observed at real intersections, the weather, lighting, and time parameters in the digital twin scenario are expanded and configured to generate dynamic traffic backgrounds with diverse environmental conditions. Preferably, weather parameters include sunny, cloudy, light rain, heavy rain, and foggy weather; lighting parameters include early morning, morning, noon, afternoon, evening, and night; and time parameters include peak hours and off-peak hours. Through the above configuration, multiple environmental combination samples can be generated in the same intersection scenario.

[0069] Step 4: Identification of Conflict-Sensitive Spatiotemporal Regions and Generation of Effective Long-Tail Interaction Scenarios

[0070] In the basic traffic flow evolution process generated in step three, conflict-sensitive spatiotemporal regions within the target intersection are identified. Specifically, based on the running trajectories, arrival times, relative distances, and relative speeds of different traffic participants within the target intersection, joint analysis is performed to determine potential trajectory intersection areas and the degree of temporal overlap of traffic participants within these potential trajectory intersection areas. Regions with a temporal overlap degree higher than a preset threshold are identified as conflict-sensitive spatiotemporal regions.

[0071] Restricted behavior reconstruction is performed on traffic participants located within the conflict-sensitive spatiotemporal region. This restricted behavior reconstruction includes adjusting the entry timing, traffic priority relationships, local paths, speed changes, and local occlusion relationships of traffic participants to controllably change the temporal, spatial, and occlusion relationships between them, thereby forming a long-tailed interaction process. Preferably, the restricted behavior reconstruction can construct multiple candidate reconstruction schemes, calculate the impact of each candidate scheme on the degree of temporal compression, the degree of change in minimum relative distance, and the degree of occlusion enhancement, and, under the premise of satisfying road topology constraints and kinematic constraints, select the candidate reconstruction scheme that increases the degree of conflict exposure as the execution scheme.

[0072] like Figure 5 Through the above methods, one or more long-tail interaction scenarios can be generated in a targeted manner, including scenarios such as cutting in line, overtaking, delayed avoidance, sudden occlusion, and cross-interference. For the generated long-tail interaction scenarios, a joint evaluation is performed based on the degree of temporal compression, the degree of change in minimum relative distance, and the degree of occlusion enhancement. When the joint evaluation result is higher than a preset threshold and satisfies road topology and kinematic constraints, the corresponding scenario is determined as a valid long-tail interaction scenario. This scenario, along with traffic participant trajectory information, occlusion relationship information, conflict-sensitive spatiotemporal area information, and scenario type identification information, is stored as a scenario sample set for subsequent multimodal perception data collection and ground truth label generation.

[0073] Step 5: Configure the multimodal virtual sensor array, and directly generate multimodal synchronous acquisition and annotation information.

[0074] Since step one has already completed the spatial alignment between the digital twin scene and the real intersection scene, step five no longer requires secondary coordinate correction of the collected perception data and target annotation information. Instead, a multimodal virtual sensor array is directly configured within the aligned digital twin scene. Specifically, the virtual camera, virtual LiDAR, and virtual depth sensor are configured according to the installation location, orientation parameters, and internal parameters of the roadside sensors at the real intersection.

[0075] Preferably, in this embodiment, each scene is configured with four RGB cameras, four depth cameras, four 128-beam LiDARs, one GNSS module, and one IMU module. The RGB cameras and depth cameras each have a resolution of 1920×1080, a field of view of 60°, and a sampling frequency of 10Hz. The 128-beam LiDARs have a field of view of 360°×50°, a sampling frequency of 10Hz, a detection range of 160m, and a point cloud output capability of approximately 2.4M points / s.

[0076] The multimodal virtual sensor array is synchronously triggered under a unified simulation clock, performing multi-view, multimodal synchronous acquisition of the basic traffic flow scene generated in step three and the effective long-tail interaction scene generated in step four to obtain time-corresponding image data, point cloud data, and depth data. The image data, point cloud data, depth data, and target annotation information are directly generated within a unified coordinate framework after spatial alignment in step one. The target annotation information includes at least target location, target size, target category, and target trajectory information. Therefore, the perception data and target annotation information in the vehicle-road cooperative multimodal dataset maintain a correspondence within the same coordinate framework, eliminating the need for secondary coordinate correction of the acquired perception data and target annotation information.

[0077] Example 2: Dataset Generation Results and Effect Verification

[0078] Based on the method in Example 1, a dataset was constructed. This dataset is built upon digital twin scenarios of four real intersections, using digital twin maps consistent with the real intersection spaces. It possesses a unified coordinate framework, a consistent road network structure, an expanded traffic participant asset library, the ability to configure diverse traffic flow and environmental parameters, and the ability to generate effective long-tail interaction scenarios. According to the corresponding statistical results in the paper, the constructed dataset has 150+ expanded traffic participant assets, a label size of 70k, a bounding box size of 15M, and a frame size of 70k×3.

[0079] In this embodiment, Tables 1 and 2 use the same 3D target detection evaluation metrics as the V2X-I benchmark. BEV AP represents the average accuracy of the bird's-eye view, which is calculated by projecting the predicted 3D detection boxes and the ground truth 3D annotation boxes onto the bird's-eye view plane, determining the matching relationship according to the intersection-union ratio (IU) threshold of the corresponding category, and calculating the average accuracy based on the precision-recall curves at different confidence levels. 3D AP represents the average accuracy of 3D detection, which is calculated directly based on the 3D intersection-union ratio between the predicted 3D detection boxes and the ground truth 3D annotation boxes. IoU represents the intersection-union ratio. In this embodiment, following the V2X-I table settings, IoU=0.5 is used as the matching threshold for the Vehicle category, and IoU=0.25 is used for the Cyclist and Pedestrian categories. Easy, Middle, and Hard represent different detection difficulty levels in the benchmark dataset, reflecting the target detection performance under different conditions of observability, occlusion, distance, or point cloud / image observation sufficiency. Under the same category, difficulty level, and evaluation metric, a higher value indicates better detection performance.

[0080] Table 1

[0081]

[0082] Table 2

[0083]

[0084] To verify the supporting effect of the dataset generated in this invention on the training of the vehicle-road cooperative perception model, the single-modal detector BEVHeight and the multi-modal detector MVXNet were selected for pre-training transfer validation on the V2X-I benchmark dataset. Experimental results show that after pre-training with the dataset generated in this invention, BEVHeight outperforms the scheme that uses only the target dataset for training in the vehicle category in both BEV AP and 3D AP; it also achieves better results in the pedestrian category. The corresponding results are given in Table 1.

[0085] Furthermore, the validation results using MVXNet demonstrate that the dataset generated in this invention also exhibits good transferability in multimodal fusion detection tasks. For categories such as Vehicle, Cyclist, and Pedestrian, superior detection results were achieved across multiple difficulty levels, as shown in Table 2. These results indicate that this invention not only generates a geometrically consistent, well-distributed, and long-tailed vehicle-road cooperative multimodal dataset, but also serves as an effective pre-training resource for both single-modal and multimodal detection models.

[0086] In summary, this invention constitutes a complete technical chain from digital twin scene construction, spatial alignment, traffic participant expansion, basic traffic flow generation, long-tail interaction scene construction to multimodal synchronous acquisition and annotation output through steps one to five. It can generate vehicle-road cooperative multimodal datasets with high fidelity and high diversity under a unified coordinate framework.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating vehicle-road cooperative datasets based on high-fidelity digital twins, characterized in that, include: Step 1: Construct a digital twin environment map based on UAV photogrammetry and import it into CARLA; collect point clouds of simulated and real intersections, select non-collinear homonymous feature points multiple times for registration, obtain a spatial calibration matrix through mean fusion and residual verification, adjust the map pose to achieve coordinate system one mapping; combine real terrain undulation data to perform adaptive elevation fine-tuning, generate a three-dimensional lane-level road network, and complete the construction of the digital twin base. Step 2: Based on the spatial calibration matrix, the traffic participants observed on the real roadside are mapped to the twin scene, and their occurrence frequency, visibility scale and occlusion relationship are statistically analyzed to establish generation rules; vehicles, pedestrians, non-motorized vehicles and road-related targets are generated according to the rules, and low-frequency categories and abnormal configurations are enhanced to generate a heterogeneous traffic participant model group. Step 3: Based on the spatial calibration matrix and heterogeneous model group, the spatiotemporal distribution of real traffic flow is mapped to the twin scene, and entry order, passage path, speed change and following and yielding control rules are established; under the constraints of these rules, the parameters of deployment density, time sequence, speed and path are expanded to generate diversified basic traffic flow, and combined with real weather, illumination and time characteristics to expand the configuration and generate dynamic traffic background. Step 4: In the basic traffic flow evolution, jointly analyze the trajectory and identify the conflict-sensitive spatiotemporal region; perform restricted behavior reconstruction on the traffic participants in the region, adjust the temporal sequence, priority relationship, path, speed and occlusion relationship, and generate one or more long-tail interaction scenarios such as cutting in, overtaking, occlusion emergence, delay avoidance and cross interference under the topological and kinematic constraints. Step 5: Configure a multimodal virtual sensor array in the spatially aligned scene according to the real sensor parameters, and collect data synchronously from multiple perspectives and in multiple modes on the basic traffic flow and long-tail interaction scene under a unified simulation clock. Then, directly generate target annotation information under a unified coordinate framework to construct a vehicle-road cooperative multimodal dataset.

2. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The specific steps for obtaining the spatial calibration matrix in step one include: using the imported CARLA digital twin environment map as a simulation map, and collecting point clouds from the simulation map; obtaining the point clouds from the real intersection corresponding to the target real intersection; importing the two sets of point clouds into the same point cloud registration tool, and selecting at least three pairs of non-collinear homonymous feature points, wherein the homonymous feature points include at least one of curb intersections, lane line intersections, stop line endpoints, traffic island corners, or road ancillary facility corners; and obtaining the rotation-translation matrix composed of rotation matrices and translation vectors based on the three-dimensional correspondence of the homonymous feature points. The matrix is ​​used to repeatedly select multiple sets of three or more pairs of non-collinear feature points with the same name, and calculate multiple candidate rotation-translation matrices. The registration residuals corresponding to each candidate matrix are statistically analyzed, and abnormal candidate matrices with residuals greater than a preset threshold are removed. The mean of the translation components in the retained candidate matrices is calculated, and the mean fusion and normalization of the rotation components are performed to obtain the final spatial calibration matrix. The matrix is ​​used to adjust the overall position and orientation of the digital twin map so that the map point cloud generated in CARLA coincides or nearly coincides with the point cloud collected from the real intersection under a unified coordinate framework, thus completing the spatial alignment.

3. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The establishment of traffic participant generation rules in step two specifically includes: mapping traffic participants observed by roadside sensors at real intersections to a digital twin scene coordinate system based on a spatial calibration matrix; statistically analyzing the frequency of occurrence, visible scale, and occlusion relationship of various traffic participants from a roadside perspective under the unified coordinate system, and establishing traffic participant generation rules based on the statistical results. These rules are used to constrain the selection of traffic participant categories, the configuration of visible scale, and the configuration of occlusion relationships. The enhancement of low-frequency categories and abnormal configurations includes: under the premise of maintaining the overall distribution constraints of roadside observations at the target intersection, performing directional compensation generation for traffic participant categories with occurrence frequencies below a preset threshold and traffic participant configurations that deviate from the normal roadside perspective morphological distribution.

4. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The establishment of traffic flow spatiotemporal distribution information in step three specifically includes: mapping the motion trajectories of traffic participants observed by roadside sensors at real intersections to the digital twin scene coordinate system based on a spatial calibration matrix; performing statistical analysis on the mapped motion trajectories to obtain the entry timing distribution, travel path distribution, speed change distribution, and relative spacing distribution of various traffic participants; establishing traffic flow control rules based on the distributions to constrain the entry order, travel path, speed change, car-following relationship, and yielding relationship of the heterogeneous traffic participant model group; and expanding the parameters of deployment density, entry timing, operating speed, and path combination under the constraints of these rules to generate various basic traffic flow states in free flow, stable flow, and locally congested flow.

5. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The identification of conflict-sensitive spatiotemporal regions in step four specifically includes: based on basic traffic flow, joint analysis of the trajectories, arrival times, relative distances, and relative speeds of different traffic participants within the target intersection to determine potential trajectory intersection areas and the degree of temporal overlap of traffic participants within these areas, and identifying areas with a temporal overlap exceeding a preset threshold as conflict-sensitive spatiotemporal regions; the restricted behavior reconstruction includes: constructing multiple candidate reconstruction schemes for target traffic participants within conflict-sensitive spatiotemporal regions, wherein the candidate reconstruction schemes include at least an entry time adjustment scheme, a traffic priority relationship adjustment scheme, a local path offset scheme, a speed change adjustment scheme, and a local occlusion enhancement scheme; calculating the impact of each candidate reconstruction scheme on the degree of temporal compression, the degree of change in minimum relative distance, and the degree of occlusion enhancement, and selecting the candidate reconstruction scheme that increases the degree of conflict exposure as the execution scheme, provided that road topology constraints and kinematic constraints are met.

6. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 5, characterized in that, The local occlusion enhancement scheme includes: adjusting the relative positional relationship, relative temporal relationship, or relative speed relationship between the target traffic participant and the occluded traffic participant, so that the visible area of ​​the target traffic participant from the roadside view is reduced, the visible duration is shortened, or the occlusion period is extended, thereby forming occlusion sudden occurrence, overtaking after occlusion, or delayed avoidance after occlusion scenarios; the triggering conditions for the occlusion sudden occurrence scenario include: the visible area of ​​the target traffic participant from the roadside view is lower than a preset threshold, the relative temporal interval between the target traffic participant and the occluded traffic participant is lower than a preset threshold, and the time when the target traffic participant enters the potential trajectory intersection area after the occlusion is removed meets the preset conflict window condition.

7. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 5, characterized in that, The degree of conflict exposure is jointly determined based on the temporal compression degree, the change in minimum relative distance, and the reduction in visible area from the roadside perspective before and after the implementation of the candidate reconstruction scheme. The temporal compression degree characterizes the reduction in the time interval between different traffic participants entering the potential trajectory intersection area; the change in minimum relative distance characterizes the change in the closest approach; and the reduction in visible area characterizes the reduction in the visible outline range or the reduction in the visible duration of the target traffic participant. Under the premise of satisfying road topology and kinematic constraints, the candidate reconstruction scheme that increases the temporal compression degree, decreases the minimum relative distance, and increases the visible area reduction degree is selected as the execution scheme. The selection of candidate reconstruction schemes also includes a feasibility verification of the operational status of the reconstructed traffic participants. The feasibility verification includes at least lane boundary constraint verification, speed continuity verification, acceleration change constraint verification, and steering continuity verification.

8. The method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The long-tail interaction scenarios described in step four include cutting in line, overtaking, delayed yielding, sudden occlusion, and cross-interference scenarios. Specifically, the generation of the cross-interference scenario involves: for multiple traffic participants with potential trajectory intersections, adjusting the entry time, passage order, or local path of at least one traffic participant based on their temporal and relative path relationships when entering the conflict-sensitive spatiotemporal region, so that multiple traffic participants form a temporal compression and spatial overlap enhancement interaction process within the conflict-sensitive spatiotemporal region; jointly evaluating the degree of temporal compression, minimum relative distance change, and occlusion enhancement of the reconstructed multiple traffic participants within the conflict-sensitive spatiotemporal region; when the joint evaluation result is higher than a preset threshold and satisfies road topology and kinematic constraints, the corresponding scenario is determined as a valid long-tail interaction scenario, and stored along with traffic participant trajectory information, occlusion relationship information, conflict-sensitive spatiotemporal region information, and scenario type identification information, as a scenario sample set for subsequent multimodal perception data acquisition and ground truth label generation.

9. A method for generating a vehicle-road cooperative dataset based on high-fidelity digital twins according to claim 1, characterized in that, The configuration of the multimodal virtual sensor array in step five specifically includes: in the digital twin scenario after spatial alignment, setting the virtual camera, virtual LiDAR, and virtual depth sensor according to the installation position, orientation parameters, and intrinsic parameters of the roadside sensors at the real intersection; the multimodal virtual sensor array is synchronously triggered under a unified simulation clock to synchronously collect basic traffic flow scenarios and effective long-tail interaction scenarios to obtain time-corresponding image data, point cloud data, and depth data; the image data, point cloud data, depth data, and target annotation information are directly generated under a unified coordinate framework, and the target annotation information includes at least the target position, target size, target category, and target trajectory information; the perception data and target annotation information in the vehicle-road cooperative multimodal dataset maintain a correspondence under the same coordinate framework, without the need for secondary coordinate correction of the collected perception data and target annotation information.

10. A vehicle-road cooperative dataset generation system based on high-fidelity digital twins, used to implement the method described in any one of claims 1-9, characterized in that, The system includes: The digital twin base construction module is used to build a digital twin environment map based on multi-source photogrammetric data from UAVs, and obtain a robust spatial calibration matrix through multiple point selection registrations and mean fusion to achieve high-precision spatial alignment between the digital twin scene and the real intersection scene. The traffic participant generation module is used to establish traffic participant generation rules based on the roadside observation features of real intersections, generate a heterogeneous traffic participant model group that conforms to the roadside observation features, and enhance low-frequency categories and abnormal configurations. The traffic flow and environment generation module is used to establish control rules based on the spatiotemporal distribution information of traffic flow at real intersections, and to extend parameters under the constraints of these rules to generate diverse basic traffic flows. At the same time, it extends the configuration of weather, lighting and time parameters to generate dynamic traffic backgrounds. The long-tail scene generation module is used to identify conflict-sensitive spatiotemporal areas during the evolution of basic traffic flow and to reconstruct effective long-tail interaction scenes in a targeted manner through restricted behavior. The multimodal acquisition and annotation output module is used to configure a multimodal virtual sensor array in a spatially aligned digital twin scene, perform multi-view and multimodal synchronous acquisition, and directly generate target annotation information under a unified coordinate framework to construct a vehicle-road cooperative multimodal dataset.

Images