Urban traffic perception data generation and analysis method based on virtual simulation

Abstract

The invention provides an urban traffic perception data generation and analysis method based on virtual simulation, and belongs to the field of automatic driving perception and data processing. According to the method, on the basis of a simulator, a client-server architecture is adopted, and automatic generation from a virtual scene to a standard data set is achieved. Comprising the following steps: a data extraction stage: acquiring multi-modal data such as an RGB image, a 3D bounding box and semantic segmentation in real time in a configurable scene; in the real reference generation stage, through the steps of coordinate mapping, initial frame generation, shielding filtering, fine fitting and the like, a 2D label with pixel-level precision is automatically output. According to the method, diversified scene data covering various weathers and road conditions can be efficiently generated at low cost, and the training effect and the adaptive capacity of the automatic driving perception model are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving perception technology, computer vision and data generation, specifically to a method for generating and analyzing urban traffic perception data based on virtual simulation, which is particularly suitable for constructing training datasets for autonomous driving target detection models. Background Technology

[0002] The safe and reliable operation of autonomous driving systems heavily relies on high-precision perception modules; conversely, improving the performance of these modules requires massive amounts of accurately labeled and scenario-rich traffic perception data for training. Currently, the acquisition of traffic perception data primarily depends on real-world road data collection and manual annotation, but this method has significant drawbacks:

[0003] 1. High cost and low efficiency of annotation: Manual annotation requires professionals to draw target bounding boxes frame by frame. Especially in complex scenes, experienced annotators take an average of 88 seconds to annotate a single bounding box. Moreover, as the number of targets in the image increases, the annotation efficiency will decrease significantly, making it difficult to support the construction of large-scale datasets. 2. Labeling errors are hard to avoid: Manual labeling is subject to subjective factors, making it difficult to guarantee the accuracy of bounding box positions. It is also easy to miss small targets at a distance, resulting in inconsistent dataset quality. 3. Insufficient scene coverage: Real-world road data collection is insufficient to cover rare and special scenarios (such as extreme weather, sudden road conditions) and harsh environments (heavy rain, heavy fog, strong light at night), while these scenarios are crucial to the robustness of autonomous driving systems; 4. High data acquisition threshold: The collection of large-scale real data requires a large investment of vehicles, sensors and manpower, which can only be undertaken by a few large enterprises or research institutions, thus limiting the popularization and development of autonomous driving technology.

[0004] In recent years, virtual simulation technology based on 3D game engines has become an important direction for solving the above problems. Autonomous driving simulators such as Carla, with their rich scene configuration capabilities and comprehensive API interfaces, have been widely used. However, existing simulators can only provide raw sensor data and 3D object information, and cannot directly output realistic 2D bounding box annotations for object detection. Manual post-processing annotation is still required, failing to fundamentally eliminate reliance on human intervention. Furthermore, existing virtual data generation tools are mostly non-open-source commercial products, or have shortcomings such as limited functionality, insufficient bounding box accuracy, and lack of consideration for occlusion handling, making it difficult to meet the training needs of high-performance perception models.

[0005] Therefore, there is an urgent need for a fully automated, high-precision method for generating traffic perception data that can generate diverse scenarios, thereby reducing the cost of dataset construction, improving data quality and scenario coverage, and supporting the rapid development of autonomous driving perception technology. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating and analyzing urban traffic perception data based on virtual simulation.

[0007] To achieve the above objectives, the present invention provides the following technical solution: This application provides a method for generating and analyzing urban traffic perception data based on virtual simulation. Based on an autonomous driving simulation platform, it acquires virtual scene information through simulation scene construction and multimodal data collection, and processes the virtual scene information to generate a standard labeled dataset for training an object detection model. The method includes the following steps: S1. Configure and run a virtual traffic scene in the simulation platform, control traffic participants in the scene, and synchronously collect multimodal perception data including image data, three-dimensional object information and scene semantic information. S2. Process the multimodal perception data collected in S1 to generate two-dimensional target annotation information that meets the requirements of the target detection framework. The processing includes at least: mapping the three-dimensional object information to the two-dimensional image plane and optimizing the mapping result based on the scene semantic information.

[0008] Optionally, in step S1, the configurable parameters of the virtual traffic scene include at least one of geographical environment, meteorological conditions, and lighting conditions.

[0009] Optionally, in step S1, the image data is a visible light RGB image from a preset viewpoint; the three-dimensional object information includes: 3D bounding boxes of traffic participants in three-dimensional space and their category labels; the scene semantic information is a semantic segmentation image obtained by pixel-level classification of the image data.

[0010] Optionally, in step S2, the process of mapping the three-dimensional object information to the two-dimensional image plane includes: based on the spatial transformation relationship provided by the simulation platform, transforming the corner coordinates of the three-dimensional 3D bounding box to the world coordinate system, then projecting them to the camera coordinate system, and finally obtaining its coordinates on the two-dimensional image through perspective projection transformation.

[0011] Optionally, in step S2, after completing the mapping, the step further includes generating an initial two-dimensional bounding box: for each object's set of projection points on the two-dimensional image, calculate its minimum bounding rectangle, and use the minimum bounding rectangle as the object's initial two-dimensional bounding box.

[0012] Optionally, in step S2, after generating the initial two-dimensional bounding box, an occlusion filtering step is further included: based on the semantic segmentation image, it is determined whether the object corresponding to the initial two-dimensional bounding box is visible in the current image, and the objects determined to be invisible are filtered.

[0013] Optionally, the occlusion filtering step specifically includes: calculating the center point coordinates of the initial two-dimensional bounding box and querying the pixel category of the semantic segmentation image at those coordinates; if the queried category matches the category label of the object, the object is determined to be visible; otherwise, it is determined to be occluded.

[0014] Optionally, in step S2, after the occlusion filtering step, a bounding box fitting step is further included: taking the semantic segmentation image as a reference, the initial two-dimensional bounding box of the visible object retained after occlusion filtering is subjected to contour fitting processing, so that the final output two-dimensional bounding box closely matches the actual pixel area of ​​the object in the image.

[0015] Optionally, the annotation file finally output in step S2 has a recording format adapted to one or more mainstream object detection frameworks, and the annotation file is stored in association with its corresponding image file to form a training dataset.

[0016] On the other hand, this application provides a system for implementing the above method, comprising: The simulation server module is used to build and run configurable virtual traffic scenarios; The data acquisition client module is communicatively connected to the simulation server module and is used to control traffic participants in the scene and acquire the multimodal perception data; The automated annotation processing module is used to receive and process the data collected by the data acquisition client module, execute the automated annotation steps, and output a standard format annotation dataset.

[0017] Compared with the prior art, this application has the following beneficial effects: This invention proposes a method and system for generating and analyzing urban traffic perception data based on virtual simulation. By integrating a virtual simulation framework, a multimodal data collaborative acquisition module, and an automated annotation engine, it effectively addresses the pain points of high cost, limited accuracy, and insufficient scene coverage in traditional traffic perception data generation. Its core innovation lies in constructing a dual system of dynamic scene generation and high-precision annotation iterative optimization.

[0018] At the data processing level, this system is designed with a multimodal data collaborative acquisition and preprocessing module: relying on the simulation server to configure virtual scene parameters, RGB images, 3D bounding boxes and semantic segmentation images are acquired synchronously; after completing data preprocessing through coordinate system calibration, color encoding verification and other steps, the data is input into the dynamic annotation generation engine.

[0019] The dynamic annotation generation engine serves as the technology hub, employing a four-step iterative process of "coordinate transformation - initial bounding box generation - occlusion filtering - fine fitting": first, the 3D bounding box is mapped to a 2D initial bounding box through perspective transformation; then, occluded objects are removed by combining semantic segmentation results; and finally, the bounding box is precisely fitted to the object contour by pixel-by-pixel shrinking, which greatly improves the annotation accuracy in complex traffic scenarios.

[0020] The dataset quality closed-loop control engine is the core of the system's decision-making. It constructs a two-way closed-loop mechanism of positive accuracy verification (calibrating parameters by comparing virtual annotations with real data) and negative scene adaptation (dynamically adjusting simulation parameters based on model training feedback), thereby achieving dual assurance of annotation accuracy and scene adaptability.

[0021] At the application level, the system can generate scarce data such as severe weather and rare special scenarios in batches. Its annotation files are compatible with mainstream object detection frameworks and can flexibly supplement data according to model training needs. At the same time, the system is developed based on the open-source Carla simulation server, which effectively reduces the threshold for data acquisition and provides an efficient and reliable data solution for the research and development of autonomous driving perception systems. Attached Figure Description

[0022] Figure 1 This schematically illustrates a data generation and analysis architecture diagram in an exemplary embodiment of the present disclosure.

[0023] Figure 2 The illustration shows the implementation steps of an embodiment of the invention in an exemplary embodiment of this disclosure.

[0024] Figure 3 This illustration shows an example of automated annotation of an urban traffic scene in an exemplary embodiment of this disclosure.

[0025] Figure 4 This schematic diagram illustrates the composition of an electronic device according to an exemplary embodiment of the present disclosure.

[0026] Figure 5 The schematic diagram illustrates the composition of a storage medium in an exemplary embodiment of the present disclosure.

[0027] Figure 6 The schematic diagram illustrates a data generation and analysis method flowchart in an exemplary embodiment of the present disclosure. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Furthermore, in this invention, an element referred to as fixed to or disposed on another element may be directly disposed on the other element, or there may be an intermediate element. When an element is considered to be connected to another element, it may be directly connected to the other element, or there may be an intermediate element present simultaneously. The terms vertical, horizontal, left, right, and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0030] See Figures 1-6 This application provides a method for generating and analyzing urban traffic perception data based on virtual simulation. Based on an autonomous driving simulation platform, it acquires virtual scene information through simulation scene construction and multimodal data acquisition, and processes the virtual scene information to generate a standard labeled dataset for training a target detection model. The method includes the following steps: S1. Configure and run a virtual traffic scene in the simulation platform, control traffic participants in the scene, and synchronously collect multimodal perception data including image data, three-dimensional object information and scene semantic information. S2. Process the multimodal perception data collected in S1 to generate two-dimensional target annotation information that meets the requirements of the target detection framework. The processing includes at least: mapping the three-dimensional object information to the two-dimensional image plane and optimizing the mapping result based on the scene semantic information.

[0031] In this embodiment, step S1 specifically involves: building a virtual traffic environment using the open-source Carla autonomous driving simulation platform, supporting custom parameter settings for town layouts (such as different road structures from Town01 to Town10), weather systems (configurable for severe and normal weather conditions such as rain, fog, snow, and sunny days), and lighting conditions (simulating day-night cycles, twilight backlighting, etc.). Ego vehicles (using mainstream car models such as Tesla Model 3) are created using the Carla Python API, and surrounding vehicles (including cars, trucks, motorcycles, and bicycles) and pedestrians are added according to preset density or random generation rules. Simultaneously, dynamic movement trajectories (such as going straight, turning, crossing the road, stopping and waiting, etc.) are assigned to traffic participants to simulate real traffic flow characteristics. The data acquisition module communicates with the server in real time through the Carla client, supporting both timed acquisition (e.g., triggered every 0.5 seconds) and event-triggered acquisition (e.g., automatic acquisition when special events such as traffic light switching, vehicle collisions, and pedestrian sudden stops are detected). The synchronously acquired multimodal data includes:

[0032] RGB image with a resolution of 540×960 (from the front-facing wide-angle camera of the ego vehicle); 3D bounding box data for each traffic participant (including object ID, category label, 3D coordinates, and pose information); Semantic segmentation images from the same viewpoint as the RGB images (using Carla's built-in semantic tag encoding, with single-channel pixel values ​​corresponding to object categories). All collected data are accompanied by a unified timestamp to ensure the spatiotemporal synchronization of multimodal information. The raw data are stored in local dataset directories in PNG format (image type) and JSON format (structured data type) respectively, providing a foundation for subsequent processing.

[0033] In this embodiment, step S2 specifically includes: Mapping 3D object information to 2D images: Using the camera intrinsic and extrinsic parameter transformation matrix provided by Carla, the eight corner points of the 3D bounding box of each traffic participant are transformed from the vehicle local coordinate system to the world coordinate system, and then projected onto the camera imaging plane; the 3D point (x, y, z) is mapped to 2D pixel coordinates (x', y') using the perspective projection formula, as follows:

[0034] Where f is the camera focal length. , This is the image center offset, determined by the camera parameters.

[0035] Based on the eight projected 2D points, the minimum bounding rectangle is calculated as the initial 2D bounding box, i.e. , , , Obtain the initial bounding box coordinates And the corresponding object category label.

[0036] Annotation optimization based on semantic segmentation images: First, occluded objects are filtered, and the center pixel coordinates are calculated for each initial bounding box.

[0037] If the category label of a pixel in the semantic segmentation image does not match the object category, the object is determined to be completely occluded and removed. Next, fine-fitting of the bounding box is performed, using the semantic segmentation image as a reference, iteratively shrinking the four sides of the initial bounding box: if a certain side (such as...) If there are no target category pixels on the corresponding left boundary, then move 1 pixel inward. +1), repeat this process until all edges touch the target pixel, to obtain an accurate 2D bounding box that closely follows the object's outline.

[0038] Annotation file generation and dataset construction: The optimized bounding boxes are converted into the standard format of the object detection framework, such as the YOLO format: category ID center x center y width w height h, where the center coordinates and width and height are normalized to the range [0, 1]. The annotation files are associated with the corresponding RGB images by filename to form a complete dataset containing image-annotation pairs. Users can generate diverse training data in batches by adjusting simulation scene parameters, such as severe weather like rainstorms and dense fog, and rare scenarios like reverse driving and pedestrians suddenly entering, to meet the training needs of different models.

[0039] In one specific implementation, in step S1, the configurable parameters of the virtual traffic scene include at least one of geographical environment, meteorological conditions, and lighting conditions.

[0040] In this embodiment, the configurable parameters can be obtained dynamically via the Carla Python API interface. For geographic environment parameters, it supports loading preset towns (such as Town05) by calling `client.load world(town name)`, or expanding the scene by importing custom map files in Open DRIVE format. Meteorological condition parameters can be obtained through: The weather conditions are adjusted in real time using methods such as `world.setweather(carla.WeatherParameters(rainintensity=0.8, fogdensity=0.5))`, where `rain intensity` (0-1) controls the rainfall intensity and `fog density` (0-1) controls the fog density; the sunlight condition parameters are obtained by using `world.getsun` to get the current sun attitude. Alternatively, you can call `world.set sun(azimuth, altitude)` to set the sun's azimuth (0-360 degrees) and altitude (-90 to 90 degrees). For example, setting `altitude=-10` can simulate a sunset backlighting scene. The system also supports batch importing parameter combinations from user-preset parameter configuration files, such as those in JSON format, to quickly generate diverse scenes and improve data acquisition efficiency.

[0041] In one specific implementation, in step S1, the image data is a visible light RGB image from a preset viewpoint; the three-dimensional object information includes: 3D bounding boxes of traffic participants in three-dimensional space and their category labels; the scene semantic information is a semantic segmentation image obtained by pixel-level classification of the image data.

[0042] In this embodiment, image data is acquired using the ego vehicle's front-facing 120° wide-angle camera, with a fixed resolution of 540×960 pixels and a camera focal length of 25mm. Automatic exposure and white balance adjustment are supported to ensure stable image quality under different lighting conditions. The acquired RGB images, 3D bounding boxes, and semantic segmentation images share the same timestamp, achieving frame-level synchronization of multimodal data. In the 3D object information, the 3D bounding box of each traffic participant is defined by the coordinates of the object's center in the world coordinate system. Based on the RGB image, the image includes three dimensions: length (L), width (w), and height (h), as well as a rotation angle θ around the y-axis. Category labels use standardized enumerated values: for example, "car" corresponds to 0, "truck" to 1, "pedestrian" to 2, and "bicycle" to 3. The object ID is a globally unique identifier used to associate the same target across different modalities. The scene semantic segmentation image maintains the same viewpoint and resolution as the RGB image, employing single-channel 8-bit pixel encoding. Each pixel value is mapped to a predefined semantic category, such as pixel value 7 corresponding to "road," 10 to "vehicle," and 11 to "pedestrian." This is generated in real-time through Carla's built-in semantic rendering pipeline, ensuring strict alignment with the pixel positions of the RGB image and providing accurate semantic reference for subsequent occlusion detection and bounding box optimization.

[0043] In one specific implementation, step S2, the process of mapping the three-dimensional object information to the two-dimensional image plane includes: based on the spatial transformation relationship provided by the simulation platform, transforming the corner coordinates of the three-dimensional 3D bounding box to the world coordinate system, then projecting them to the camera coordinate system, and finally obtaining its coordinates on the two-dimensional image through perspective projection transformation.

[0044] In this embodiment, the specific transformation process is as follows: The corner points of the 3D bounding box of each traffic participant are initially located in its own local coordinate system, with the geometric center of the object as the origin, the x-axis along the object's direction of movement, the y-axis to the left, and the z-axis upward. Coordinate transformation must first be completed using the object's world coordinate system pose information. The object's pose is represented by a quaternion returned by the Carla API, and this quaternion is converted into a rotation matrix. Translation vector of world coordinates relative to the center of the object The corner points of the local coordinate system Convert to world coordinate system points :

[0045]

[0046] Subsequently, the rotation matrix of the camera extrinsic parameters was used. Translation vector Transform the points from the world coordinate system to the camera coordinate system, where It is calculated from the camera's rotation angle, which includes at least the pitch angle, yaw angle, and roll angle; This refers to the camera's position in the world coordinate system, which can be obtained in real time using the `camera.get transform` interface. (Camera intrinsic parameters) This is read through the camera.get intrinsic interface. In this embodiment, the intrinsic parameters of the front-facing camera are set to f=1000 pixels. =480 pixels =270 pixels, matching the RGB image resolution of 540×960. A validity check is performed on the 2D pixel coordinates obtained from the projection: if the pixel coordinates of a corner point exceed the image width of 960 or height of 540, it is marked as invalid; for objects with invalid corner points, only valid corner points are retained for the initial bounding box calculation. If the number of valid corner points is less than 3, the object's annotation candidate is directly removed to reduce noise in subsequent processing.

[0047] In one specific implementation, step S2, after completing the mapping, further includes the step of generating an initial two-dimensional bounding box: for each object's set of projection points on the two-dimensional image, calculate its minimum bounding rectangle, and use the minimum bounding rectangle as the object's initial two-dimensional bounding box.

[0048] In this embodiment, for each traffic participant's valid two-dimensional projected corner point set after invalid corner points exceeding the image range have been removed, the x-coordinates of all corner points are first traversed, and the minimum value is taken as the left boundary of the initial bounding box. The maximum value is used as the right boundary. Similarly, iterate through the y-coordinates and take the minimum value as the upper boundary. The maximum value is used as the lower boundary. This yields the initial two-dimensional bounding box of the object: If an object has ≥3 valid projected corner points, an initial bounding box is generated normally; if the number is less than 3, the object is determined to be incompletely rendered in the current image or its projection is invalid, and the initial bounding box generation step for that object is skipped. The coordinate values ​​of the initial bounding box are all integers to ensure alignment with the image pixel grid, providing basic input for subsequent semantic segmentation-based image optimization steps. The category labels of the initial bounding box are consistent with the category labels in the original 3D object information and are synchronously recorded in a temporary annotation list for subsequent association and verification with the optimized bounding box.

[0049] In one specific implementation, step S2, after generating the initial two-dimensional bounding box, further includes an occlusion filtering step: based on the semantic segmentation image, determining whether the object corresponding to the initial two-dimensional bounding box is visible in the current image, and filtering out objects that are determined to be invisible.

[0050] In this embodiment, the occlusion filtering step is divided into two steps: The first step is to verify the central region by calculating the center coordinates (x, y, y) of the initial 2D bounding box. c y c The system extracts a 5×5 pixel region around the center coordinate in the semantic segmentation image (the region size can be adjusted by configuration parameters); it counts the percentage of pixels in this region that match the object category label. If the percentage is lower than a preset threshold, the object is determined to be severely occluded and marked as to be filtered.

[0051] The second step is a bounding box overall coverage check. It iterates through all pixels within the initial bounding box, counts the total number of pixels whose semantic labels match the object category, and calculates the ratio of this ratio to the total number of pixels in the bounding box. If the coverage is below 50%, the object is confirmed to be invisible in the current image and removed from the annotation list. For objects with coverage between 50% and 80%, the system records them as partially occluded and adds the attribute label "partial occlusion: True" to the annotation file for targeted handling during subsequent model training. These two hierarchical filtering steps effectively eliminate invalid annotations caused by occlusion, reduce the interference of annotation errors on model training, and further improve the accuracy and reliability of the dataset.

[0052] In one specific implementation, the occlusion filtering step specifically includes: calculating the center point coordinates of the initial two-dimensional bounding box and querying the pixel category of the semantic segmentation image at those coordinates; if the queried category matches the category label of the object, the object is determined to be visible; otherwise, it is determined to be occluded.

[0053] In this embodiment, the coordinates of the initial two-dimensional bounding box are first used. Calculate the center point The formula is:

[0054] The calculation result is rounded to integer pixel coordinates to match the pixel grid of the semantic segmentation image. Then, the pixel reading interface of the semantic segmentation image is called to obtain... The pixel value at a given location is mapped to a predefined semantic category. As mentioned earlier, pixel value 10 corresponds to vehicles, 11 to pedestrians, etc. If the mapped category perfectly matches the current object's category label, the object is considered visible, and its initial bounding box is retained for subsequent optimization. If they don't match, the object is considered completely occluded, and its labeling candidate is directly removed, without further fine-fitting of the bounding box. This occlusion filtering method is logically simple and computationally inefficient, making it suitable for batch data generation scenarios with high real-time requirements. Users can configure parameters to enable this strategy or the hierarchical filtering strategy described earlier to balance data quality and generation efficiency.

[0055] In one specific implementation, step S2, after the occlusion filtering step, further includes a bounding box fitting step: using the semantic segmentation image as a reference, the initial two-dimensional bounding box of the visible object retained after occlusion filtering is subjected to contour fitting processing, so that the final output two-dimensional bounding box closely matches the actual pixel area of ​​the object in the image.

[0056] In this embodiment, the semantic segmentation pixel encoding value corresponding to the current object category is first obtained, such as the pixel value of 10 corresponding to the vehicle category. Then, edge-by-edge iterative optimization is performed on the four edges of the initial two-dimensional bounding box after occlusion filtering. 1. Left boundary Adjustment: Fix the range of y as ,examine Does the target category pixel exist on the vertical line segment? If not, then... Move 1 pixel along the positive x-axis Repeat this operation until the target pixel appears on the line segment, or achieve Avoid having a width of 0.

[0057] 2. Right boundary Adjustment: Fix the range of y as ,examine Does the target category pixel exist on the vertical line segment? If not, then... Move 1 pixel along the negative x-axis Repeat until the target pixel is reached, or .

[0058] 3. Upper boundary Adjustment: Fix the range of x as ,examine Does the target category pixel exist on the horizontal line segment? If not, then... Move 1 pixel along the positive y-axis Repeat until the target pixel is reached, or .

[0059] 4. Lower boundary Adjustment: Fix the range of x as ,examine Does the target category pixel exist on the horizontal line segment? If not, then... Move 1 pixel along the negative y-axis Repeat until the target pixel is reached, or .

[0060] After completing a single iteration on all four sides, repeat the above four steps until the bounding box coordinates remain unchanged after two consecutive iterations, or the number of iterations reaches the preset limit. The final optimized bounding box must meet the following width requirement. pixels and height Minimum pixel size requirement; if the optimized size is smaller than this threshold, the initial bounding box after occlusion filtering is retained to avoid excessive shrinkage leading to annotation failure. The optimized bounding box coordinates are precisely aligned with the pixel grid, closely matching the actual contours of objects in the semantic segmentation image, providing more accurate location annotation basis for subsequent object detection model training.

[0061] In one specific implementation, the annotation file finally output in step S2 has a recording format adapted to one or more mainstream object detection frameworks, and the annotation file is stored in association with its corresponding image file to form a training dataset.

[0062] In this embodiment, the annotation file supports automatic generation and adaptation of formats from three mainstream object detection frameworks: 1. YOLO Format: Generates a .txt file for each single image, with the filename matching the corresponding RGB image (e.g., "img_20240501_001.jpg" corresponds to "img_20240501_001.txt"). Each line records the annotation of a single object, in the following format: [Category ID][Center x-normalized value][Center y-normalized value][Width normalized value][Height normalized value][Object global ID][Partial occlusion marker]. The normalized value is calculated based on the image resolution (540×960), the object global ID is a 32-bit string, and the partial occlusion marker is represented by 0 / 1, where 0 indicates no occlusion and 1 indicates partial occlusion.

[0063] 2. VOCXML format: Each image corresponds to a .xml file, containing... <annotation>Root node, child nodes include <filename>(image path) <size>(Width, Height, Depth) <object>List. Each <object>Nodes contain <name>(Category Name) <bndbox> (pixel coordinates)<object_id> (Global ID)<partial_occlusion> Extended fields such as (True / False) strictly adhere to the PASCALVOC specification.

[0064] 3. COCOJSON format: Generates a unified dataset description file (e.g., "train_dataset.json"), containing three core arrays: images, annotations, and categories. Images records image paths, resolutions, and IDs; each annotation record corresponds to an object, containing fields such as image_id (associated image), category_id, bounding box (x, y, w, h pixel coordinates), area (bounding box area), object_id, and partial_occlusion; categories defines the mapping between category names and IDs.

[0065] The data storage employs a hierarchical directory structure: the dataset root directory is divided into train, val, and test subdirectories. Each subdirectory contains rgb_images (RGB images), semantic_masks (semantic segmentation images), and labels (labeled files, categorized by format as yolo / voc / coco). Images and labels are associated via filename prefixes (e.g., "rgb_images / img_001.jpg" corresponds to "labels / yolo / img_001.txt"). The system supports local disk, HDFS, or cloud storage (such as AWS S3), and uses gzip compression for COCOJSON files to improve read efficiency.

[0066] The system automatically performs annotation validation: checking annotation completeness (number of images matches number of annotations), coordinate validity (normalized values ​​are within the range of 0-1), and category consistency (annotation categories are within a predefined list); it generates correction reports for invalid annotations to ensure the dataset meets model training requirements. The final generated dataset can be directly imported into frameworks such as YOLOv8, Faster R-CNN, and MMDetection for training without additional format conversion.

[0067] On the other hand, this application provides a system for implementing the above method, comprising: The simulation server module is used to build and run configurable virtual traffic scenarios; The data acquisition client module is communicatively connected to the simulation server module and is used to control traffic participants in the scene and acquire the multimodal perception data; The automated annotation processing module is used to receive and process the data collected by the data acquisition client module, execute the automated annotation steps, and output a standard format annotation dataset.

[0068] In this embodiment, the simulation server module builds a virtual traffic scene engine based on the Carla simulation platform. It supports importing real urban road network data in Open DRIVE format, such as the road network of the core area of ​​Haidian District, Beijing, and automatically generates a high-precision 3D road model containing elements such as lane lines, traffic lights, and road signs. It also provides a visual configuration interface, allowing users to customize traffic flow parameters (e.g., 2000 vehicles per hour during morning rush hour), the proportion of traffic participant types (e.g., 70% cars, 20% buses, 10% pedestrians), environmental conditions (e.g., 50-meter visibility in rainy weather, 100 lux light intensity at night), and sensor layout (e.g., installing 5 panoramic cameras and 2 LiDARs at intersections). During runtime, the simulation server module outputs the 3D pose (x, y, z coordinates and quaternion pose), category labels, speed, and other metadata of all traffic participants in the scene in real time at a 30Hz frame rate, as well as raw data collected by each sensor, such as RGB images, semantic segmentation masks, and LiDAR point clouds. This data is then pushed to the data acquisition client module via a message queue (e.g., RabbitMQ).

[0069] The data acquisition client module is developed in Python and establishes a long-lived connection with the simulation server module via TCP / IP protocol. It can send control commands to adjust scene parameters, such as triggering sudden traffic events like rear-end collisions or switching sensor states to change camera perspectives. The module has a built-in data synchronization engine that performs frame alignment of multimodal data based on timestamps with an error of less than 1ms. Each frame of data is encapsulated as a JSON-formatted metadata header plus a binary data body (e.g., JPEG encoding for RGB images and PNG encoding for semantic segmentation images), and then uploaded in batches to the distributed storage queue of the automated annotation processing module. Furthermore, the module supports breakpoint resume functionality; when the network is interrupted, it automatically saves unuploaded data and resumes transmission upon connection restoration, ensuring data integrity.

[0070] The automated annotation processing module is deployed on a high-performance computing server or workstation, employing a modular software architecture. This includes a data receiving module responsible for receiving data uploaded from the client and storing it on a local high-capacity storage or LAN storage server; preprocessing services such as image resolution normalization and 3D object coordinate transformation; annotation execution services such as parallel corner filtering, initial bounding box generation, occlusion filtering, and bounding box fitting, utilizing GPUs to accelerate pixel-level computation of semantic segmentation images; format conversion services such as converting annotation results to standard formats like YOLO / VOC / COCO; and quality verification services such as generating a dataset quality report, including metrics like annotation accuracy, occlusion ratio distribution, and class balance. The processed annotation dataset is stored in a structured directory on the local file system, ready for direct use by subsequent model training programs.

[0071] The modules interact through standardized interfaces. The 3D data output by the simulation server module and the sensor data are serialized in Protobuf format to ensure cross-platform compatibility. Communication between the data acquisition client and the automated annotation module uses HTTPS protocol encryption to ensure data transmission security. The system also provides a visual monitoring panel to display the module's operating status, data acquisition progress, and annotation task completion rate in real time, facilitating user operation and maintenance management.

[0072] Example 2 This embodiment, in conjunction with the accompanying drawings, provides a more detailed description of the technical solution of this application: This invention provides a method for generating and analyzing urban traffic perception data based on virtual simulation. This method is implemented using the Carla autonomous driving simulation simulator, and the core process is divided into two main stages: data extraction and real reference generation. Figure 1 The overall architecture of this solution is shown, which consists of three parts: the Carla server on the left, the data extraction module in the middle, and the real value generation module on the right.

[0073] For the Carla server, this solution uses the unmodified native version from the Carla repository. Therefore, if Carla is already installed on the system, it can be used directly without any modifications. The data extraction module is implemented as a Carla client (e.g., ...). Figure 1 As shown in the middle, the simulation scene can be controlled based on weather, lighting conditions, and the surrounding vehicles and pedestrians. Due to the use of the Carla client architecture, the data extraction phase can run in real-time, following the Carla server's timeline; while the ground truth generation phase is executed as an independent offline process, calculating the 2D bounding boxes of vehicles and pedestrians based on the extracted data. The specific steps are as follows:

[0074] Step 1: Collect multimodal data using the Carla simulator client On the Carla server, a virtual driving scenario (such as different towns, weather conditions, and lighting conditions) is built, and ego vehicles, surrounding vehicles, and pedestrians are deployed. The following data is collected in real-time via a Python API client:

[0075] (1) Visible light image: RGB image from the front-facing camera of the ego vehicle, denoted as... ^H×W×3 (RGB three channels), resolution set to H=540, W=960, viewing angle 90 degrees; (2) 3D bounding box data: The position and pose of each object (vehicle or pedestrian) in the scene in the 3D vehicle coordinate system, represented by a set of eight corner points. and its class label T∈{vehicle, pedestrian}; (3) Semantic segmentation image: a pixel-level classification image corresponding to the RGB image, where each pixel S[x][y] represents the object type through color coding (e.g., blue for vehicles and red for pedestrians).

[0076] Data acquisition supports both periodic automatic triggering and user-manual triggering modes to cover both common and rare special scenarios. Raw data is saved as PNG images and text files for subsequent processing.

[0077] Step 2: Convert the 3D bounding box coordinates to a 2D image coordinate system Using the transformation matrix provided by Carla, the corner points of the 3D bounding box of each object are transformed from the vehicle coordinate system to the world coordinate system, and then projected onto the camera coordinate system. Combining the camera focal length, field of view, and image resolution, perspective transformation is used to transform the 3D points... Mapped to 2D image pixel coordinates This process accurately models the camera pose and scene geometry, providing a foundation for 2D annotation.

[0078] Step 3: Generate initial 2D bounding box proposal Eight projected 2D points for each object The minimum bounding rectangle is calculated as the initial bounding box, and the algorithm is shown in the formula:

[0079] Step 4: Filter occluded objects based on semantic image segmentation To address the occlusion problem, the visibility of objects is verified using a semantically segmented image S. For each initial bounding box, its center pixel coordinates are calculated.

[0080]

[0081] If S[x c ][y c If an object matches the object category T, it is considered visible; otherwise, it is considered occluded and culled. This method ensures that only visible objects are labeled, avoiding interference from false targets.

[0082] Step 5: Fine-fit the bounding box to the object boundary. To eliminate the gap between the initial bounding box and the actual object contour, the four sides of the bounding box are iteratively shrunk using the semantic segmentation image as a reference: (1) If there is no target pixel on a certain side, move it one pixel inward (e.g., x). min +1); (2) Loop until all edges touch the target pixel, and output a precise bounding box that closely follows the object boundary. .

[0083] This process achieves pixel-level precision annotation.

[0084] Step 6: Output annotation files and generate datasets The final bounding boxes are converted to a standard format for object detection frameworks (such as YOLO's text format), recording the object category, center coordinates (x, y), width w, and height h. The labeled files are then associated with the corresponding images to form a complete dataset. Users can generate diverse datasets in batches by adjusting simulation parameters (such as weather and the number of objects).

[0085] This method effectively addresses the pain points of high cost and large errors associated with traditional manual annotation through a fully automated process. It is particularly suitable for generating training data in harsh environments and can significantly improve the robustness of object detection models in real-world scenarios. Specific implementation details can be found above and will not be repeated here.

[0086] Example 3 In an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.

[0087] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely in hardware, entirely in software (including firmware, microcode, etc.), or in a combination of hardware and software, collectively referred to herein as "circuit," "module," or "system."

[0088] The following reference Figure 4 To describe an electronic device 400 according to this embodiment of the present invention. Figure 4 The electronic device 400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0089] like Figure 4 As shown, the electronic device 400 is manifested in the form of a general-purpose computing device. The components of the electronic device 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including storage unit 420 and processing unit 410).

[0090] The storage unit stores program code that can be executed by the processing unit 410, causing the processing unit 410 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 410 can perform the following method: S1. Configure and run a virtual traffic scene in the simulation platform, control traffic participants in the scene, and synchronously collect multimodal perception data including image data, three-dimensional object information, and scene semantic information; S2. Process the multimodal perception data collected in S1 to generate two-dimensional target annotation information that meets the requirements of the target detection framework. The processing includes at least mapping the three-dimensional object information to a two-dimensional image plane and optimizing the mapping result based on the scene semantic information.

[0091] Storage unit 420 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 4201 and / or cache memory 4202, and may further include a read-only memory (ROM) 4203.

[0092] Storage unit 420 may also include a program / utility 4204 having a set (at least one) program module 4205, such program module 4205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0093] Bus 430 can represent one or more of several types of bus structures, including memory cell bus or memory cell controller, peripheral bus, graphics acceleration port, processing unit, or local bus using any of the multiple bus structures.

[0094] Electronic device 400 can also communicate with one or more external devices 300 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 400, and / or any device that enables electronic device 400 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 450. Furthermore, electronic device 400 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 460. As shown, network adapter 460 communicates with other modules of electronic device 400 via bus 430. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0095] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0096] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.

[0097] refer to Figure 5 As shown, a program product 500 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0098] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0099] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0101] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Python and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0102] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0103] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.< / bndbox> < / name> < / object> < / object> < / size> < / filename> < / annotation>

Claims

1. A method for generating and analyzing urban traffic perception data based on virtual simulation, characterized in that, Based on the autonomous driving simulation platform, virtual scene information is obtained through simulation scene construction and multimodal data acquisition, and the virtual scene information is processed to generate a standard labeled dataset for training the target detection model. The method includes the following steps: S1. Configure and run a virtual traffic scene in the simulation platform, control traffic participants in the scene, and synchronously collect multimodal perception data including image data, three-dimensional object information and scene semantic information. S2. Process the multimodal perception data collected in S1 to generate two-dimensional target annotation information that meets the requirements of the target detection framework. The processing includes at least: mapping the three-dimensional object information to the two-dimensional image plane and optimizing the mapping result based on the scene semantic information.

2. The method according to claim 1, characterized in that, In step S1, the configurable parameters of the virtual traffic scene include at least one of geographical environment, meteorological conditions, and lighting conditions.

3. The method according to claim 1, characterized in that, In step S1, the image data is a visible light RGB image with a preset viewing angle; The three-dimensional object information includes: the 3D bounding box of the traffic participant in three-dimensional space and its category label; The scene semantic information is a semantic segmentation image obtained by pixel-level classification of the image data.

4. The method according to claim 1 or 3, characterized in that, In step S2, the process of mapping the three-dimensional object information to the two-dimensional image plane includes: based on the spatial transformation relationship provided by the simulation platform, transforming the corner coordinates of the three-dimensional 3D bounding box to the world coordinate system, then projecting them to the camera coordinate system, and finally obtaining its coordinates on the two-dimensional image through perspective projection transformation.

5. The method according to claim 1 or 3, characterized in that, In step S2, after completing the mapping, the step further includes generating an initial two-dimensional bounding box: for each object's set of projection points on the two-dimensional image, calculate its minimum bounding rectangle, and use the minimum bounding rectangle as the object's initial two-dimensional bounding box.

6. The method according to claim 5, characterized in that, In step S2, after generating the initial two-dimensional bounding box, an occlusion filtering step is further included: based on the semantic segmentation image, it is determined whether the object corresponding to the initial two-dimensional bounding box is visible in the current image, and the object determined to be invisible is filtered.

7. The method according to claim 6, characterized in that, The occlusion filtering step specifically includes: calculating the center point coordinates of the initial two-dimensional bounding box and querying the pixel category of the semantic segmentation image at those coordinates; if the queried category matches the category label of the object, the object is determined to be visible, otherwise it is determined to be occluded.

8. The method according to claim 6, characterized in that, In step S2, after the occlusion filtering step, a bounding box fitting step is further included: taking the semantic segmentation image as a reference, the initial two-dimensional bounding box of the visible object retained after occlusion filtering is subjected to contour fitting processing, so that the final output two-dimensional bounding box closely matches the actual pixel area of ​​the object in the image.

9. The method according to claim 1, characterized in that, The annotation file output in step S2 has a recording format adapted to one or more mainstream object detection frameworks. The annotation file is stored in association with its corresponding image file to form a training dataset.

10. A system for implementing the method according to any one of claims 1-9, characterized in that, include: The simulation server module is used to build and run configurable virtual traffic scenarios; The data acquisition client module is communicatively connected to the simulation server module and is used to control traffic participants in the scene and acquire the multimodal perception data. The automated annotation processing module is used to receive and process the data collected by the data acquisition client module, execute the automated annotation steps, and output a standard format annotation dataset.

Images