An automated simulation data labeling system and method

Abstract

The application provides an automatic simulation data labeling system and method, which comprises: a calibration tool for loading a parking scene FBX model, reading model vertex information, and calculating the point cloud relationship of the model shell vertex relative to the model centroid; an Unreal engine plug-in for obtaining vertex point cloud information of the FBX model, outputting a rendering graph and an RT graph with depth information based on a bin file function containing the vertex point cloud information and a film rendering queue provided by the Unreal engine; an automatic labeling interface module for realizing a graphical interface for automatic labeling and setting labeling parameters based on the graphical interface; a picture labeling module for labeling pictures based on the rendering graph, the RT graph with depth information, and the point cloud information through predefined labeling rules; and a Log module for recording error codes returned in the labeling process and saving pictures with labeling errors. Through the scheme, the automatic labeling of simulation pictures can be improved, manpower and material resources can be saved, and corner case scene coverage can be realized.

Description

Technical Field

[0001] This invention belongs to the field of data annotation, and in particular relates to an automated simulation data annotation system and method. Background Technology

[0002] With the rapid development of autonomous driving technology, virtual simulation is playing an increasingly important role in autonomous driving research and development and testing, and simulation tools and models can play an important role in mass production applications of automobiles.

[0003] When training the visual perception AI model for Auto Parking Assess (APA) in a typical car, a large number of labeled images are often required. These labeled images not only need to be collected on-site, but also need to be manually labeled. This process is not only time-consuming and labor-intensive, but also difficult to cover some special corner case scenarios. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an automated simulation data annotation system and method to solve the problems that existing manual data annotation is time-consuming and labor-intensive, and difficult to cover corner case scenarios.

[0005] In a first aspect of the present invention, an automated simulation data annotation system is provided, comprising:

[0006] The calibration tool is used to load a pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid.

[0007] This Unreal Engine plugin is used to obtain vertex point cloud information of FBX models. Based on the bin file function containing vertex point cloud information and the Unreal Engine's built-in movie rendering queue, it outputs a rendered image and an RT image with depth information.

[0008] The automatic annotation interface module is used to implement a graphical interface for automatic annotation and to set annotation parameters based on the graphical interface;

[0009] The image annotation module is used to generate annotation boxes for images based on pre-annotated rendered images, RT images with depth information, and point cloud information, according to predefined annotation rules.

[0010] The Log module is used to record error codes returned during the annotation process and save images of annotation errors.

[0011] In a second aspect of the present invention, an automated simulation data annotation method is provided, comprising:

[0012] Load the pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid.

[0013] Obtain vertex point cloud information of the FBX model, and output a rendered image and a RT image with depth information based on the bin file function containing vertex point cloud information and the movie rendering queue built into the Unreal Engine.

[0014] Set annotation parameters based on the automatic annotation graphical interface;

[0015] Based on pre-annotated rendered images, RT images with depth information, and point cloud information, annotation boxes are generated for image annotation using predefined annotation rules.

[0016] Record the error codes returned during the annotation process and save the images of the annotation errors.

[0017] In a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect of the present invention.

[0018] In a fourth aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method provided in the first aspect of the present invention.

[0019] In this embodiment of the invention, a virtual parking scene model is constructed, and the model is rendered and output as a rendered image and a RT image based on the Unreal plugin. The image is automatically annotated according to predefined annotation rules, thereby realizing the automatic collection and annotation of parking scene images. This not only saves manpower and resources and improves the efficiency of collection and annotation, but also covers corner case scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the structure of an automated simulation data annotation system provided in one embodiment of the present invention;

[0022] Figure 2A flowchart illustrating an automated simulation data annotation method according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be understood that the terms "comprising" and other similar expressions in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, or apparatus that includes a series of steps or units and is not limited to the listed steps or units. Furthermore, "first" and "second" are used to distinguish different objects and are not intended to describe a specific order.

[0026] Please see Figure 1 The present invention provides a schematic diagram of the structure of an automated simulation data annotation system, comprising:

[0027] The calibration tool 110 is used to load the pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid.

[0028] Construct an FBX model of the parking scene, load the FBX model to be annotated using the FBX SDK, and read all vertex information of the loaded FBX model. Through basic mathematical calculations, determine which vertices are shell vertices, calculate the point cloud relationship between the shell vertices and the model's centroid, and save the result.

[0029] In this embodiment, the parking scene image simulation can be achieved by using the parking scene FBX model, which can not only meet the requirements of the amount of data to be collected, but also cover a variety of corner case scenarios.

[0030] Unreal Engine Plugin 120 is used to obtain vertex point cloud information of FBX models. Based on the bin file function containing vertex point cloud information and the movie rendering queue that comes with Unreal Engine, it outputs a rendered image and an RT image with depth information.

[0031] The bin file function saves the vertex point cloud information of the FBX model corresponding to the current frame to the bin file based on at least the current frame number, screen viewport size, and output image size, and realizes frame synchronization in the movie rendering queue.

[0032] The Unreal Engine plugin defines a function to save a bin file. This function saves the vertex point cloud information of the model to a bin file based on parameters such as the current frame rate, screen viewport size, and output image size. Since the Unreal Engine's built-in movie rendering queue introduces a certain delay after each frame's code call, this function can also perform frame synchronization.

[0033] Unreal Engine's built-in movie rendering queue creates rendering curves. At each frame of the movie rendering queue curve, a function saved in the bin file by the plugin is called. This function stores the point cloud information of the current frame.

[0034] The scene is mapped using post-processing functions, causing the scene to call the movie rendering queue to output a RT map (i.e., a depth field rule map) and a rendered map containing depth information.

[0035] The automatic annotation interface module 130 is used to implement the graphical interface for automatic annotation and to set annotation parameters based on the graphical interface;

[0036] The graphical interface allows for easy parameter settings for the automatic annotation tool and the loading of other automatic annotation modules, facilitating the invocation and development of new features and achieving decoupling between the front-end and back-end. The automatic annotation system employs a multi-threaded design, with each annotation module using a separate thread, preventing the entire software from failing due to an error in one module. It also supports multi-threaded loading, Chinese / English language switching, and configuration file reading and saving.

[0037] Image annotation module 140 is used to generate annotation boxes for image annotation based on pre-annotated rendered images, RT images with depth information, and point cloud information, according to predefined annotation rules.

[0038] By using rendered color images, depth field (RT) images, and bin files containing point cloud information, and through calculations based on predefined rules, bounding boxes for different targets (such as obstacles, parking lines, etc.) can be obtained on the images.

[0039] In some embodiments, the image annotation module 140 further includes:

[0040] The image conversion module is used to convert the labeled RT image into a fisheye image and an inverse perspective transformation image;

[0041] The process involves calculating the corresponding transformation matrix based on camera intrinsic parameters and distortion parameters, using OpenCV and the transformation matrix to move points on the image to their corresponding positions, and outputting the transformed fisheye image. The fisheye image is then cropped, and based on the IPM distortion parameters, it is converted into the corresponding IPM image (i.e., a bird's-eye view after perspective transformation).

[0042] The Log module 150 is used to record error codes returned during the annotation process and save images of annotation errors.

[0043] The function return value of the image annotation module is recorded, along with its input parameters and output values. The error codes are then formatted and stored in a log file. Using the return value and a predefined format, the return value can be formatted into the required log text. Each line of this text contains the name of the function that reported the error, the line number where the error occurred, the function's return value, the timestamp of the error, and the input parameters. The log text is then saved as a timestamped log file using the save function.

[0044] In this embodiment, the standard images output by the Unreal engine can be automatically converted into the required fisheye or IPM images, and the target bounding boxes can be automatically marked on the converted images. This can save manpower and resources, improve image annotation efficiency, avoid frequent on-site image collection, and achieve coverage of corner case scenarios.

[0045] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0046] Figure 2 This is a flowchart illustrating an automated simulation data annotation method provided in an embodiment of the present invention. The method includes:

[0047] S201. Load the pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid.

[0048] S202. Obtain the vertex point cloud information of the FBX model, and output the rendered image and the RT image with depth information based on the bin file function containing the vertex point cloud information and the movie rendering queue built into the Unreal Engine.

[0049] The bin file function saves the vertex point cloud information of the FBX model corresponding to the current frame to the bin file based on at least the current frame number, screen viewport size, and output image size, and realizes frame synchronization in the movie rendering queue.

[0050] S203. Set annotation parameters based on the automatic annotation graphical interface;

[0051] S204. Based on the pre-annotated rendered image, the RT image with depth information, and the point cloud information, the image is annotated to generate annotation boxes according to predefined annotation rules.

[0052] Optionally, the annotated rendering can be converted into a fisheye image and an inverse perspective transformation image;

[0053] The process involves calculating the corresponding transformation matrix based on camera intrinsic parameters and distortion parameters, using OpenCV and the transformation matrix to move points on the image to their corresponding positions, and outputting the transformed fisheye image. The fisheye image is then cropped and converted into a corresponding IPM image based on the IPM distortion parameters.

[0054] S205. Record the error codes returned during the annotation process and save the images of the annotation errors.

[0055] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the systems and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0056] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device is used for automatic annotation of simulation data. Figure 3 As shown, the electronic device 3 of this embodiment includes: a memory 310, a processor 320, and a system bus 330. The memory 310 includes an executable program 3101 stored thereon. As those skilled in the art will understand, Figure 3 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] The following is combined with Figure 3 A detailed introduction to each component of the electronic device:

[0058] The memory 310 can be used to store software programs and modules. The processor 320 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 310. The memory 310 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cached data), etc. In addition, the memory 310 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0059] The memory 3100 contains an executable program 3101 for network request methods. This executable program 3101 can be divided into one or more modules / units, which are stored in the memory 310 and executed by the processor 320 for tasks such as simulation data annotation. Each module / unit can be a series of computer program instruction segments capable of performing a specific function, describing the execution process of the computer program 3101 in the electronic device 3. For example, the computer program 3101 can be divided into functional modules such as a calibration tool, an Unreal engine plugin, an automatic annotation interface module, an image annotation module, and a Log module.

[0060] The processor 320 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 310, and by calling data stored in the memory 310, it performs various functions and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, application programs, etc., and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 320.

[0061] The system bus 330 is used to connect various functional components within the computer, transmitting data, address, and control information. Its type can be, for example, a PC I bus, an ISA bus, or a CAN bus. Instructions from the processor 320 are transmitted to the memory 310 via the bus, and the memory 310 sends data back to the processor 320. The system bus 330 is responsible for data and instruction exchange between the processor 320 and the memory 310. Of course, the system bus 330 can also connect to other devices, such as network interfaces and display devices.

[0062] In this embodiment of the invention, the executable program executed by the processing 320 of the electronic device includes:

[0063] Load the pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid.

[0064] Obtain vertex point cloud information of the FBX model, and output a rendered image and a RT image with depth information based on the bin file function containing vertex point cloud information and the movie rendering queue built into the Unreal Engine.

[0065] Set annotation parameters based on the automatic annotation graphical interface;

[0066] Based on pre-annotated rendered images, RT images with depth information, and point cloud information, annotation boxes are generated for image annotation using predefined annotation rules.

[0067] Record the error codes returned during the annotation process and save the images of the annotation errors.

[0068] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0069] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0070] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An automated simulation data annotation system, characterized in that, include: The calibration tool is used to load a pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid. The process of loading the pre-annotated parking scene FBX model includes: constructing the FBX model of the virtual parking scene; This Unreal Engine plugin is used to obtain vertex point cloud information of FBX models. Based on the bin file function containing vertex point cloud information and the Unreal Engine's built-in movie rendering queue, it outputs a rendered image and an RT image with depth information. The automatic annotation interface module is used to implement a graphical interface for automatic annotation and to set annotation parameters based on the graphical interface; The image annotation module is used to generate annotation boxes for images based on pre-annotated rendered images, RT images with depth information, and point cloud information, according to predefined annotation rules. The Log module is used to record error codes returned during the annotation process and save images of annotation errors.

2. The system according to claim 1, characterized in that, The bin file function saves the vertex point cloud information of the FBX model corresponding to the current frame to the bin file based on at least the current frame number, screen viewport size, and output image size, and realizes frame synchronization in the movie rendering queue.

3. The system according to claim 1, characterized in that, The image annotation module also includes: The image conversion module is used to convert the labeled RT image into a fisheye image and an inverse perspective transformation image; The process involves calculating the corresponding transformation matrix based on camera intrinsic parameters and distortion parameters, using OpenCV and the transformation matrix to move points on the image to their corresponding positions, and outputting the transformed fisheye image. The fisheye image is then cropped and converted into a corresponding IPM image based on the IPM distortion parameters.

4. An automated simulation data annotation method, characterized in that, include: Load the pre-annotated parking scene FBX model, read the model vertex information, and calculate the point cloud relationship between the model shell vertices and the model centroid. The process of loading the pre-annotated parking scene FBX model includes: constructing the FBX model of the virtual parking scene; Obtain vertex point cloud information of the FBX model, and output a rendered image and a RT image with depth information based on the bin file function containing vertex point cloud information and the movie rendering queue built into the Unreal Engine. Set annotation parameters based on the automatic annotation graphical interface; Based on pre-annotated rendered images, RT images with depth information, and point cloud information, annotation boxes are generated for image annotation using predefined annotation rules. Record the error codes returned during the annotation process and save the images of the annotation errors.

5. The method according to claim 4, characterized in that, The bin file function saves the vertex point cloud information of the FBX model corresponding to the current frame to the bin file based on at least the current frame number, screen viewport size, and output image size, and realizes frame synchronization in the movie rendering queue.

6. The method according to claim 4, characterized in that, The function based on the bin file containing vertex point cloud information and the Unreal Engine's built-in movie rendering queue outputs a rendered image and a RT image with depth information, which also include: Convert the labeled RT image into a fisheye image and an inverse perspective transformation image; The process involves calculating the corresponding transformation matrix based on camera intrinsic parameters and distortion parameters, using OpenCV and the transformation matrix to move points on the image to their corresponding positions, and outputting the transformed fisheye image. The fisheye image is then cropped and converted into a corresponding IPM image based on the IPM distortion parameters.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of an automated simulation data annotation method as described in any one of claims 4 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed, it implements the steps of an automated simulation data annotation method as described in any one of claims 4 to 6.

Images