Annotation method, device, medium and program product applied to autonomous driving simulation

Abstract

An embodiment of the present application provides a labeling method for autonomous driving simulation, including: obtaining radar point clouds of the same type; obtaining instance numbers of each point in the radar point cloud of the same type, and generating a visible object number list based on the instance numbers of each point; generating a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes multiple indexes and multiple contents, the multiple contents and the multiple indexes correspond one-to-one, at least one index includes the instance number in the visible object number list, and the multiple contents include points in the radar point cloud of the same type; obtaining camera information, the camera information includes the intrinsic parameters, extrinsic parameters and position information of the virtual camera that outputs the image to be labeled in three-dimensional space; and labeling the two-dimensional bounding box of the detected object on the camera plane of the virtual camera based on the matching list and the camera information.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, in particular to the field of autonomous driving simulation technology, and more specifically to a labeling method, device, electronic device, storage medium and program product applied to autonomous driving simulation. Background Art

[0002] The development and verification of autonomous driving algorithms require large amounts of data for training or testing, such as visual images, LiDAR point clouds, and other sensor data. This data must be combined with ground truth values, such as the 2D and 3D bounding boxes of detected objects. Using a real vehicle for data collection and manually annotating the ground truth values ​​is extremely costly. Consequently, autonomous driving simulation software is gaining increasing attention. It can quickly generate virtual sensor data in batches and, through development, can automatically annotate the ground truth values.

[0003] In existing autonomous driving simulation software, users can easily obtain the real-time position of objects in three-dimensional space. However, obtaining the two-dimensional bounding box of an object in an image captured by a virtual camera in the simulation requires secondary development. Current methods for automatic two-dimensional bounding box annotation simply project the object's three-dimensional bounding box vertices onto the two-dimensional camera plane to obtain the two-dimensional bounding box. As a result, some of the annotated two-dimensional bounding boxes are incorrect, and for objects such as vehicles, the annotated two-dimensional bounding boxes are not accurate or compact enough.

[0004] The above information disclosed in this section is only for understanding the background of the technical concept of the present application and therefore the above information may contain information that does not constitute the prior art. Summary of the Invention

[0005] In view of the above problems, the present application provides a labeling method, device, electronic device, storage medium and program product applied to autonomous driving simulation.

[0006] According to a first aspect of the present application, a labeling method for autonomous driving simulation is provided, the method comprising:

[0007] Obtaining a radar point cloud of the same type, wherein each point in the radar point cloud of the same type includes the following information of the detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point, wherein each point in the radar point cloud of the same type has the same type number;

[0008] Obtaining the instance number of each point in the radar point cloud of the same type, and generating a visible object number list according to the instance number of each point;

[0009] Generate a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes a plurality of indexes and a plurality of contents, the plurality of contents and the plurality of indexes respectively correspond one-to-one, at least one index of the plurality of indexes includes an instance number in the visible object number list, and the plurality of contents include points in the radar point cloud of the same type;

[0010] Obtaining camera information, the camera information including intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated; and

[0011] Marking a two-dimensional bounding box of the detected object on the camera plane of the virtual camera based on the matching list and the camera information,

[0012] The step of marking a two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera according to the matching list and the camera information specifically includes:

[0013] Projecting multiple contents with the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points on the camera plane; and

[0014] A two-dimensional bounding box surrounding the multiple projection points is determined, where the two-dimensional bounding box surrounding the multiple projection points is a two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

[0015] According to some exemplary embodiments, obtaining radar point clouds of the same type includes:

[0016] Acquire an initial radar point cloud, wherein each point in the initial radar point cloud contains the following information of a detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point; and

[0017] Points with the same category number are selected from the initial radar point cloud to form radar point clouds of the same category.

[0018] According to some exemplary embodiments, the method further includes: acquiring a background static object list, wherein the background static object list includes three-dimensional spatial position information and three-dimensional bounding box vertex position information of each static object in the simulation environment.

[0019] According to some exemplary embodiments, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category specifically includes: selecting points with category numbers corresponding to vehicles from the initial radar point cloud to form a first radar point cloud of the same category.

[0020] According to some exemplary embodiments, generating a matching list based on the radar point clouds of the same type and the visible object number list specifically includes:

[0021] Matching the visible object number list and the background static object list to form a first matching result,

[0022] Wherein, the first matching result includes a first sublist and a second sublist;

[0023] The first sublist includes a plurality of indexes and a plurality of contents, wherein the plurality of indexes in the first sublist include instance numbers of moving vehicles recorded in the visible object number list, the instance numbers of the moving vehicles are not equal to a first specified value, wherein the first specified value represents the instance numbers of parked vehicles, and the plurality of contents in the first sublist are all empty;

[0024] The second sublist includes multiple indexes and multiple contents. The multiple indexes in the second sublist include parked vehicles in the background static object list, and the multiple contents in the second sublist are all empty.

[0025] According to some exemplary embodiments, generating a matching list based on the radar point clouds of the same type and the visible object number list further includes:

[0026] Filtering out points whose instance numbers are not equal to a first specified value in the first radar point cloud of the same type to form a point cloud of the moving vehicle; and

[0027] According to the correspondence between each index in the first sublist and the instance number of each point in the point cloud of the moving vehicle, each point in the point cloud of the moving vehicle is stored in a plurality of contents in the first sublist to obtain an updated first sublist.

[0028] According to some exemplary embodiments, generating a matching list based on the radar point clouds of the same type and the visible object number list further includes:

[0029] Filtering points whose instance numbers are equal to a first specified value in the first radar point cloud of the same type to form a point cloud of the parked vehicle;

[0030] For each point in the point cloud of the parked vehicle, traverse the background static object list;

[0031] When an i-th point in the point cloud of the parked vehicle is enclosed within the three-dimensional bounding box of the j-th static object in the background static object list, determining a correspondence between the i-th point and the j-th static object, where i is greater than or equal to 1 and less than or equal to the number of points in the point cloud of the parked vehicle, and j is greater than or equal to 1 and less than or equal to the number of static objects in the background static object list;

[0032] According to the corresponding relationship, each point in the point cloud of the parked vehicle is stored in correspondence with multiple contents in the second sub-list to obtain an updated second sub-list.

[0033] The matching list includes an updated first sub-list and an updated second sub-list.

[0034] According to some exemplary embodiments, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category specifically includes: selecting points with category numbers corresponding to pedestrians from the initial radar point cloud to form a second radar point cloud of the same category.

[0035] According to some exemplary embodiments, generating a matching list based on the radar point clouds of the same type and the visible object number list specifically includes:

[0036] According to the visible object number list, a first matching result is formed,

[0037] Wherein, the first matching result includes a third sublist;

[0038] The third sublist includes multiple indexes and multiple contents. The multiple indexes in the third sublist include instance numbers of pedestrians recorded in the visible object number list, and the multiple contents in the third sublist are all empty.

[0039] According to some exemplary embodiments, generating a matching list based on the radar point clouds of the same type and the visible object number list further includes:

[0040] According to the correspondence between each index in the third sublist and the instance number of each point in the second radar point cloud of the same type, each point in the second radar point cloud of the same type is stored in a plurality of contents in the third sublist to obtain an updated third sublist.

[0041] According to some exemplary embodiments, the method further includes: obtaining a list of controllable foreground objects of pedestrians, wherein the list of controllable foreground objects of pedestrians includes instance numbers, three-dimensional spatial position information, and three-dimensional bounding box vertex position information of each pedestrian in the foreground of the simulation environment.

[0042] According to some exemplary embodiments, the method further comprises:

[0043] comparing each index in the third sublist with the instance number of each pedestrian in the controllable foreground object list of pedestrians;

[0044] When the index in the third sublist is not in the controllable foreground object list of the pedestrian, remove the index and the content corresponding to the index from the third sublist to update the third sublist, and determine the updated third sublist as the fourth sublist.

[0045] The matching list includes the fourth sub-list.

[0046] According to some exemplary embodiments, the method further comprises:

[0047] Projecting the three-dimensional bounding box vertex position information of each background static object in the background static object list onto the two-dimensional camera plane to obtain the two-dimensional bounding box list of the background static object;

[0048] Obtain an instance segmentation map, wherein the virtual camera that outputs the instance segmentation map has the same configuration as the virtual camera that outputs the image to be labeled, and each pixel in the instance segmentation map includes a type number of the object to which the pixel belongs;

[0049] For each two-dimensional bounding box in the list of two-dimensional bounding boxes of background static objects: determining, based on the category number corresponding to each pixel in the instance segmentation map, the number of pixels in the two-dimensional bounding box that contain the same type of background static object in the instance segmentation map; calculating the ratio of the number of pixels to the total number of pixels in the two-dimensional bounding box; if the ratio exceeds a predetermined threshold, retaining the two-dimensional bounding box; if the ratio does not exceed the predetermined threshold, discarding the two-dimensional bounding box; and

[0050] The updated two-dimensional bounding box list is used as the labeling result of the two-dimensional bounding box of the background static object on the camera plane of the virtual camera.

[0051] According to some exemplary embodiments, the static objects in the background static object list include at least one of a traffic light, a traffic sign, a guardrail, and a tree.

[0052] According to a second aspect of the present application, a labeling device for autonomous driving simulation is provided, the device comprising:

[0053] a same-type radar point cloud acquisition module, configured to: acquire a same-type radar point cloud, wherein each point in the same-type radar point cloud contains the following information about the detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point, wherein each point in the same-type radar point cloud has the same type number;

[0054] a visible object number list generating module, configured to: obtain the instance number of each point in the radar point cloud of the same type, and generate a visible object number list according to the instance number of each point;

[0055] a matching list generating module, configured to generate a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes a plurality of indexes and a plurality of contents, the plurality of contents and the plurality of indexes respectively corresponding one-to-one, at least one index of the plurality of indexes includes an instance number in the visible object number list, and the plurality of contents include points in the radar point cloud of the same type;

[0056] A camera information acquisition module is used to: acquire camera information, wherein the camera information includes intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated; and

[0057] A two-dimensional bounding box annotation module is used to: annotate a two-dimensional bounding box of the detected object on the camera plane of the virtual camera based on the matching list and the camera information,

[0058] The step of marking a two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera according to the matching list and the camera information specifically includes:

[0059] Projecting a plurality of points in the content having the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain a plurality of projection points located on the camera plane; and

[0060] A two-dimensional bounding box surrounding the multiple projection points is determined, where the two-dimensional bounding box surrounding the multiple projection points is a two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

[0061] According to the third aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method as described above.

[0062] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the method described above.

[0063] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program, which implements the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0065] Figure 1 A schematic diagram of a scene diagram of an autonomous driving simulation according to an embodiment of the present application is shown;

[0066] Figure 2 is a flowchart of a labeling method applied to autonomous driving simulation according to some exemplary embodiments of the present application;

[0067] Figure 3 is a flow chart of a labeling method for labeling a vehicle according to some exemplary embodiments of the present application;

[0068] Figure 4 yes Figure 3 An exemplary flow chart of the steps of generating a match list in the method shown;

[0069] Figure 5 is a flowchart of a marking method for marking pedestrians according to some exemplary embodiments of the present application;

[0070] Figure 6 yes Figure 5 An exemplary flow chart of the steps of generating a match list in the method shown;

[0071] Figure 7 yes Figure 5 An exemplary flow chart of the steps of labeling a two-dimensional bounding box of a pedestrian in the method shown;

[0072] Figure 8 is a flow chart of a labeling method for labeling static objects such as traffic lights according to some exemplary embodiments of the present application;

[0073] Figure 9 A block diagram schematically illustrates a structure of a labeling device applied to autonomous driving simulation according to an embodiment of the present application; and

[0074] Figure 10 A block diagram of an electronic device suitable for implementing the marking method according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0075] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0076] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0077] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0078] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0079] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0080] First, let’s explain the technical terms that appear in this article as follows:

[0081] Point cloud: A massive amount of point cloud data that expresses the spatial distribution and surface characteristics of an object in a single spatial reference frame. For example, point cloud data can be acquired using sensors such as lidar and millimeter-wave radar.

[0082] Radar point cloud: point cloud data obtained by sensors such as lidar and millimeter-wave radar.

[0083] Autonomous driving simulation: Using mathematical modeling to digitally recreate autonomous driving application scenarios, establishing system models that are as close to the real world as possible. Simulation testing allows for analysis and research, ultimately testing and validating autonomous driving systems and algorithms. To achieve this, autonomous driving simulation software has gradually emerged. Autonomous driving simulation software primarily includes the following components: Static scene construction: It provides the ability to edit static scenes, such as roads and surrounding infrastructure, and to build or generate large-scale virtual road environments based on real road networks or high-precision maps. Dynamic scene simulation: It simulates the movements of various traffic participants and generates traffic flows similar to real-world scenarios, or can recreate scenarios based on real-world data. Sensor models: It includes or supports importing autonomous driving sensor models such as cameras, lidar, millimeter-wave radar, GPS, ultrasonic radar, and IMU, supporting various sensor model output methods. Vehicle dynamics models: It includes or supports editable vehicle dynamics models, and provides a rich set of interfaces for connecting to ADAS and autonomous driving systems. Currently, various autonomous driving simulation software or platforms on the market incorporate these components, each with its own unique characteristics. For example, CARLA is an autonomous driving simulation software or platform used for the development, training, and validation of autonomous driving systems. CARLA is developed on the Unreal Engine, using a server-based and multi-client architecture. It provides simple automated behavior simulations for vehicles and pedestrians, and a comprehensive set of Python interfaces for controlling vehicles and traffic lights in a scene. This facilitates co-simulation with autonomous driving systems, enabling decision-making systems and end-to-end reinforcement learning training.

[0084] 3D bounding box: A rectangular box that completely surrounds an object in 3D space.

[0085] 2D bounding box: A rectangle that completely encloses an object in a 2D plane.

[0086] Instance number: A unique number that can identify an object in the simulation environment.

[0087] Category number: A unique number that can identify a type of object in the simulation environment.

[0088] Figure 1 The following schematically shows a scene diagram of an autonomous driving simulation according to an embodiment of the present application. For example, Figure 1 The diagram shown may be a 1-frame simulation scene diagram generated in the autonomous driving simulation software, for example, the autonomous driving simulation software may be CARLA software. Figure 1 In the illustrated example, a three-dimensional annotation box 100 of a vehicle is exemplarily shown.

[0089] In autonomous driving simulation software such as CARLA, each frame of the simulation may include the following information:

[0090] (1) Radar point cloud: Each point in the radar point cloud contains the following information about the detected object: the instance number of the detected object, the type number of the detected object, and the three-dimensional spatial position information of the point;

[0091] (2) Instance segmentation map: The virtual camera that outputs the instance segmentation map needs to have the same configuration as the virtual camera that outputs the image to be labeled (for example, the same configuration means that the camera's intrinsic and extrinsic parameters are consistent). Each pixel in the instance segmentation map contains the type number of the object to which the pixel belongs;

[0092] (3) Camera information: the internal parameters, external parameters, and position information of the virtual camera in three-dimensional space;

[0093] (4) Controllable foreground object list: a list of all objects of a certain type in the current simulation environment (e.g., all moving vehicles, all pedestrians). Each item in the list contains an object instance number, the object's position information in 3D space, and the object's 3D bounding box vertex position information;

[0094] (5) Background static object list: A list of all static objects of a certain type in the current simulation environment (for example, all parked vehicles, traffic lights, and traffic signs). Each item in the list contains the position information of the object in three-dimensional space and the vertex position information of the object's three-dimensional bounding box.

[0095] In the actual development and testing process, it is necessary to obtain the two-dimensional bounding box of the object in the image captured by the virtual camera in the simulation.

[0096] For example, the methods provided in the CARLA software documentation describe a method for annotating 2D bounding boxes for vehicles and traffic lights: obtain a list of 3D bounding boxes for the objects being measured, project the 3D bounding box vertices onto the 2D camera plane, and obtain a list of 2D bounding boxes. However, the inventors discovered that this method provided in the CARLA documentation fails to account for occlusion relationships between objects, resulting in inaccurate 2D bounding box annotation results.

[0097] For example, some open-source projects have developed secondary features based on CARLA. These use the points of the segmented radar point cloud and the 3D bounding boxes of the controllable foreground object list to filter out unobstructed controllable foreground objects. The vertices of the 3D bounding boxes of the controllable foreground objects are then projected onto the 2D camera plane to obtain a 2D bounding box. However, the inventors discovered that these open-source projects only annotate moving vehicles and do not cover parked vehicles. Furthermore, the 2D bounding boxes obtained by directly projecting the 3D bounding box vertices of the objects onto the 2D camera plane are often inaccurate and incompact for vehicles.

[0098] Based on this, an embodiment of the present application provides a labeling method for autonomous driving simulation, the method comprising: obtaining radar point clouds of the same type, each point in the radar point cloud of the same type containing the following information of the detected object: the instance number of the detected object, the type number of the detected object and the three-dimensional spatial position information of the point, and the type numbers of the points in the radar point cloud of the same type are the same; obtaining the instance numbers of the points in the radar point cloud of the same type, and generating a visible object number list according to the instance numbers of the points; generating a matching list according to the radar point cloud of the same type and the visible object number list, wherein the matching list comprises multiple indexes and multiple contents, the multiple contents and the multiple indexes respectively corresponding one-to-one, at least one index among the multiple indexes comprises the instance number in the visible object number list, and the multiple contents comprise points in the radar point cloud of the same type; obtaining camera information, the camera information including the intrinsic parameters, extrinsic parameters and position information of the virtual camera in the three-dimensional space that outputs the image to be annotated; and annotating the two-dimensional bounding box of the detected object on the camera plane of the virtual camera according to the matching list and the camera information, wherein the marking of the two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera according to the matching list and the camera information specifically includes: projecting multiple points in the content with the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points located on the camera plane; and determining a two-dimensional bounding box surrounding the multiple projection points, the two-dimensional bounding box surrounding the multiple projection points is the two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera. In the annotation method provided in the embodiment of the present application, the source of the two-dimensional bounding box is the bounding box surrounding the two-dimensional projection points of the segmented radar point cloud, rather than the two-dimensional projection points of the vertices of the three-dimensional bounding box, and the annotated two-dimensional bounding box is more compact and accurate.

[0099] In an exemplary embodiment of the present application, the labeling method can label various types of objects in a simulation. For example, the various types of objects can include vehicles, pedestrians, traffic lights, traffic signs, and other types of objects.

[0100] Figure 2 This is a flowchart of a labeling method applied to autonomous driving simulation according to some exemplary embodiments of the present application. The method may include steps S210 to S250.

[0101] It should be noted that, in the embodiments of the present application, unless otherwise specified, the execution order of the various steps included in the method is not limited to the order in which they are recorded in this document. In the absence of conflict, the various steps can be executed in parallel or in any other order.

[0102] In step S210, a radar point cloud of the same type is obtained, where each point in the radar point cloud of the same type contains the following information of the detected object: the instance number of the detected object, the type number of the detected object, and the three-dimensional spatial position information of the point, and the type number of each point in the radar point cloud of the same type is the same.

[0103] In an embodiment of the present application, obtaining radar point clouds of the same type may include: obtaining an initial radar point cloud, each point in the initial radar point cloud containing the following information of the detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point; and selecting points with the same type number from the initial radar point cloud to form radar point clouds of the same type.

[0104] In some exemplary embodiments, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category may specifically include: selecting points with category numbers corresponding to vehicles from the initial radar point cloud to form a first radar point cloud of the same category.

[0105] Alternatively or additionally, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category may specifically include: selecting points with category numbers corresponding to pedestrians from the initial radar point cloud to form a second radar point cloud of the same category.

[0106] That is to say, in the embodiment of the present application, the radar point cloud of the same type can be the radar point cloud of all vehicles in the simulation, the radar point cloud of all pedestrians, and the radar point cloud of all static objects such as traffic lights or traffic signs.

[0107] In step S220 , the instance number of each point in the radar point cloud of the same type is obtained, and a visible object number list is generated according to the instance number of each point.

[0108] For example, when the radar point clouds of the same type are radar point clouds of all vehicles in the simulation, the visible object number list includes instance numbers of each point in the radar point clouds of all vehicles in the simulation.

[0109] In a case where the radar point clouds of the same type are radar point clouds of all pedestrians in the simulation, the visible object number list includes instance numbers of respective points in the radar point clouds of all pedestrians in the simulation.

[0110] In a case where the radar point clouds of the same type are radar point clouds of all traffic lights and traffic signs in the simulation, the visible object number list includes instance numbers of respective points in the radar point clouds of all traffic lights and traffic signs in the simulation.

[0111] In an embodiment of the present application, objects detected by points in the radar point cloud have taken into account the occlusion relationship between objects. Therefore, for the instances corresponding to each instance number in the visible object number list, the occlusion relationship between objects has been taken into account, that is, the occlusion problem has been solved.

[0112] In step S230, a matching list is generated based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes multiple indexes and multiple contents, the multiple contents and the multiple indexes correspond one-to-one, at least one index among the multiple indexes includes an instance number in the visible object number list, and the multiple contents include points in the radar point cloud of the same type.

[0113] In other words, the match list includes multiple items, each of which consists of an index and a content. The index and content correspond one-to-one, where the index is the instance number in the visible object number list and the content is the point in the radar point cloud. In other words, the match list matches the instance number with the point in the radar point cloud.

[0114] In step S240 , camera information is obtained, where the camera information includes intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated.

[0115] For example, a two-dimensional bounding box is annotated in the output image of a virtual camera, and the virtual camera can be called a virtual camera that outputs the image to be annotated.

[0116] It should be understood that the camera intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc. The camera extrinsic parameters refer to the positional relationship between the camera and the radar. The "camera position information in three-dimensional space" refers to the camera's parameters in the world coordinate system, such as the camera's position, rotation direction, etc.

[0117] In step S250 , a two-dimensional bounding box of the detected object on the camera plane of the virtual camera is marked according to the matching list and the camera information.

[0118] In an embodiment of the present application, marking the two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera based on the matching list and the camera information can specifically include: projecting multiple points in the content with the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points located on the camera plane; and determining a two-dimensional bounding box surrounding the multiple projection points, wherein the two-dimensional bounding box surrounding the multiple projection points is the two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

[0119] In some exemplary embodiments, the method may further include: obtaining a background static object list of a certain type of background objects, wherein the background static object list includes 3D spatial position information and 3D bounding box vertex position information of each static object in the simulation environment.

[0120] In some exemplary embodiments, the method may further include: obtaining a list of controllable foreground objects of a certain type of foreground objects, for example, the list of controllable foreground objects of a certain type of foreground objects includes the instance number, three-dimensional spatial position information and three-dimensional bounding box vertex position information of each pedestrian in the foreground of the simulation environment.

[0121] Below, the embodiments of the present application will be further described in detail by taking four types of objects, namely vehicles, pedestrians, traffic lights, and traffic signs, as examples.

[0122] It should be noted that in the simulation scenarios provided in the embodiments of this application, vehicles include both moving and parked vehicles. Moving vehicles are foreground dynamic objects, while parked vehicles are background static objects. Pedestrians are foreground dynamic objects, and traffic lights and traffic signs are background static objects. These four types of objects are typical representatives of foreground dynamic objects and background dynamic objects, and the embodiments of this application are not limited to these four types of objects.

[0123] Figure 3 3 is a flow chart of a vehicle labeling method according to some exemplary embodiments of the present application. The method may include steps S310 to S340.

[0124] In step S310 , an initial radar point cloud is obtained, where each point in the initial radar point cloud contains the following information of the detected object: the instance number of the detected object, the type number of the detected object, and the three-dimensional spatial position information of the point.

[0125] In some exemplary embodiments, the points in the initial radar point cloud correspond to multiple categories of objects in the simulation scene, for example, multiple categories of objects such as vehicles, pedestrians, traffic lights, traffic signs, trees, guardrails, etc. in the simulation environment.

[0126] In step S320, the initial radar point cloud is processed. For example, the processing of the initial radar point cloud includes: selecting points with the same category number from the initial radar point cloud to form radar point clouds of the same category.

[0127] In this embodiment, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category may specifically include selecting points with category numbers corresponding to vehicles from the initial radar point cloud to form a first radar point cloud of the same category. In other words, the points in the first radar point cloud of the same category correspond to all vehicles in the simulation.

[0128] In step S330 , the instance number of each point in the first radar point cloud of the same type is obtained, and a visible object number list is generated according to the instance number of each point.

[0129] For example, in this embodiment, the visible object number list includes instance numbers of various points in the radar point clouds of all vehicles in the simulation.

[0130] In step S340 , a matching list is generated according to the first radar point cloud of the same type and the visible object number list.

[0131] Figure 4 yes Figure 3 An exemplary flow chart of the steps of generating a matching list in the method shown in FIG. Figure 3 and Figure 4 The step of generating a matching list may include sub-steps S3410 to S3440.

[0132] In sub-step S3410, the visible object number list and the background static object list are matched to form a first matching result.

[0133] In this embodiment, the first matching result includes a first sub-list and a second sub-list, wherein the first sub-list is a list of moving vehicles and the second sub-list is a list of parked vehicles.

[0134] The first sub-list includes multiple indexes and multiple contents. The multiple indexes in the first sub-list include the instance numbers of the moving vehicles recorded in the visible object number list. The instance numbers of the moving vehicles are not equal to the first specified value, wherein the first specified value represents the instance number of the parked vehicle. The multiple contents in the first sub-list are all empty.

[0135] The second sublist includes multiple indexes and multiple contents. The multiple indexes in the second sublist include parked vehicles in the background static object list, and the multiple contents in the second sublist are all empty.

[0136] For example, using CARLA software, the instance number of a moving vehicle is non-zero, and the instance number of a parked vehicle is 0. In this case, each item in the first sublist indexes all non-zero instance numbers in the visible object number list of a vehicle, and the first sublist is empty. Each item in the second sublist indexes a vehicle in the background static number list, and the second sublist is empty.

[0137] For the first sub-list, that is, the sub-list for traveling vehicles, the following sub-steps S3420 and S3430 may be executed.

[0138] In sub-step S3420, points in the first radar point cloud of the same type whose instance numbers are not equal to a first specified value are filtered out to form a point cloud of a moving vehicle, wherein the first specified value represents the instance number of a parked vehicle.

[0139] For example, the first prescribed value may be 0, but the embodiments of the present application do not specifically limit the first prescribed value.

[0140] In sub-step S3430, based on the correspondence between each index in the first sub-list and the instance number of each point in the point cloud of the moving vehicle, each point in the point cloud of the moving vehicle is stored in a plurality of contents in the first sub-list to obtain an updated first sub-list.

[0141] For example, in the updated first sublist, each index in the first sublist is all non-zero instance numbers in the visible object number list of the vehicle, and each content of the first sublist is the point of the moving vehicle corresponding to the instance number.

[0142] For the second sub-list, ie, the sub-list for parked vehicles, the following sub-steps S3440 to S3470 may be executed.

[0143] In sub-step S3440 , points with instance numbers equal to a first specified value in the first radar point cloud of the same type are screened out to form a point cloud of the parked vehicle.

[0144] For example, using CARLA software, points with instance numbers equal to 0 can be filtered out from all vehicle radar point clouds, so that point clouds of all parked vehicles can be formed.

[0145] Because all parked vehicle point clouds in the simulation have the same instance number, for example, 0, it is impossible to determine which parked vehicle each point in the parked vehicle point cloud corresponds to. In the embodiment of the present application, the parked vehicle point cloud is further determined by traversing the background static object list to determine the corresponding parked vehicle for each point in the parked vehicle point cloud.

[0146] In sub-step S3450, the background static object list is traversed for each point in the point cloud of the parked vehicle.

[0147] In sub-step S3460, when the i-th point in the point cloud of the parked vehicle is enclosed within the three-dimensional bounding box of the j-th static object in the background static object list, a correspondence between the i-th point and the j-th static object is determined, where i is greater than or equal to 1 and less than or equal to the number of points in the point cloud of the parked vehicle, and j is greater than or equal to 1 and less than or equal to the number of static objects in the background static object list.

[0148] In sub-step S3470, each point in the point cloud of the parked vehicle is stored in correspondence with multiple contents in the second sub-list according to the corresponding relationship to obtain an updated second sub-list.

[0149] For example, in the updated second sub-list, each index in the second sub-list is a parked vehicle in the background static object list, and each content in the second sub-list is a parked vehicle point corresponding to the parked vehicle.

[0150] In an embodiment of the present application, the matching list includes an updated first sub-list and an updated second sub-list.

[0151] Return to reference Figure 2 After forming the matching list, step S250 may be executed, that is, marking a two-dimensional bounding box of the detected object on the camera plane of the virtual camera according to the matching list and the camera information.

[0152] Specifically, multiple points in the content with the same index in the matching list are projected onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points located on the camera plane; and a two-dimensional bounding box surrounding the multiple projection points is determined, and the two-dimensional bounding box surrounding the multiple projection points is the two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

[0153] For example, multiple points in the content of each item in the updated first sublist of the matching list can be projected onto the two-dimensional camera plane of the virtual camera in combination with the camera information to obtain multiple projected points on the camera plane. A two-dimensional bounding box enclosing the multiple projected points is then determined. The two-dimensional bounding box enclosing the multiple projected points serves as the two-dimensional bounding box of the moving vehicle in the two-dimensional camera plane of the virtual camera. In this way, the two-dimensional bounding box of the moving vehicle in the simulation environment is annotated.

[0154] For another example, multiple points in the content of each item in the updated second sublist of the matching list can be projected onto the two-dimensional camera plane of the virtual camera in combination with the camera information to obtain multiple projected points on the camera plane. A two-dimensional bounding box enclosing the multiple projected points is then determined. The two-dimensional bounding box enclosing the multiple projected points serves as the two-dimensional bounding box of the parked vehicle in the two-dimensional camera plane of the virtual camera. In this way, the two-dimensional bounding box of the parked vehicle in the simulation environment is annotated.

[0155] In an embodiment of the present application, the source of the vehicle's two-dimensional bounding box is the bounding box surrounding the two-dimensional projection points of the vehicle's radar point cloud, rather than the two-dimensional projection points of the vehicle's three-dimensional bounding box vertices, which makes the generated two-dimensional bounding box more compact and accurate.

[0156] Furthermore, in an embodiment of the present application, in the process of marking vehicles, both moving vehicles and parked vehicles are marked at the same time, making the marking result more effective.

[0157] Figure 5 This is a flowchart of a marking method for marking pedestrians according to some exemplary embodiments of the present application. The method may include steps S510 to S540.

[0158] In step S510 , an initial radar point cloud is obtained, where each point in the initial radar point cloud contains the following information of the detected object: the instance number of the detected object, the type number of the detected object, and the three-dimensional spatial position information of the point.

[0159] In step S520, the initial radar point cloud is processed. For example, the processing of the initial radar point cloud includes: selecting points with the same category number from the initial radar point cloud to form radar point clouds of the same category.

[0160] In this embodiment, selecting points with the same category number from the initial radar point cloud to form a radar point cloud of the same category may specifically include selecting points with category numbers corresponding to pedestrians from the initial radar point cloud to form a second radar point cloud of the same category. In other words, the points in the second radar point cloud of the same category correspond to all pedestrians in the simulation.

[0161] In step S530 , the instance number of each point in the second radar point cloud of the same type is obtained, and a visible object number list is generated according to the instance number of each point.

[0162] For example, in this embodiment, the visible object number list includes instance numbers of each point in the radar point cloud of all pedestrians in the simulation.

[0163] In step S540 , a matching list is generated according to the second radar point cloud of the same type and the visible object number list.

[0164] Figure 6 yes Figure 5 An exemplary flow chart of the steps of generating a matching list in the method shown in FIG. Figure 5 and Figure 6 The step of generating a matching list may include sub-steps S5410 to S5420.

[0165] In sub-step S5410, a first matching result is formed based on the visible object number list, wherein the first matching result includes a third sub-list; the third sub-list includes multiple indexes and multiple contents, the multiple indexes in the third sub-list include instance numbers of pedestrians recorded in the visible object number list, and the multiple contents in the third sub-list are all empty.

[0166] For example, taking CARLA software as an example, the instance numbers of all pedestrians are non-zero values. In this case, each index in the third sublist is all non-zero instance numbers in the pedestrian's visible object number list, and the content of the third sublist is an empty list.

[0167] In sub-step S5420, based on the correspondence between each index in the third sub-list and the instance number of each point in the second radar point cloud of the same type, each point in the second radar point cloud of the same type is stored in a plurality of contents in the third sub-list to obtain an updated third sub-list.

[0168] For example, in the updated third sublist, each index in the third sublist is all non-zero instance numbers in the pedestrian's visible object number list, and each content in the third sublist is a pedestrian point corresponding to the instance number.

[0169] In this embodiment, the matching list may include an updated third sub-list.

[0170] Return to reference Figure 2 After forming the matching list, step S250 may be executed, that is, marking a two-dimensional bounding box of the detected object (eg, a pedestrian) on the camera plane of the virtual camera according to the matching list and the camera information.

[0171] Optionally, Figure 7 yes Figure 5 An exemplary flow chart of the steps of labeling a pedestrian's two-dimensional bounding box in the method shown.

[0172] Combined with reference Figure 2 、 Figure 5 and Figure 7 In some exemplary embodiments of the present application, the method may further include steps S710 to S730.

[0173] In step S710 , a controllable foreground object list of pedestrians is obtained, wherein the controllable foreground object list of pedestrians includes instance numbers, three-dimensional spatial position information, and three-dimensional bounding box vertex position information of each pedestrian in the foreground of the simulation environment.

[0174] In step S720 , each index in the third sublist is compared with the instance number of each pedestrian in the controllable foreground object list of pedestrians.

[0175] In step S730, when the index in the third sublist is not in the list of controllable foreground objects of the pedestrian, the index and the content corresponding to the index are removed from the third sublist to update the third sublist, and the updated third sublist is determined as the fourth sublist.

[0176] In this embodiment, the matching list includes the fourth sub-list.

[0177] In simulation software such as CARLA, on the one hand, the point in the radar point cloud corresponding to the two-wheeled vehicle driver is numbered as a pedestrian. On the other hand, in the list of controllable foreground objects, the instance number of the two-wheeled vehicle driver is not in the list of controllable foreground objects for pedestrians, but in the list of controllable foreground objects for vehicles. The inventors have discovered that in this case, it is more reasonable to label the two-wheeled vehicle driver and the two-wheeled vehicle as a single entity—that is, to label the entire entity as a vehicle. In an embodiment of the present application, by executing steps S710 to S730, the two-wheeled vehicle driver can be removed from the pedestrian list, making the labeling result more accurate and reasonable.

[0178] Return to reference Figure 2 After forming the matching list (e.g., the fourth sublist), step S250 may be executed, i.e., marking a two-dimensional bounding box of the detected object (e.g., a pedestrian) on the camera plane of the virtual camera based on the fourth sublist and the camera information.

[0179] Specifically, multiple points in the content with the same index in the fourth sublist are projected onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points located on the camera plane; and a two-dimensional bounding box surrounding the multiple projection points is determined, and the two-dimensional bounding box surrounding the multiple projection points is the two-dimensional bounding box of the pedestrian in the two-dimensional camera plane of the virtual camera.

[0180] For example, multiple points in the content of each item in the updated fourth sublist of the matching list can be projected onto the two-dimensional camera plane of the virtual camera in combination with the camera information to obtain multiple projected points on the camera plane. A two-dimensional bounding box enclosing the multiple projected points is then determined. The two-dimensional bounding box enclosing the multiple projected points serves as the two-dimensional bounding box of the pedestrian in the two-dimensional camera plane of the virtual camera. In this way, the two-dimensional bounding box of the pedestrian in the simulated environment is annotated.

[0181] In an embodiment of the present application, the pedestrian's two-dimensional bounding box is derived from the bounding box of the two-dimensional projection points surrounding the vehicle radar point cloud, rather than the two-dimensional projection points of the pedestrian's three-dimensional bounding box vertices, which makes the generated two-dimensional bounding box more compact and accurate.

[0182] Figure 8 The flowchart of the labeling method for labeling static objects such as traffic lights according to some exemplary embodiments of the present application may include steps S810 to S840. It should be noted that in the embodiments of the present application, a traffic light is used as an example to illustrate the labeling method for labeling static objects. However, the embodiments of the present application are not limited to this example. The labeling method proposed in the embodiments of the present application can also be applied to other types of static objects such as traffic signs, trees, and guardrails.

[0183] In step S810, the three-dimensional bounding box vertex position information of each background static object in the background static object list is projected onto the two-dimensional camera plane to obtain a two-dimensional bounding box list of the background static object. For example, the background static object list may be a background static object list of a traffic light.

[0184] In step S820, an instance segmentation map is obtained, wherein the virtual camera that outputs the instance segmentation map has the same configuration as the virtual camera that outputs the image to be labeled, and each pixel in the instance segmentation map includes the type number of the object to which the pixel belongs, for example, the type number of a traffic light.

[0185] It should be noted that the “same configuration” here may mean that the intrinsic parameters and extrinsic parameters of the two virtual cameras are the same.

[0186] In step S830, for each two-dimensional bounding box in the list of two-dimensional bounding boxes of background static objects, the following steps are performed: Based on the category number corresponding to each pixel in the instance segmentation map, the number of pixels within the two-dimensional bounding box that contain the same type of background static object in the instance segmentation map is determined; the ratio of the number of pixels to the total number of pixels in the two-dimensional bounding box is calculated; if the ratio exceeds a preset threshold, the two-dimensional bounding box is retained; if the ratio does not exceed the preset threshold, the two-dimensional bounding box is discarded. In this manner, the two-dimensional bounding box list is updated.

[0187] It should be noted that if the ratio does not exceed the pre-set threshold, it indicates that the traffic light is occluded, so the two-dimensional bounding box of the traffic light needs to be discarded. In the embodiment of the present application, the instance segmentation map is used to determine the occlusion of static objects such as traffic lights and traffic signs, making the annotation results more stable and accurate.

[0188] In step S840 , the updated two-dimensional bounding box list is used as a labeling result of the two-dimensional bounding boxes of the background static objects on the camera plane of the virtual camera.

[0189] In the labeling method provided in the embodiment of the present application, dynamic objects such as vehicles and pedestrians and static objects such as traffic lights and traffic labels can be accurately and compactly labeled. In other words, the labeling method provided in the embodiment of the present application has strong compatibility, stable and accurate labeling results, and can be extended to the labeling of more types of objects in autonomous driving simulation.

[0190] Based on the above-mentioned annotation method, the embodiment of the present application also provides an annotation device for use in autonomous driving simulation. Figure 9 The device is described in detail.

[0191] Figure 9 The structural block diagram of the annotation device applied to autonomous driving simulation according to an embodiment of the present application is schematically shown.

[0192] like Figure 9 As shown, the labeling device applied to autonomous driving simulation may include a radar point cloud acquisition module 910 of the same type, a visible object number list generation module 920, a matching list generation module 930, a camera information acquisition module 940 and a two-dimensional bounding box labeling module 950.

[0193] The same-type radar point cloud acquisition module 910 is used to: acquire the same-type radar point cloud, where each point in the same-type radar point cloud contains the following information of the detected object: the instance number of the detected object, the type number of the detected object, and the three-dimensional spatial position information of the point. The type number of each point in the same-type radar point cloud is the same.

[0194] In some exemplary embodiments, the same type of radar point cloud acquisition module 910 may be used to perform the operations S210, S310, S320, S510, and S520 described above, which will not be described in detail herein.

[0195] The visible object number list generating module 920 is configured to obtain the instance number of each point in the radar point cloud of the same type, and generate a visible object number list according to the instance number of each point.

[0196] In some exemplary embodiments, the visible object number list generating module 920 may be configured to execute the operations S220 , S330 , and S530 described above, which will not be described in detail herein.

[0197] The matching list generation module 930 is used to generate a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes multiple indexes and multiple contents, the multiple contents and the multiple indexes correspond one-to-one, at least one index among the multiple indexes includes an instance number in the visible object number list, and the multiple contents include points in the radar point cloud of the same type.

[0198] In some exemplary embodiments, the matching list generation module 930 may be used to perform the operations S230, S340, and S540 described above, which will not be described in detail here.

[0199] The camera information acquisition module 940 is used to acquire camera information, where the camera information includes intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated.

[0200] In some exemplary embodiments, the camera information acquisition module 940 may be used to perform the operation S240 described above, which will not be described in detail here.

[0201] The two-dimensional bounding box annotation module 950 is configured to: annotate a two-dimensional bounding box of the detected object on the camera plane of the virtual camera according to the matching list and the camera information.

[0202] For example, marking the two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera based on the matching list and the camera information may specifically include: projecting multiple points in the content with the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain multiple projection points located on the camera plane; and determining a two-dimensional bounding box surrounding the multiple projection points, wherein the two-dimensional bounding box surrounding the multiple projection points is the two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

[0203] In some exemplary embodiments, the two-dimensional bounding box labeling module 950 may be used to perform the operation S250 described above, which will not be described in detail here.

[0204] Alternatively or additionally, the annotation apparatus for autonomous driving simulation may further include a background static object list acquisition module 960. Background static object list acquisition module 960 may be configured to acquire a background static object list. The background static object list includes 3D spatial position information and 3D bounding box vertex position information of each static object in the simulation environment.

[0205] Alternatively or additionally, the annotation device for autonomous driving simulation may further include a controllable foreground object list acquisition module 970. The controllable foreground object list acquisition module 970 may be used to acquire a list of controllable foreground objects. For example, the controllable foreground object list of pedestrians may include the instance number, 3D spatial position information, and 3D bounding box vertex position information of each pedestrian in the foreground of the simulation environment.

[0206] According to an embodiment of the present application, any multiple modules or units in the above-mentioned modules or units can be merged into one module or unit to realize, or any one module or unit therein can be split into multiple modules or units.Or, at least part of the functions of one or more modules or units in these modules or units can be combined with at least part of the functions of other modules or units, and realized in one module or unit.According to an embodiment of the present application, at least one of the above-mentioned modules or units can be at least partially realized as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be realized by hardware or firmware such as any other reasonable way of integrating or encapsulating a circuit, or realized in any one of three implementation modes of software, hardware and firmware or with any appropriate combination of several thereof.Or, at least one of the above-mentioned modules or units can be at least partially realized as a computer program module, which can execute corresponding functions when the computer program module is run.

[0207] Figure 10 A block diagram of an electronic device suitable for implementing the marking method according to an embodiment of the present application is schematically shown.

[0208] like Figure 10 As shown, the electronic device 1000 according to an embodiment of the present application includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. The processor 1001 may, for example, include a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include an onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present application.

[0209] Various programs and data required for the operation of the electronic device 1000 are stored in the RAM 1003. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. The processor 1001 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 1002 and / or the RAM 1003. It should be noted that the programs may also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0210] According to an embodiment of the present application, electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to bus 1004. Electronic device 1000 may further include one or more of the following components connected to I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. Communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to I / O interface 1005 as needed. Removable media 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 1010 as needed, so that computer programs read from the removable media can be installed into storage section 1008 as needed.

[0211] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0212] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 1002 and / or RAM 1003 described above and / or one or more memories other than ROM 1002 and RAM 1003.

[0213] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the annotation method provided in the embodiments of the present application.

[0214] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 1001. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0215] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1009, and / or installed from the removable medium 1011. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0216] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, the above-mentioned functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0217] According to an embodiment of the application, the program code for executing the computer program provided by an embodiment of the application can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level processes and / or object-oriented programming languages ​​and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or completely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0219] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of this application may be made, even if such combinations or combinations are not explicitly described in this application. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of this application may be made, without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A labeling method applied to autonomous driving simulation, characterized in that: The method comprises: Obtaining a radar point cloud of the same type, wherein each point in the radar point cloud of the same type includes the following information of the detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point, wherein each point in the radar point cloud of the same type has the same type number; Obtaining the instance number of each point in the radar point cloud of the same type, and generating a visible object number list according to the instance number of each point; Generate a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes a plurality of indexes and a plurality of contents, the plurality of contents and the plurality of indexes respectively correspond one-to-one, at least one index of the plurality of indexes includes an instance number in the visible object number list, and the plurality of contents include points in the radar point cloud of the same type; Obtaining camera information, the camera information including intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated; and Marking a two-dimensional bounding box of the detected object on the camera plane of the virtual camera based on the matching list and the camera information, The step of marking a two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera according to the matching list and the camera information specifically includes: Projecting a plurality of points in the content having the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain a plurality of projection points located on the camera plane; and A two-dimensional bounding box surrounding the multiple projection points is determined, where the two-dimensional bounding box surrounding the multiple projection points is a two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

2. The method according to claim 1, characterized in that The obtaining of the same type of radar point cloud comprises: Acquire an initial radar point cloud, wherein each point in the initial radar point cloud contains the following information of a detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point; and Points with the same category number are selected from the initial radar point cloud to form radar point clouds of the same category.

3. The method according to claim 2, characterized in that The method further includes: acquiring a background static object list, wherein the background static object list includes three-dimensional spatial position information and three-dimensional bounding box vertex position information of each static object in the simulation environment.

4. The method according to claim 3, characterized in that The selecting points with the same category number from the initial radar point cloud to form radar point clouds of the same category specifically includes: Points with category numbers corresponding to vehicles are selected from the initial radar point cloud to form a first radar point cloud of the same category.

5. The method according to claim 4, characterized in that Generating a matching list based on the radar point clouds of the same type and the visible object number list specifically includes: Matching the visible object number list with the background static object list to form a first matching result, Wherein, the first matching result includes a first sublist and a second sublist; The first sublist includes a plurality of indexes and a plurality of contents, wherein the plurality of indexes in the first sublist include instance numbers of moving vehicles recorded in the visible object number list, the instance numbers of the moving vehicles are not equal to a first specified value, wherein the first specified value represents the instance numbers of parked vehicles, and the plurality of contents in the first sublist are all empty; The second sublist includes multiple indexes and multiple contents. The multiple indexes in the second sublist include parked vehicles in the background static object list, and the multiple contents in the second sublist are all empty.

6. The method according to claim 5, characterized in that The generating of a matching list based on the radar point clouds of the same type and the visible object number list specifically includes: Filtering out points whose instance numbers are not equal to a first specified value in the first radar point cloud of the same type to form a point cloud of the moving vehicle; and According to the correspondence between each index in the first sublist and the instance number of each point in the point cloud of the moving vehicle, each point in the point cloud of the moving vehicle is stored in a plurality of contents in the first sublist to obtain an updated first sublist.

7. The method according to claim 6, characterized in that The generating of a matching list based on the radar point clouds of the same type and the visible object number list specifically includes: Filtering points having instance numbers equal to a first specified value in the first radar point cloud of the same type to form a point cloud of the parked vehicle; For each point in the point cloud of the parked vehicle, traverse the background static object list; When an i-th point in the point cloud of the parked vehicle is enclosed within the three-dimensional bounding box of the j-th static object in the background static object list, determining a correspondence between the i-th point and the j-th static object, where i is greater than or equal to 1 and less than or equal to the number of points in the point cloud of the parked vehicle, and j is greater than or equal to 1 and less than or equal to the number of static objects in the background static object list; According to the corresponding relationship, each point in the point cloud of the parked vehicle is stored in correspondence with multiple contents in the second sub-list to obtain an updated second sub-list. The matching list includes an updated first sub-list and an updated second sub-list.

8. The method according to any one of claims 3 to 7, characterized in that The selecting points with the same category number from the initial radar point cloud to form radar point clouds of the same category specifically includes: Points with category numbers corresponding to pedestrians are selected from the initial radar point cloud to form a second radar point cloud of the same category.

9. The method according to claim 8, characterized in that Generating a matching list based on the radar point clouds of the same type and the visible object number list specifically includes: According to the visible object number list, a first matching result is formed, Wherein, the first matching result includes a third sublist; The third sublist includes multiple indexes and multiple contents. The multiple indexes in the third sublist include instance numbers of pedestrians recorded in the visible object number list, and the multiple contents in the third sublist are all empty.

10. The method according to claim 9, characterized in that The generating of a matching list based on the radar point clouds of the same type and the visible object number list specifically includes: According to the correspondence between each index in the third sublist and the instance number of each point in the second radar point cloud of the same type, each point in the second radar point cloud of the same type is stored in a plurality of contents in the third sublist to obtain an updated third sublist.

11. The method according to claim 10, characterized in that The method further includes: obtaining a controllable foreground object list of pedestrians, wherein the controllable foreground object list of pedestrians includes instance numbers, three-dimensional spatial position information, and three-dimensional bounding box vertex position information of each pedestrian in the foreground of the simulation environment.

12. The method according to claim 11, characterized in that The method further comprises: comparing each index in the third sublist with the instance number of each pedestrian in the controllable foreground object list of pedestrians; When the index in the third sublist is not in the controllable foreground object list of the pedestrian, remove the index and the content corresponding to the index from the third sublist to update the third sublist, and determine the updated third sublist as the fourth sublist. The matching list includes the fourth sub-list.

13. The method according to any one of claims 3 to 7 and 9 to 12, characterized in that The method further comprises: Projecting the three-dimensional bounding box vertex position information of each background static object in the background static object list onto the two-dimensional camera plane to obtain the two-dimensional bounding box list of the background static object; Obtain an instance segmentation map, wherein the virtual camera that outputs the instance segmentation map has the same configuration as the virtual camera that outputs the image to be labeled, and each pixel in the instance segmentation map includes a type number of the object to which the pixel belongs; For each two-dimensional bounding box in the list of two-dimensional bounding boxes of background static objects: determining, based on the category number corresponding to each pixel in the instance segmentation map, the number of pixels in the two-dimensional bounding box that contain the same type of background static object in the instance segmentation map; calculating the ratio of the number of pixels to the total number of pixels in the two-dimensional bounding box; if the ratio exceeds a predetermined threshold, retaining the two-dimensional bounding box; if the ratio does not exceed the predetermined threshold, discarding the two-dimensional bounding box; and The updated two-dimensional bounding box list is used as the labeling result of the two-dimensional bounding box of the background static object on the camera plane of the virtual camera.

14. The method according to claim 13, characterized in that The static objects in the background static object list include at least one of traffic lights, traffic signs, guardrails and trees.

15. A labeling device used in autonomous driving simulation, characterized in that: The device comprises: a same-type radar point cloud acquisition module, configured to: acquire a same-type radar point cloud, wherein each point in the same-type radar point cloud contains the following information about the detected object: an instance number of the detected object, a type number of the detected object, and three-dimensional spatial position information of the point, wherein each point in the same-type radar point cloud has the same type number; a visible object number list generating module, configured to: obtain the instance number of each point in the radar point cloud of the same type, and generate a visible object number list according to the instance number of each point; a matching list generating module, configured to generate a matching list based on the radar point cloud of the same type and the visible object number list, wherein the matching list includes a plurality of indexes and a plurality of contents, the plurality of contents and the plurality of indexes respectively corresponding one-to-one, at least one index of the plurality of indexes includes an instance number in the visible object number list, and the plurality of contents include points in the radar point cloud of the same type; A camera information acquisition module is used to: acquire camera information, wherein the camera information includes intrinsic parameters, extrinsic parameters, and position information of the virtual camera in three-dimensional space that outputs the image to be annotated; and A two-dimensional bounding box annotation module is used to: annotate a two-dimensional bounding box of the detected object on the camera plane of the virtual camera based on the matching list and the camera information, The step of marking a two-dimensional bounding box of the detected object in the two-dimensional plane of the virtual camera according to the matching list and the camera information specifically includes: Projecting a plurality of points in the content having the same index in the matching list onto the two-dimensional camera plane of the virtual camera to obtain a plurality of projection points located on the camera plane; and A two-dimensional bounding box surrounding the multiple projection points is determined, where the two-dimensional bounding box surrounding the multiple projection points is a two-dimensional bounding box of the detected object in the two-dimensional camera plane of the virtual camera.

16. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images