Data annotation methods, devices, electronic equipment and storage media
By inserting virtual 3D bounding boxes into 3D point cloud data and selecting target bounding boxes for annotation, the problem of inaccurate annotation of 3D point cloud data is solved, and the accuracy and reliability of annotation are improved.
Patent Information
- Application Number
- CN202410579008.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-10
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-10
AI Technical Summary
The annotation effect of existing 3D point cloud data is poor, especially the annotation of missed obstacles is inaccurate.
By identifying missed obstacles in the environmental image, multiple virtual 3D bounding boxes are inserted, and a target virtual 3D bounding box is selected based on the relevance to annotate the 3D point cloud data.
It improves the labeling accuracy of 3D point cloud data, especially the labeling effect of missed obstacles, thus enhancing the accuracy and reliability of the data.
Smart Images

Figure CN118506316B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle technology, and more specifically, to a data annotation method, apparatus, electronic device, and computer-readable storage medium. Background Technology
[0002] In the field of vehicle technology, after a vehicle acquires 3D point cloud data of its surrounding environment, it can use bounding boxes containing point cloud data of obstacles to annotate the point cloud data of those obstacles. However, related technologies suffer from poor annotation results for 3D point cloud data. Summary of the Invention
[0003] This application proposes a data annotation method, apparatus, electronic device, and computer-readable storage medium to improve the above-mentioned deficiencies.
[0004] In a first aspect, embodiments of this application provide a data annotation method, the method comprising:
[0005] Based on the environmental images and 3D point cloud data collected by the vehicle at the target time point, the missed related obstacles are identified; the environmental images include at least one related obstacle located within the target detection area; the missed related obstacles refer to related obstacles in the environmental images and in the 3D point cloud data that have missing point cloud data;
[0006] Multiple virtual 3D bounding boxes are inserted within the relevant area; the relevant area is determined from the target detection area based on the missed detection of relevant obstacles in the environmental image.
[0007] Based on the correlation between multiple virtual 3D bounding boxes and the missed detection related obstacles, a target virtual 3D bounding box is selected from the multiple virtual 3D bounding boxes;
[0008] The point cloud data corresponding to the missed obstacles in the 3D point cloud data are labeled based on the target virtual 3D bounding box.
[0009] Secondly, embodiments of this application also provide a data annotation device, the device comprising:
[0010] The determination module is used to determine missed related obstacles based on the environmental images and 3D point cloud data collected by the vehicle at the target time point; the environmental images include at least one related obstacle located within the target detection area; missed related obstacles refer to related obstacles in the environmental images and in the 3D point cloud data that have missing point cloud data;
[0011] The insertion module is used to insert multiple virtual 3D bounding boxes within a relevant region; the relevant region is determined from the target detection region based on missed relevant obstacles in the environmental image.
[0012] The selection module is used to select a target virtual 3D bounding box from multiple virtual 3D bounding boxes based on the correlation between the multiple virtual 3D bounding boxes and the missed related obstacles.
[0013] The annotation module is used to annotate the point cloud data corresponding to missed obstacles in the 3D point cloud data based on the target virtual 3D bounding box.
[0014] Thirdly, embodiments of this application also provide an electronic device, which includes: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.
[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing processor-executable program code, which, when executed by the processor, causes the processor to perform the above-described method.
[0016] This application provides a data annotation method, apparatus, electronic device, and computer-readable storage medium. In this application, instead of directly using the point cloud data of the missing related obstacles to determine the 3D bounding box for annotation, multiple virtual 3D bounding boxes are inserted in the relevant area. Then, a target virtual 3D bounding box is selected from the multiple virtual 3D bounding boxes. The accuracy of the determined target virtual 3D bounding box is relatively high, thereby achieving accurate annotation of the point cloud data of the missing related obstacles based on the target virtual 3D bounding box and improving the annotation effect of 3D point cloud data.
[0017] Other features and advantages of the embodiments of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the embodiments of this application. The objects and other advantages of the embodiments of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart of a data annotation method according to an embodiment of this application is shown.
[0020] Figure 2 A flowchart of a data annotation method according to yet another embodiment of this application is shown.
[0021] Figure 3 A schematic diagram of a data annotation process according to an embodiment of this application is shown.
[0022] Figure 4 A structural block diagram of a data annotation apparatus according to an embodiment of this application is shown.
[0023] Figure 5 A structural block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. The components of the embodiments of the present application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are within the scope of protection of the present application.
[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0026] Please see Figure 1 , Figure 1 A flowchart of a data annotation method according to an embodiment of this application is shown, for use in an electronic device. The method includes:
[0027] S101. Based on the environmental images and 3D point cloud data collected by the vehicle at the target time point, identify the relevant obstacles that were missed.
[0028] In this application, the environmental image includes at least one relevant obstacle located within the target detection area; a missed relevant obstacle refers to a relevant obstacle in the environmental image and in the 3D point cloud data that has missing point cloud data.
[0029] The vehicle can be an electric vehicle or a gasoline-powered vehicle, and can be a sedan, SUV, bus, or truck, etc. The environmental image can be 2D data captured by a camera inside the vehicle, and the 3D point cloud data can be 3D data collected by radar inside the vehicle. The target detection area refers to the area within the detection range of the vehicle's radar (usually the area centered on the vehicle). If there are multiple radars, the target detection area refers to the area within the union of the detection ranges of all the radars.
[0030] A target time point refers to any point in time; it can be any point in time during the vehicle's movement or any point in time when the vehicle is stationary. For example, for a vehicle in motion, each current moment can be considered a target time point.
[0031] It is worth mentioning that, in this application, the vehicle may have multiple cameras, which capture images of the environment surrounding the vehicle at the target time point. The resulting images are then fused into a single image, which serves as the aforementioned environmental image. Similarly, if the vehicle has multiple radars, these radars collect data on the environment surrounding the vehicle at the target time point, generating individual 3D data for each radar. The 3D data from these radars are then fused to obtain the aforementioned 3D point cloud data.
[0032] Obstacles around a vehicle can include relevant obstacles within the target detection area and unrelated obstacles outside the target detection area. Obstacles can include pedestrians, vehicles, buildings, and plants.
[0033] The environmental image may include all relevant obstacles, and the 3D point cloud data may include the point cloud data of each relevant obstacle. The point cloud data of each relevant obstacle may be complete data or data with missing point cloud data.
[0034] For each relevant obstacle, if there is no missing point cloud data for that obstacle, it indicates that the amount of point cloud data for that obstacle in the 3D point cloud data is large and the point cloud data for that obstacle in the 3D point cloud data is relatively accurate. Therefore, the relevant obstacle is identified as a valid relevant obstacle. Based directly on the point cloud data of the valid relevant obstacles in the 3D point cloud data, the 3D bounding box surrounding the valid relevant obstacle is determined as a valid 3D bounding box. The 3D point cloud data is then directly labeled using this valid 3D bounding box. At this time, the valid 3D bounding box is relatively accurate, which can achieve accurate labeling of the 3D point cloud data.
[0035] For each relevant obstacle, if there is missing point cloud data for that obstacle, it indicates that the amount of point cloud data for that obstacle in the 3D point cloud data is small and inaccurate. Therefore, the obstacle is identified as a missed obstacle. If the 3D bounding box surrounding the missed obstacle is determined directly based on the point cloud data of the missed obstacle, the accuracy is poor. If the 3D point cloud data is directly labeled using the bounding box, the labeling accuracy is also poor. Therefore, after identifying the relevant obstacle as a missed obstacle, it is labeled according to the subsequent steps of this application.
[0036] In some implementations, S101 may include: determining the type or three-dimensional contour of each relevant obstacle through an environmental image, and then determining the estimated point cloud data of the relevant obstacle based on the type or three-dimensional contour of the relevant obstacle; if the estimated point cloud data of the relevant obstacle does not match the actual point cloud data of the relevant obstacle in the three-dimensional point cloud data, the relevant obstacle is determined to be a missed relevant obstacle.
[0037] Here, mismatch can refer to situations where the similarity between the estimated shape information in the estimated point cloud data of a related obstacle and the actual shape information in the actual point cloud data of the related obstacle is less than a similarity threshold, or the ratio of the data volume of the estimated point cloud data of a related obstacle to the data volume of the actual point cloud data of the related obstacle is greater than a ratio threshold. In this application, the similarity threshold and the ratio threshold can be values set based on requirements, for example, the similarity threshold and the ratio threshold are 0.8 and 1.25, respectively.
[0038] In another embodiment, S101 may include: obtaining the relevant two-dimensional detection region of each relevant obstacle from the environmental image; obtaining the relevant three-dimensional detection region of each relevant obstacle from the three-dimensional point cloud data; mapping the relevant three-dimensional detection regions of each relevant obstacle onto the camera coordinate system where the environmental image is located to obtain the relevant mapped two-dimensional region corresponding to each relevant obstacle; determining the region similarity between the relevant mapped two-dimensional region of each relevant obstacle and the relevant two-dimensional detection region; and obtaining relevant obstacles with a region similarity less than the region similarity threshold as missed relevant obstacles.
[0039] In this application, for each relevant obstacle, the region within a rectangular bounding box including the obstacle can be obtained from the environmental image as the relevant two-dimensional detection region of the obstacle, wherein the rectangular bounding box including the obstacle refers to the smallest bounding rectangle including the obstacle. Similarly, for each relevant obstacle, the region within a smallest cuboid bounding box including the obstacle can be obtained from the three-dimensional point cloud data as the relevant three-dimensional detection region of the obstacle.
[0040] In this application, the 3D point cloud data is based on the world coordinate system. The vehicle can store the coordinate system mapping relationship between the world coordinate system and the camera coordinate system of the camera (used to acquire environmental images) in the vehicle. According to the coordinate system mapping relationship, the relevant 3D detection areas of each relevant obstacle can be mapped to the camera coordinate system of the environmental image, so as to map each relevant 3D detection area into a 2D area, and obtain the relevant mapped 2D area corresponding to each relevant obstacle; wherein, one relevant 3D detection area corresponds to one relevant mapped 2D area.
[0041] In this application, when the relevant three-dimensional detection area is a cuboid, the relevant three-dimensional detection area of the relevant obstacle can be mapped onto the camera coordinate system to obtain the two-dimensional mapping result of the relevant obstacle. Then, the maximum bounding rectangle of the two-dimensional mapping result of the relevant obstacle is obtained as the relevant mapping two-dimensional area of the relevant obstacle.
[0042] After determining the relevant mapping two-dimensional regions for each relevant obstacle, the regional similarity between the relevant mapping two-dimensional regions and the relevant two-dimensional detection regions can be determined. Regional similarity indicates the similarity between the relevant three-dimensional detection regions and the relevant two-dimensional detection regions. Higher similarity between the two-dimensional and three-dimensional detection regions indicates a lower probability of missing point cloud data for related obstacles. Conversely, lower similarity between the two-dimensional and three-dimensional detection regions indicates a higher probability of missing point cloud data for related obstacles.
[0043] In this application, the intersection-union ratio (IUGR) of the relevant mapped 2D region of the relevant obstacle and the relevant 2D detection region can be calculated as the region similarity between the relevant mapped 2D region of the relevant obstacle and the relevant 2D detection region; the contour similarity (contour line similarity) between the contour of the relevant mapped 2D region of the relevant obstacle and the contour of the relevant 2D detection region can also be determined as the region similarity between the relevant mapped 2D region of the relevant obstacle and the relevant 2D detection region; the IUGR between the relevant mapped 2D region of the relevant obstacle and the relevant 2D detection region can also be determined, and the contour similarity between the contour of the relevant mapped 2D region of the relevant obstacle and the contour of the relevant 2D detection region can also be determined. Then, the IUGR and the contour similarity are weighted and summed (the weights of the IUGR and the contour similarity can be set according to the requirements, for example, both are 0.5), and the summation result is used as the region similarity between the relevant mapped 2D region of the relevant obstacle and the relevant 2D detection region.
[0044] After determining the regional similarity between the relevant mapped 2D region and the relevant 2D detection region for each relevant obstacle, for each relevant obstacle, if the regional similarity is less than the regional similarity threshold, the relevant obstacle is identified as a missed relevant obstacle. The similarity threshold can be determined based on the method of determining regional similarity. For example, if the regional similarity is the intersection-union ratio (IUU) of the relevant mapped 2D region and the relevant 2D detection region, the regional similarity threshold could be 0.4. Alternatively, if the regional similarity is the contour similarity between the contour of the relevant mapped 2D region and the contour of the relevant 2D detection region, the regional similarity threshold could be 0.6.
[0045] For each relevant obstacle, if the region similarity is not less than the region similarity threshold, it indicates that the amount of point cloud data for that relevant obstacle in the 3D point cloud data is large and the point cloud data for that relevant obstacle in the 3D point cloud data is relatively accurate. The relevant obstacle is determined as a valid relevant obstacle. Based directly on the point cloud data of the valid relevant obstacles in the 3D point cloud data, the 3D bounding box surrounding the valid relevant obstacle is determined as a valid 3D bounding box. The 3D point cloud data is directly labeled using the valid 3D bounding box. At this time, the valid 3D bounding box is relatively accurate, and accurate labeling of the 3D point cloud data is achieved.
[0046] For each relevant obstacle, if the region similarity is less than the region similarity threshold, it indicates that the amount of point cloud data for that obstacle in the 3D point cloud data is small and the point cloud data for that obstacle in the 3D point cloud data is inaccurate. Therefore, the relevant obstacle is identified as a missed obstacle. If the accuracy of determining the 3D bounding box surrounding the missed obstacle is directly based on the point cloud data of the missed obstacle in the 3D point cloud data, the accuracy of the 3D point cloud data annotation is poor. If the 3D bounding box is directly used to annotate the 3D point cloud data, the annotation accuracy of the 3D point cloud data is poor. Therefore, after identifying the relevant obstacle as a missed obstacle, the annotation is carried out according to the subsequent steps of this application.
[0047] S102. Insert multiple virtual 3D boxes within the relevant area.
[0048] Among them, the relevant area is determined from the target detection area based on the missed detection of relevant obstacles in the environmental image.
[0049] Optionally, the method for determining the relevant region may include: determining the three-dimensional position of the missed relevant obstacle in the world coordinate system based on the two-dimensional position of the missed relevant obstacle in the environmental image; then, constructing a relevant region centered on the three-dimensional position of the missed relevant obstacle in the world coordinate system; and then inserting multiple virtual three-dimensional bounding boxes within the relevant region. The relevant region is no smaller than the area enclosed by the smallest bounding box of the missed relevant obstacle.
[0050] Specifically, based on the aforementioned coordinate system mapping relationship, the two-dimensional location of the missed related obstacles in the environmental image can be mapped onto the world coordinate system to obtain the three-dimensional location of the missed related obstacles in the world coordinate system.
[0051] In some implementations, the method for determining the relevant region may further include: obtaining a relevant two-dimensional detection region, including missed related obstacles, from an environmental image; mapping the relevant two-dimensional detection region onto the target three-dimensional coordinate system where the vehicle is located to obtain a relevant mapped three-dimensional region; and determining the relevant region from the target detection region based on the relevant relative direction of the relevant mapped three-dimensional region relative to the vehicle's location. The description of the relevant two-dimensional detection region is as described above and will not be repeated here.
[0052] The target 3D coordinate system can refer to the coordinate system on which the 3D point cloud data is based. Typically, 3D point cloud data is based on the world coordinate system, therefore, the target 3D coordinate system can be the world coordinate system.
[0053] For each missed obstacle, the relevant two-dimensional detection area of the missed obstacle is mapped onto the target three-dimensional coordinate system to obtain a three-dimensional region, which serves as the relevant mapped three-dimensional region. Alternatively, the relevant two-dimensional detection area of the missed obstacle can be mapped onto the target three-dimensional coordinate system to obtain a three-dimensional region, and then a minimum cuboid bounding box including this three-dimensional region can be constructed as the relevant mapped three-dimensional region.
[0054] After determining the relevant mapping three-dimensional region of the missed obstacle, the direction of this region relative to the vehicle's location can be determined as the relevant relative direction. For example, if the relevant mapping three-dimensional region of the missed obstacle is located 45° to the right of the vehicle's front, the relevant relative direction is determined to be 45° to the right of the front.
[0055] After determining the relative direction of the relevant mapping 3D region of the missed obstacle relative to the vehicle's location, the region located within the target detection area and situated within that relative direction is identified as the relevant region. For example, if the target detection area is a circle centered on the vehicle, and the relevant mapping 3D region of the missed obstacle is located 45° to the right of the vehicle's front, the relative direction is determined to be 45° to the right. The region within the target detection area (with a radius equal to a 45° angle with the front of the vehicle) in this direction is then identified as the relevant region.
[0056] In some implementations, the size of the virtual 3D bounding box can be set based on requirements. In other implementations, the target 3D size corresponding to the obstacle category to which the missed detection-related obstacle belongs can be obtained from a real dataset; the real dataset includes the 3D size corresponding to at least one obstacle category; then, multiple virtual 3D bounding boxes, each with the target 3D size, are inserted into the relevant area. In this embodiment, the 3D size can be in the form of length × width × height.
[0057] Obstacle categories can be defined based on requirements; for example, obstacle categories can include cars, trucks, buses, pedestrians, trash cans, and road signs. Each obstacle category corresponds to a three-dimensional dimension, and different obstacle categories may have different three-dimensional dimensions.
[0058] It can acquire multiple obstacles belonging to the same obstacle category, construct a cuboid 3D bounding box for each obstacle, and then calculate the average of the length, width, and height of the cuboid 3D bounding boxes of the multiple obstacles to obtain the average length, width, and height as the length, width, and height in the 3D dimensions of the obstacle category.
[0059] Optionally, in some implementations, before S102, the method may include: obtaining the target 3D size corresponding to the obstacle category to which the missed detection related obstacle belongs from the real dataset; the real dataset includes the 3D size corresponding to at least one obstacle category; determining the target depth spacing based on the target 3D size; correspondingly, S102 may include: inserting multiple virtual 3D boxes in the relevant area with the target depth spacing being the interval between any two adjacent virtual 3D boxes.
[0060] As mentioned above, if the three-dimensional dimensions are in the form of length, width, and height, the target multiple of the length in the three-dimensional dimensions can be obtained as the target depth spacing. The target multiple can be, for example, 1 / 5 or 1 / 4.
[0061] After determining the target depth spacing, multiple virtual 3D boxes can be inserted in the relevant area, with the target depth spacing being the interval between any two adjacent virtual 3D boxes.
[0062] For example, the relevant region is a region constructed with the three-dimensional position of the missed relevant obstacle in the world coordinate system as the center. Multiple virtual three-dimensional boxes can be arrayed within the relevant region with the target depth spacing as the interval between any two adjacent virtual three-dimensional boxes.
[0063] For example, the relevant region is the area located in the relevant mapping 3D region relative to the vehicle's position in the relevant direction, and within the target detection region. Multiple virtual 3D bounding boxes can be inserted along the straight line of the relative direction within the relevant region, with the interval between any two adjacent virtual 3D bounding boxes being the target depth spacing. For instance, if the relevant region is 50m in the direction 45° to the right of the vehicle, and the target depth spacing is 2m, then 25 virtual 3D bounding boxes (with their centers on the straight line of the relevant region) can be inserted along the straight line of the relevant region, and these 25 virtual 3D bounding boxes are distributed along the straight line in the direction 45° to the right of the vehicle.
[0064] S103. Based on the correlation between the multiple virtual 3D bounding boxes and the missed detection related obstacles, select a target virtual 3D bounding box from the multiple virtual 3D bounding boxes.
[0065] The correlation is used to indicate the degree of fit between the virtual 3D bounding box and the missed relevant obstacle. The greater the correlation, the greater the fit between the virtual 3D bounding box and the missed relevant obstacle, and the greater the probability that the virtual 3D bounding box exactly surrounds the missed relevant obstacle. Conversely, the lower the correlation, the less the fit between the virtual 3D bounding box and the missed relevant obstacle, and the less the probability that the virtual 3D bounding box exactly surrounds the missed relevant obstacle.
[0066] In this embodiment, for each missed-detection related obstacle, the volume overlap between each virtual 3D bounding box corresponding to the missed-detection related obstacle and the minimum circumscribed cuboid 3D bounding box surrounding the missed-detection related obstacle can be determined as the correlation between each virtual 3D bounding box corresponding to the missed-detection related obstacle and the missed-detection related obstacle. The volume overlap is the ratio of the overlapping volume between the virtual 3D bounding box corresponding to the missed-detection related obstacle and the minimum circumscribed cuboid 3D bounding box of the missed-detection related obstacle to the total volume when the virtual 3D bounding box corresponding to the missed-detection related obstacle and the minimum circumscribed cuboid 3D bounding box of the missed-detection related obstacle overlap.
[0067] For each missed detection related obstacle, multiple virtual 3D bounding boxes are determined. The correlation between each of the multiple virtual 3D bounding boxes of the missed detection related obstacle and the missed detection related obstacle itself is determined. Then, based on the correlation between each of the multiple virtual 3D bounding boxes of the missed detection related obstacle and the missed detection related obstacle itself, a virtual 3D bounding box is determined as the target virtual 3D bounding box.
[0068] Optionally, the virtual 3D bounding box with the highest relevance can be selected as the target virtual 3D bounding box from multiple virtual 3D bounding boxes.
[0069] S104. Based on the target virtual 3D bounding box, annotate the point cloud data corresponding to the missed related obstacles in the 3D point cloud data.
[0070] For each missed-detection related obstacle, the corresponding virtual 3D bounding box is used to annotate the point cloud data of the 3D point cloud data. This achieves accurate annotation of missed-detection related obstacles where the point cloud data is missing.
[0071] In this embodiment, a physical 3D bounding box surrounding the missed obstacle is generated directly at the location of the target virtual 3D bounding box in the 3D point cloud data, so as to mark the missed obstacle in the missing point cloud data.
[0072] In this embodiment, instead of directly using the point cloud data of the missed obstacles to determine the 3D bounding boxes for annotation, multiple virtual 3D bounding boxes are inserted in the relevant area. Then, a target virtual 3D bounding box is selected from the multiple virtual 3D bounding boxes. The accuracy of the determined target virtual 3D bounding box is relatively high, thereby achieving accurate annotation of the point cloud data of the missed obstacles based on the target virtual 3D bounding box and improving the annotation effect of the 3D point cloud data.
[0073] Please see Figure 2 , Figure 2 A flowchart of a data annotation method according to another embodiment of this application is shown, for use in an electronic device, the method comprising:
[0074] S201. Based on the environmental images and 3D point cloud data collected by the vehicle at the target time point, identify the missed obstacles; insert multiple virtual 3D boxes in the relevant area.
[0075] The description of S201 is the same as that of S101-S102 above, and will not be repeated here.
[0076] S202. Obtain relevant two-dimensional detection boxes surrounding the missed-detection related obstacles from the environmental image; map each virtual three-dimensional box onto the camera coordinate system where the environmental image is located to obtain the corresponding mapped virtual box for each virtual three-dimensional box; determine the overlap rate between the mapped virtual box and the relevant two-dimensional detection boxes, as the correlation between the virtual three-dimensional box corresponding to the mapped virtual box and the missed-detection related obstacles.
[0077] For each missed-detection related obstacle, the smallest bounding rectangle including the missed-detection related obstacle is obtained in the environmental image, which is used as the relevant two-dimensional detection box of the missed-detection related obstacle. Then, based on the coordinate system mapping relationship in the previous embodiment, each virtual three-dimensional box corresponding to the missed-detection related obstacle is mapped to the camera coordinate system to obtain the mapped two-dimensional mapping result. The largest bounding rectangle including each two-dimensional mapping result is obtained (one virtual three-dimensional box of a missed-detection related obstacle corresponds to one two-dimensional mapping result, and one mapping result corresponds to one largest bounding rectangle), which is used as the mapped virtual box of each virtual three-dimensional box corresponding to the missed-detection related obstacle. For each virtual three-dimensional box of the missed-detection related obstacle, the overlap rate between the mapped virtual box corresponding to the virtual three-dimensional box and the relevant two-dimensional detection box of the missed-detection related obstacle is determined, which is used as the correlation between the virtual three-dimensional box and the missed-detection related obstacle.
[0078] The overlap rate between the mapped virtual box and the relevant 2D detection box can be the ratio of the overlapping area of the mapped virtual box and the relevant 2D detection box to the total area when the mapped virtual box and the relevant 2D detection box overlap.
[0079] S203. Select the initial target virtual 3D bounding box from multiple virtual 3D bounding boxes according to the relevance from high to low; determine a target virtual 3D bounding box from the initial target virtual 3D bounding box.
[0080] The target quantity can be set based on demand, for example, the target quantity is 10.
[0081] For each missed related obstacle, select a target number of virtual 3D bounding boxes from the multiple virtual 3D bounding boxes corresponding to the missed related obstacle, in descending order of relevance, as the initial target virtual 3D bounding boxes; then, select target virtual 3D bounding boxes from the initial target virtual 3D bounding boxes based on the distance between each initial target virtual 3D bounding box and the historical trajectory of the missed related obstacle.
[0082] Among them, the historical trajectory of the missed related obstacle refers to the trajectory of the missed related obstacle at a time point before the target time point, and the distance between the initial target virtual 3D bounding box and the historical trajectory of the missed related obstacle refers to the relative distance between the center of the initial target virtual 3D bounding box and the historical trajectory of the missed related obstacle.
[0083] In this embodiment, the initial target virtual 3D bounding box that is closest to the historical trajectory of the missed related obstacle can be selected from the initial target virtual 3D bounding box as the target virtual 3D bounding box.
[0084] First, multiple initial target virtual 3D bounding boxes are selected. Then, a target virtual 3D bounding box is selected from these initial boxes. This avoids the situation where the most relevant box is not the best fit when directly selecting the one with the highest relevance. Therefore, in this embodiment, multiple initial target virtual 3D bounding boxes are selected first, and then a target virtual 3D bounding box is determined by other methods (such as the distance between the initial target virtual 3D bounding box and the historical trajectory of the missed related obstacle). By combining different selection methods, the selected target virtual 3D bounding box is made more accurate.
[0085] S204. Based on the target virtual 3D bounding box, annotate the point cloud data corresponding to the missed related obstacles in the 3D point cloud data.
[0086] The description of S204 is the same as that of S104 above, and will not be repeated here.
[0087] In this embodiment, the data annotation process is as follows: Figure 3 As shown, the vehicle performs 2D detection at the target time point to obtain an environmental image, and simultaneously performs 3D detection to obtain 3D point cloud data. The relevant 3D detection area is determined from the 3D point cloud data, and the relevant 2D detection area is determined from the environmental image. Then, the relevant 3D detection area is projected onto the camera coordinate system to obtain the relevant mapped 2D area. It is then determined whether the regional similarity between the relevant mapped 2D area and the relevant 2D detection area is less than the regional similarity threshold.
[0088] If the size is not smaller than the specified size, label directly: label directly based on the rectangular box containing the relevant obstacles in the 3D point cloud data.
[0089] If the size is smaller than the target size, the relevant obstacle is identified as a missed obstacle. The target 3D size corresponding to the missed obstacle is determined, and multiple virtual 3D bounding boxes are inserted according to the target 3D size. Then, the virtual 3D bounding boxes are mapped onto the camera coordinate system to obtain the mapped virtual bounding box. Then, based on the overlap rate between the mapped virtual bounding box and the relevant 2D detection boxes in the environment image that include the missed obstacle, the target number of initial target virtual 3D bounding boxes is determined. Then, a target virtual 3D bounding box is determined from the target number of initial target virtual 3D bounding boxes to annotate the point cloud data of the missed obstacle in the 3D point cloud data.
[0090] In this embodiment, each virtual 3D bounding box is mapped onto the camera coordinate system where the environmental image is located, resulting in a corresponding mapped virtual bounding box. The overlap rate between the mapped virtual bounding box and the relevant 2D detection box is determined as the correlation between the virtual 3D bounding box corresponding to the mapped virtual bounding box and the missed related obstacles. The correlation can accurately indicate the probability that the virtual 3D bounding box is a 3D bounding box that includes the missed related obstacles, making the accuracy of the target virtual 3D bounding box determined based on the correlation high. This improves the accuracy of data annotation based on the target virtual 3D bounding box and enhances the annotation effect.
[0091] In addition, in this embodiment, 2D data (environmental images) and 3D data (3D point cloud data) are fused to determine the target virtual 3D bounding box. By combining data from different dimensions, the determined target virtual 3D bounding box is more accurate, avoiding the low accuracy of determining the 3D bounding box using only 3D data, and improving the data annotation effect.
[0092] See appendix Figure 4 , Figure 4 This diagram illustrates a structural block diagram of a data annotation apparatus according to an embodiment of this application. The apparatus 1200 includes:
[0093] The determination module 1210 is used to determine missed related obstacles based on the environmental images and three-dimensional point cloud data collected by the vehicle at the target time point; the environmental images include at least one related obstacle located within the target detection area; the missed related obstacles refer to related obstacles in the environmental images and in the three-dimensional point cloud data that have missing point cloud data;
[0094] The insertion module 1220 is used to insert multiple virtual 3D bounding boxes within a relevant region; the relevant region is determined from the target detection region based on the missed detection of relevant obstacles in the environmental image.
[0095] The selection module 1230 is used to select a target virtual 3D bounding box from multiple virtual 3D bounding boxes based on the correlation between the multiple virtual 3D bounding boxes and the missed detection related obstacles.
[0096] The annotation module 1240 is used to annotate the point cloud data corresponding to the missed obstacles in the 3D point cloud data based on the target virtual 3D bounding box.
[0097] Optionally, the insertion module 1220 is further configured to obtain relevant two-dimensional detection regions, including missed detection related obstacles, from the environmental image; map the relevant two-dimensional detection regions onto the target three-dimensional coordinate system where the vehicle is located to obtain relevant mapped three-dimensional regions; and determine the relevant regions from the target detection regions based on the relevant relative direction of the relevant mapped three-dimensional regions relative to the vehicle's location.
[0098] Optionally, the determining module 1210 is further configured to: obtain the relevant two-dimensional detection region of each relevant obstacle from the environmental image; obtain the relevant three-dimensional detection region of each relevant obstacle from the three-dimensional point cloud data; map the relevant three-dimensional detection regions of each relevant obstacle onto the camera coordinate system where the environmental image is located to obtain the relevant mapped two-dimensional region corresponding to each relevant obstacle; determine the regional similarity between the relevant mapped two-dimensional region of each relevant obstacle and the relevant two-dimensional detection region; and identify relevant obstacles with a regional similarity less than the regional similarity threshold as missed relevant obstacles.
[0099] Optionally, the selection module 1230 is further configured to obtain relevant two-dimensional detection boxes surrounding the missed detection related obstacles from the environmental image; map each virtual three-dimensional box onto the camera coordinate system where the environmental image is located to obtain the corresponding mapped virtual box for each virtual three-dimensional box; and determine the overlap rate between the mapped virtual box and the relevant two-dimensional detection box as the correlation between the virtual three-dimensional box corresponding to the mapped virtual box and the missed detection related obstacles.
[0100] Optionally, the insertion module 1220 is further configured to obtain the target 3D dimensions corresponding to the obstacle category to which the missed detection related obstacle belongs from the real dataset; the real dataset includes the 3D dimensions corresponding to at least one obstacle category; determine the target depth spacing based on the target 3D dimensions; and insert multiple virtual 3D boxes in the relevant region with the target depth spacing as the interval between any two adjacent virtual 3D boxes.
[0101] Optionally, the insertion module 1220 is further configured to obtain the target 3D dimensions corresponding to the obstacle category to which the missed detection-related obstacle belongs from the real dataset; the real dataset includes the 3D dimensions corresponding to at least one obstacle category; and insert multiple virtual 3D bounding boxes, each with the target 3D dimensions, into the relevant region.
[0102] Optionally, the selection module 1230 is also used to select an initial number of target virtual 3D bounding boxes from multiple virtual 3D bounding boxes according to their relevance from high to low; and to determine a target virtual 3D bounding box from the initial number of target virtual 3D bounding boxes.
[0103] Please see Figure 5 This diagram illustrates a structural block diagram of an electronic device 900 provided in an embodiment of this application. The electronic device 900 may be a vehicle or a module within a vehicle used to implement the data tagging method of the embodiments of this application. The electronic device 900 in this application may include one or more of the following components: a processor 910, a memory 920, and one or more application programs, wherein the one or more application programs may be stored in the memory 920 and configured to be executed by one or more processors 910, and the one or more programs are configured to perform the method as described in the foregoing method embodiments.
[0104] The processor 910 may include one or more processing cores. The processor 910 connects to various parts of the electronic device 900 using various interfaces and lines, and performs various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 920, and by calling data stored in the memory 920. Optionally, the processor 910 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 910 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 910 and may be implemented separately through a communication chip.
[0105] The memory 920 may include random access memory (RAM) or read-only memory (ROM). The memory 920 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 900 during use (such as phonebook data, audio and video data, chat log data, etc.).
[0106] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.
[0107] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0108] On the other hand, this application also provides a computer-readable storage medium storing program code that can be called by a processor to execute the methods described in the above method embodiments.
[0109] Computer-readable storage media can be electronic storage devices such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or a cluster of ROMs. Optionally, computer-readable storage media include non-transitory computer-readable storage media. The computer-readable storage medium has storage space for program code that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.
[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data annotation method, characterized in that, The method includes: Based on the environmental images and 3D point cloud data collected by the vehicle at the target time point, missed related obstacles are identified; the environmental images include at least one related obstacle located within the target detection area; the missed related obstacles refer to related obstacles in the environmental images and in the 3D point cloud data that have missing point cloud data. Multiple virtual 3D bounding boxes are inserted within the relevant area; the relevant area is determined from the target detection area based on the missed related obstacles in the environmental image; Based on the correlation between the plurality of virtual 3D bounding boxes and the missed-detection related obstacles, a target virtual 3D bounding box is selected from the plurality of virtual 3D bounding boxes; wherein, the correlation between the virtual 3D bounding box and the missed-detection related obstacle is the overlap rate between the mapped virtual bounding box corresponding to the virtual 3D bounding box and the related 2D detection box; the mapped virtual bounding box corresponding to the virtual 3D bounding box is obtained by mapping the virtual 3D bounding box to the camera coordinate system where the environmental image is located; the related 2D detection box is the 2D detection box in the environmental image that surrounds the missed-detection related obstacle; the point cloud data corresponding to the missed-detection related obstacle in the 3D point cloud data is labeled according to the target virtual 3D bounding box.
2. The method according to claim 1, characterized in that, Before inserting multiple virtual 3D boxes within the relevant area, the method further includes: Obtain relevant two-dimensional detection areas, including the missed detection related obstacles, from the environmental image; The relevant two-dimensional detection area is mapped to the target three-dimensional coordinate system where the vehicle is located to obtain the relevant mapped three-dimensional area; Based on the relative orientation of the relevant mapped 3D region with respect to the location of the vehicle, a relevant region is determined from the target detection region.
3. The method according to claim 1, characterized in that, The process of identifying missed obstacles based on environmental images and 3D point cloud data collected by the vehicle at the target time point includes: Obtain the relevant two-dimensional detection area for each of the relevant obstacles from the environmental image; Obtain the relevant three-dimensional detection area for each of the relevant obstacles from the three-dimensional point cloud data; The relevant three-dimensional detection areas of each of the relevant obstacles are mapped under the camera coordinate system where the environmental image is located to obtain the relevant mapped two-dimensional areas corresponding to each of the relevant obstacles. Determine the regional similarity between the relevant two-dimensional mapping regions of each of the relevant obstacles and the relevant two-dimensional detection regions; Obstacles with a region similarity less than the region similarity threshold are identified as missed obstacles.
4. The method according to claim 1, characterized in that, Before inserting multiple virtual 3D boxes within the relevant area, the method further includes: Obtain the target 3D dimensions corresponding to the obstacle category to which the missed detection-related obstacle belongs from the real dataset; the real dataset includes the 3D dimensions corresponding to at least one obstacle category; Determine the target depth spacing based on the target's three-dimensional dimensions; The insertion of multiple virtual 3D boxes within the relevant area includes: Multiple virtual 3D boxes are inserted in the relevant area, with the target depth spacing being the interval between any two adjacent virtual 3D boxes.
5. The method according to any one of claims 1-4, characterized in that, The insertion of multiple virtual 3D boxes within the relevant area includes: Obtain the target 3D dimensions corresponding to the obstacle category to which the missed detection-related obstacle belongs from the real dataset; the real dataset includes the 3D dimensions corresponding to at least one obstacle category; Multiple virtual 3D frames, each with the target 3D size, are inserted within the relevant area.
6. The method according to claim 1, characterized in that, The step of selecting a target virtual 3D bounding box from the plurality of virtual 3D bounding boxes based on their correlation with the missed detection related obstacles includes: An initial number of target virtual 3D bounding boxes are selected from the plurality of virtual 3D bounding boxes according to their relevance from high to low. A target virtual 3D frame is determined from the initial target virtual 3D frame of the target number.
7. A data annotation device, characterized in that, The device includes: The determination module is used to determine missed related obstacles based on environmental images and three-dimensional point cloud data collected by the vehicle at a target time point; the environmental images include at least one related obstacle located within the target detection area; the missed related obstacles refer to related obstacles in the environmental images and in the three-dimensional point cloud data that have missing point cloud data; An insertion module is used to insert multiple virtual 3D bounding boxes within a relevant region; the relevant region is determined from the target detection region based on the missed related obstacles in the environmental image; The selection module is used to select a target virtual 3D bounding box from the plurality of virtual 3D bounding boxes based on the correlation between the virtual 3D bounding boxes and the missed detection related obstacles; wherein, the correlation between the virtual 3D bounding box and the missed detection related obstacle is the overlap rate between the mapped virtual bounding box corresponding to the virtual 3D bounding box and the related 2D detection box; the mapped virtual bounding box corresponding to the virtual 3D bounding box is obtained by mapping the virtual 3D bounding box to the camera coordinate system where the environmental image is located; the related 2D detection box is the 2D detection box in the environmental image that surrounds the missed detection related obstacle; The annotation module is used to annotate the point cloud data corresponding to the missed detection related obstacles in the three-dimensional point cloud data according to the target virtual three-dimensional bounding box.
8. An electronic device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores processor-executable program code, which, when executed by the processor, causes the processor to perform the method according to any one of claims 1-6.
