A method, apparatus and electronic device for detecting 3D object annotations

Abstract

This invention provides a method, apparatus, and electronic device for detecting 3D object annotations, relating to the field of autonomous driving technology. The method includes: acquiring a 3D bounding box of a target object in 3D point cloud data; projecting the 3D bounding box onto a preset projection surface; determining whether occlusion exists within the 3D bounding box, wherein the preset projection surface includes a preset sphere, the origin of which is the origin of the sensor coordinate system corresponding to the 3D point cloud data; if occlusion exists, obtaining and determining the confidence level of the 3D bounding box based on the visibility of the target object, according to the occlusion status of the 3D bounding box. This invention enables the detection of the reliability of 3D object annotations.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method, apparatus, and electronic device for detecting three-dimensional object annotations. Background Technology

[0002] In the field of autonomous driving, LiDAR is often used to capture 3D point clouds of target objects such as vehicles, pedestrians, and road obstacles, providing crucial geometric information about complex environments for perception tasks in autonomous driving processes, such as target detection. Based on the 3D point clouds of target objects, corresponding 3D bounding boxes are constructed, enabling the point clouds to be structured and providing accurate perception information for subsequent driving control and decision-making.

[0003] In pre-deployment testing and verification tasks for autonomous driving, after 3D annotation of the target object based on the captured target object's corresponding 3D target bounding box, manual correction is often required before it can be used as ground truth for further annotation. Because targets may occlude each other, the captured 3D point cloud may not be complete, and the manually corrected 3D target bounding box may not perfectly reflect the actual situation of the target. Therefore, current technology cannot yet determine whether the target annotation is truly reliable. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, and electronic device for detecting 3D object annotations, so as to detect whether the 3D object annotations are reliable. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide a method for detecting three-dimensional object annotations, the method comprising:

[0006] Obtain the 3D bounding box of the target object in the 3D point cloud data;

[0007] The three-dimensional annotation box is projected onto a preset projection surface, and it is determined whether there is occlusion in the three-dimensional annotation box. The preset projection surface includes a preset sphere, and the origin of the preset sphere is the origin of the coordinate system where the three-dimensional point cloud data is located.

[0008] If present, based on the occlusion of the 3D bounding box, the confidence level of the 3D bounding box is obtained and determined based on the visibility of the target object.

[0009] In one embodiment of the present invention, the visibility of the target object includes visibility in the length and width directions. The step of obtaining and determining the confidence level of the 3D bounding box based on the occlusion status of the target object includes:

[0010] Based on the occlusion of the 3D bounding box, the first confidence level of the 3D bounding box is obtained and determined based on the visibility of the target object in the length direction;

[0011] Based on the occlusion of the 3D bounding box, the second confidence level of the 3D bounding box is obtained and determined based on the visibility of the target object in the width direction;

[0012] The confidence level of the three-dimensional bounding box is determined based at least on the first confidence level and the second confidence level.

[0013] In one embodiment of the present invention, the step of obtaining and determining a first confidence level of the 3D annotation box based on the occlusion status of the 3D annotation box and the visibility of the target object along its length direction includes:

[0014] In the three-dimensional annotation frame, a first annotation frame area is selected along the width direction on one side near the origin of the preset projection surface, and the first annotation frame area is projected onto the preset projection surface;

[0015] Obtain the occlusion status of the first annotation box area, and calculate the visibility of the first annotation box area based on the occlusion status;

[0016] Based on the visibility of the first annotation box region, the first confidence level of the three-dimensional annotation box of the target object in the length direction is determined.

[0017] In one embodiment of the present invention, determining the first confidence level of the three-dimensional bounding box of the target object in the length direction based on the visibility of the first bounding box region includes:

[0018] If there is no occlusion in the first annotation box area, the first confidence level of the three-dimensional annotation box of the target object in the length direction is determined to be completely reliable.

[0019] In one embodiment of the present invention, the visibility of the target object further includes visibility in the height direction, and the confidence of the 3D bounding box further includes a third confidence corresponding to the visibility in the height direction. Then, determining the confidence of the 3D object bounding box based at least on the first confidence and the second confidence includes:

[0020] The confidence level of the 3D bounding box is determined based on the first confidence level, the second confidence level, and the third confidence level.

[0021] In one embodiment of the present invention, determining the confidence level of the 3D bounding box based on the first confidence level, the second confidence level, and the third confidence level includes:

[0022] When the first confidence level, second confidence level, and third confidence level of the 3D annotation box of the target object are all completely reliable, the 3D annotation box is determined to be completely reliable.

[0023] In one embodiment of the present invention, determining the first confidence level of the three-dimensional bounding box of the target object in the length direction based on the visibility of the first bounding box region includes:

[0024] If the visibility of the first annotation box area is not less than a preset visibility threshold, the three-dimensional annotation box of the target object is determined to be credible in the length direction; if the visibility of the first annotation box area is less than the preset visibility threshold, the three-dimensional annotation box of the target object is determined to be unreliable in the length direction.

[0025] Secondly, embodiments of the present invention provide a detection device for three-dimensional object annotation, the device comprising:

[0026] The annotation box acquisition module is used to acquire the 3D annotation box of the target object to be detected in the 3D point cloud data;

[0027] An occlusion determination module is used to project the three-dimensional annotation box onto a preset projection surface and determine whether there is occlusion in the three-dimensional annotation box. The preset projection surface includes a preset sphere, and the origin of the preset sphere is the origin of the sensor coordinate system corresponding to the three-dimensional point cloud data.

[0028] The confidence determination module is used to, if present, obtain and determine the confidence of the 3D annotation box based on the occlusion status of the target object and the visibility of the 3D annotation box.

[0029] In one embodiment of the present invention, the visibility of the target object includes visibility in the length direction and the width direction, and the confidence determination module includes:

[0030] The first confidence level determination submodule is used to obtain the first confidence level of the three-dimensional annotation box based on the occlusion of the three-dimensional annotation box and the visibility of the target object in the length direction.

[0031] The second confidence determination submodule is used to obtain the second confidence of the three-dimensional annotation box based on the occlusion of the three-dimensional annotation box and determine the second confidence of the three-dimensional annotation box based on the visibility of the target object in the width direction.

[0032] The confidence level determination submodule is used to determine the confidence level of the 3D bounding box based at least on the first confidence level and the second confidence level.

[0033] In one embodiment of the present invention, the first confidence level determination submodule includes:

[0034] A region projection unit is used to select a first annotation box region along the width direction of the three-dimensional annotation box on one side near the origin of the preset projection surface, and project the first annotation box region onto the preset projection surface.

[0035] The visibility calculation unit is used to obtain the occlusion status of the first annotation box area and calculate the visibility of the first annotation box area based on the occlusion status of the first annotation box area;

[0036] The confidence determination unit is used to determine the first confidence level of the three-dimensional annotation box of the target object in the length direction based on the visibility of the first annotation box area.

[0037] In one embodiment of the present invention, the confidence level determination unit is specifically used for:

[0038] If there is no occlusion in the first annotation box area, the first confidence level of the three-dimensional annotation box of the target object in the length direction is determined to be completely reliable.

[0039] In one embodiment of the present invention, the visibility of the target object further includes visibility in the height direction, and the confidence of the three-dimensional bounding box further includes a third confidence corresponding to the visibility in the height direction. Therefore, the confidence determination submodule is specifically used for:

[0040] The confidence level of the 3D bounding box is determined based on the first confidence level, the second confidence level, and the third confidence level.

[0041] In one embodiment of the present invention, the confidence level determination submodule is specifically used for:

[0042] When the first confidence level, the second confidence level, and the third confidence level of the 3D annotation box of the target object are all completely reliable, the 3D annotation box is determined to be completely reliable.

[0043] In one embodiment of the present invention, the confidence level determination unit is specifically used for:

[0044] If the visibility of the first annotation box area is not less than a preset visibility threshold, the three-dimensional annotation box of the target object is determined to be credible in the length direction; if the visibility of the first annotation box area is less than the preset visibility threshold, the three-dimensional annotation box of the target object is determined to be unreliable in the length direction.

[0045] Thirdly, embodiments of the present invention also provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0046] Memory, used to store computer programs;

[0047] The processor, when executing a program stored in memory, implements the steps of the three-dimensional object annotation detection method described above.

[0048] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the detection method steps for three-dimensional object annotation described above.

[0049] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described above.

[0050] Beneficial effects of the embodiments of the present invention:

[0051] The method for detecting 3D object annotations provided in this invention first obtains the 3D annotation box of the target object in 3D point cloud data, and projects the 3D annotation box onto a preset projection surface, which includes a preset sphere. The origin of the preset sphere is the origin of the sensor coordinate system corresponding to the 3D point cloud data. Then, it is determined whether there is occlusion in the 3D annotation box. If so, the confidence level of the 3D annotation box is obtained based on the occlusion and determined based on the visibility of the target object. This invention calculates the visibility of the target object and then determines the confidence level of the 3D annotation box by considering the occlusion, thus quantifying the confidence level of the 3D annotation box. This allows the confidence level of the 3D annotation of the target object to be accurately detected for further judgment, improving the accuracy of 3D annotation, providing higher reliability and applicability for the application of 3D annotation, and providing a criterion for the accuracy of manual annotation.

[0052] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0054] Figure 1aThis is a flowchart illustrating the first method for detecting three-dimensional object annotations provided in an embodiment of the present invention.

[0055] Figure 1b This is an example diagram illustrating the first method of projecting a three-dimensional annotation box onto a preset projection surface, as provided in an embodiment of the present invention.

[0056] Figure 1c This invention provides an example of a projection view that projects a three-dimensional annotation box onto a preset projection surface.

[0057] Figure 2 This is one possible implementation of step S13 provided in the embodiments of the present invention;

[0058] Figure 3 This is one possible implementation of step S21 provided in the embodiments of the present invention;

[0059] Figure 4a An example diagram of a three-dimensional annotation frame provided in an embodiment of the present invention;

[0060] Figure 4b This is an example diagram of the first method of selecting and projecting a first annotation box area according to an embodiment of the present invention;

[0061] Figure 5a This is one possible implementation of step S32 provided in the embodiments of the present invention;

[0062] Figure 5b This is an example diagram illustrating a second method of projecting a three-dimensional annotation frame onto a preset projection surface, as provided in an embodiment of the present invention.

[0063] Figure 5c This is an example diagram illustrating the second method of selecting and projecting the first annotation box area according to an embodiment of the present invention;

[0064] Figure 5d A specific example diagram illustrating the calculation of the visibility of a 3D annotation box in the length direction is provided in an embodiment of the present invention;

[0065] Figure 5e This invention provides a projection view for calculating the visibility of a 3D annotation box in the length direction, as provided in an embodiment of the invention.

[0066] Figure 6 This is a schematic diagram of the structure of a three-dimensional object annotation detection device provided in an embodiment of the present invention;

[0067] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0069] Because targets may occlude each other, the captured 3D point cloud of the target may not be complete, and the 3D target annotation box after manual correction may not completely match the actual situation of the target. In order to detect the credibility of the annotation of 3D target objects, this invention provides a method, device and electronic device for detecting 3D object annotation. The following is a detailed description of the 3D object annotation detection method through specific embodiments.

[0070] The method of this invention is applied to a smart terminal and can be implemented through a smart terminal. In actual use, the smart terminal can be an in-vehicle intelligent system, an in-vehicle data processing center, etc.

[0071] See Figure 1a This invention provides a flowchart illustrating a first method for detecting 3D object annotations, comprising:

[0072] Step S11: Obtain the 3D bounding box of the target object to be detected in the 3D point cloud data.

[0073] The target object to be detected is the object that needs to be labeled and detected. In an example, in an autonomous driving application scenario, the target object could be a vehicle, pedestrian, or road obstacle. The acquired 3D point cloud data of the target object represents its 3D shape. Specifically, the 3D point cloud can be captured using LiDAR. In practical applications, when capturing the 3D point cloud of a target object, the target object may be occluded by other objects. In this case, the acquired 3D point cloud data of the target object may contain a portion of the occluded point cloud.

[0074] After acquiring the 3D point cloud of the target object, the annotator will determine the 3D bounding box of the target object based on the acquired 3D point cloud data. This bounding box represents information such as the shape, size, and position of the target object. Since there may be occluded parts of the original acquired 3D point cloud data, the 3D bounding box may be inaccurate. Therefore, it is necessary to detect the occlusion of the 3D point cloud of the target object to determine the reliability of the 3D bounding box.

[0075] Step S12: Project the three-dimensional annotation frame onto a preset projection surface and determine whether the three-dimensional annotation frame is occluded.

[0076] Projecting the 3D annotation box onto a preset projection plane can be understood as projecting the entire area (e.g., the area defined by the line connecting the boundary of the 3D annotation box and the origin of the sensor coordinate system) onto a preset projection plane. Figure 1b In this process, an occluding object exists between the target object and the sensor coordinate origin, both projected onto a preset projection surface. Specifically, the preset projection surface may include a preset sphere, which is a sphere with its origin at the origin of the sensor coordinate system corresponding to the aforementioned 3D point cloud data and its radius as a preset radius. The value of the preset radius can be pre-set or set based on the line connecting the boundary of the aforementioned 3D annotation frame and the origin of the sensor coordinate system. The sensor coordinate system corresponding to the 3D point cloud data is the coordinate system of the sensor used when acquiring the 3D point cloud data.

[0077] In one example, the preset projection surface may also include a preset horizontal plane. The preset horizontal plane also takes the origin of the sensor coordinate system as its origin. Specifically, when the sensor coordinate system is an XYZ coordinate system, the preset horizontal plane coincides with the YZ plane of the sensor coordinate system, that is, the coordinate system of the preset horizontal plane coincides with the YZ coordinate system in the sensor coordinate system.

[0078] In this embodiment of the invention, a preset projection surface can be selected according to the type of sensor. For example, when the sensor is a lidar, based on the working mode of laser emission and reception, using a preset sphere may be a more preferred approach. Therefore, a preset sphere is selected as the preset projection surface in this case. The following explanation uses a preset sphere as the preset projection surface.

[0079] The determination of whether the 3D annotation bounding box of a target object is occluded is based on the projection image of the 3D annotation bounding box onto a preset sphere. Specifically, to determine whether the 3D annotation bounding box of the target object is occluded by the 3D annotation bounding boxes of other objects, it can be done by checking whether there is an occluding object between the projection image of the 3D annotation bounding box and the origin of the preset sphere, and whether the occluding object obstructs the projection image of the 3D annotation bounding box. Based on this, it can be determined whether there are occluded point clouds in the 3D point cloud data of the target object. That is, if there is an occluding object between the projection image of the 3D annotation bounding box and the origin of the preset sphere, and the occluding object obstructs the projection image of the 3D annotation bounding box, then it indicates that there are occluded point clouds in the 3D point cloud data of the target object.

[0080] Step S13: If it exists, based on the occlusion of the 3D annotation box, obtain and determine the confidence level of the 3D annotation box based on the visibility of the target object.

[0081] If the 3D bounding box of a target object is determined to be occluded, the visibility of the target object is further obtained. Specifically, the visibility of the target object is based on the unoccluded portion of the 3D bounding box. In one example, the visibility of the 3D bounding box can be calculated based on whether there is an occluded portion and the proportion of the unoccluded portion to the 3D bounding box.

[0082] After obtaining the visibility of the target object, the confidence level of the 3D bounding box of the target object is determined based on this. The confidence level can be used as a reference indicator to judge the credibility of the 3D bounding box in the application.

[0083] like Figure 1b (Top view) and Figure 1c As shown, after projecting the 3D bounding box of the target object onto a preset sphere and unfolding it, it can be seen that... Figure 1b The projections between points A and B, i.e., the solid lines A and B, correspond to each other. Figure 1c AB in Figure 1b The projection between points C and B, i.e., the solid line B and the dashed line, correspond to each other. Figure 1c In the projection, the entire 3D bounding box should be the area covered by (AB+BC). The portion BC is invisible because it is occluded by an object. Figure 1c In the upper bound, the BC portion will coincide with the projection of the occluded object. Therefore, the visibility of the entire 3D bounding box = the confidence level of the 3D bounding box = S = AB / (AB+BC).

[0084] In one example, a preset visibility threshold for the 3D annotation box can be set in advance. When the visibility S of the 3D annotation box calculated above is not less than the threshold, the 3D annotation box is considered to be reliable.

[0085] As can be seen from the above, the three-dimensional object annotation detection method provided by the embodiments of the present invention first obtains the three-dimensional annotation box of the target object to be detected in the three-dimensional point cloud data, and projects the three-dimensional annotation box onto a preset projection surface. The origin of the preset projection surface is the origin of the coordinate system of the three-dimensional point cloud data. Then, it is determined whether there is occlusion in the three-dimensional annotation box. If there is, the confidence of the three-dimensional annotation box is obtained based on the occlusion of the three-dimensional annotation box and determined based on the visibility of the target object. The present invention calculates the visibility of the target object by the occlusion of the three-dimensional annotation box and then determines the confidence of the three-dimensional annotation box, quantifying the credibility of the three-dimensional annotation box. This allows the credibility of the three-dimensional annotation of the target object to be accurately detected and further judged, providing a criterion for the accuracy of manual annotation.

[0086] In one possible implementation, such as Figure 2As shown, when the target object is occluded, and it is necessary to further confirm the specific direction of the occlusion, the visibility of the target object can be further subdivided into visibility in the length and width directions. Step S13, based on the occlusion of the 3D bounding box, obtains and determines the confidence level of the 3D bounding box based on the visibility of the target object, including:

[0087] Step S21: Based on the occlusion of the three-dimensional annotation box, obtain and determine the first confidence level of the three-dimensional annotation box based on the visibility of the target object in the length direction;

[0088] Step S22: Based on the occlusion of the 3D bounding box, obtain and determine the second confidence level of the 3D bounding box based on the visibility of the target object in the width direction;

[0089] Step S23: Determine the confidence level of the three-dimensional bounding box based at least on the first confidence level and the second confidence level.

[0090] In one possible implementation, such as Figure 3 As shown, step S21 above, based on the occlusion of the 3D bounding box, obtains and determines the first confidence level of the 3D bounding box based on the visibility of the target object along its length direction, including:

[0091] Step S31: Select a first annotation box area along the width direction of the three-dimensional annotation box on one side near the origin of the preset projection surface, and project the first annotation box area onto the preset projection surface;

[0092] Step S32: Obtain the occlusion status of the first annotation box area, and calculate the visibility of the first annotation box area based on the occlusion status of the first annotation box area;

[0093] Step S33: Determine the first confidence level of the three-dimensional annotation box of the target object in the length direction based on the visibility of the first annotation box area.

[0094] The aforementioned first annotation box region is a region of a specified size along the width direction of the 3D annotation box, on the side closest to the origin of the preset projection plane. The specific size can be determined according to actual needs. The determined first annotation box region is projected onto the preset projection plane to obtain the occlusion status of the first annotation box region, and the visibility of the first annotation box is obtained based on the occlusion status, thereby determining the first confidence level of the 3D annotation box of the target object in the length direction.

[0095] In one embodiment of the present invention, such as Figure 4aThe image shows the 3D bounding box of the target object. Specifically, the 3D bounding box can be automatically generated by the system or manually defined. Assume c represents the length direction, b represents the width direction, and a represents the height direction. See [link to documentation]. Figure 4b The above-mentioned selection of a first annotation frame area along the width direction of the three-dimensional annotation frame, on one side near the origin of the preset projection plane, includes:

[0096] Along the width direction of the three-dimensional annotation frame, a preset width is cut off on the side closest to the origin of the preset projection plane, while keeping the dimensions in the remaining two directions unchanged (i.e., keeping the dimensions of the three-dimensional annotation frame unchanged in the height and length directions), thus obtaining the first annotation frame area.

[0097] See Figure 4b The preset width can be assigned a very small value, such as no more than 1% of the length direction, to create a significant difference from the length direction to be evaluated. Because the width is small enough, when the first annotation box area is projected onto the preset projection plane, the occlusion of the 3D annotation box is mainly reflected by whether there is occlusion in the length direction. If the first annotation box area is still occluded by other objects in the projected view, it indicates that there is occlusion in the length direction of the 3D annotation box, and the 3D annotation box is unreliable in the length direction.

[0098] Similarly, when calculating the confidence level of the 3D annotation box in the width direction, a preset length is cut off along the length direction of the 3D annotation box on the side closest to the origin of the preset projection plane, while keeping the dimensions in the remaining two directions unchanged (i.e. keeping the dimensions of the 3D annotation box unchanged in the width and height directions), to obtain the second annotation box area.

[0099] The preset length is assigned a very small value, for example, no more than 1% of the width direction, to create a significant difference from the width direction to be evaluated. Because the length direction is small enough, when the second annotation box area is projected onto the preset projection plane, the occlusion of the 3D annotation box is mainly reflected by whether there is occlusion in the width direction. If the second annotation box area is still occluded by other objects in the projected view, it indicates that there is occlusion in the width direction of the 3D annotation box, and the 3D annotation box is unreliable in the width direction.

[0100] In one possible implementation, such as Figure 5a As shown, step S32 above obtains the occlusion status of the first annotation box region and calculates the visibility of the first annotation box region based on the occlusion status, including:

[0101] Step S51: Project the first annotation box area onto a preset projection surface to obtain the first visible projection area of ​​the first annotation box area and the first annotation box projection area of ​​the first annotation box area.

[0102] As mentioned above, the origin of the preset projection surface coincides with the origin of the sensor coordinate system corresponding to the three-dimensional point cloud data. The first visible projection area is the projection area of ​​the unobstructed area of ​​the first annotation box area on the preset projection surface, and the first annotation box projection area is the projection area of ​​the first annotation box area on the preset projection surface.

[0103] In one example, the first annotation frame area of ​​the target object and the three-dimensional annotation frame of the occluded object can be projected together onto a preset projection surface. When the projection of the first annotation frame area overlaps with the projection of the three-dimensional annotation frame of the occluded object, it is determined that the overlapping part of the projection of the three-dimensional annotation frame of the target object is occluded, and the unoccluded part of the projection of the three-dimensional annotation frame is taken as the first visible projection area.

[0104] Step S52: Calculate the ratio of the first visible projection area to the first annotation box projection area to obtain the visibility of the first annotation box area.

[0105] After obtaining the first visible projection area and the first annotation box projection area, the ratio of the first visible projection area to the first annotation box projection area is calculated. That is, the ratio of the unoccluded point cloud area in the first annotation box area to the area corresponding to the 3D annotation box is calculated. This ratio can represent the visibility of the first annotation box area, and can also be used to represent the visibility of the target object.

[0106] In other words, if the first annotation box area does not obstruct other objects in the projected view (that is, the first visible projection area is exactly equal to the first annotation box projection area), it means that the 3D annotation box is completely visible in the length direction.

[0107] Scenario 1: In one example, such as Figure 4b As shown, the sensor is mounted on the vehicle, and the preset projection surface is a preset sphere, with the length direction corresponding to the X-axis direction. There is an occluding object between the target object's 3D bounding box and the sensor. When calculating the visibility of the target object's 3D bounding box in the length direction, a very small value is assigned to the width direction to obtain the first bounding box area. This first bounding box area is then projected onto the preset sphere. It can be understood that... Figure 4b In the projection view, since the length direction of the target object is fully visible, the first bounding box region will appear to have no overlap with the occluded object. At this point, the first confidence level is 1, meaning the 3D bounding box is completely reliable in the length direction.

[0108] Scenario 2: For example Figure 5b and Figure 5c As shown, when the position of the occluded object is... Figure 4bThe occlusion of the 3D annotation box varies depending on the context. If a very small value is assigned to the width direction, the resulting first annotation box area will completely overlap with the occluded object in the projection view. In this case, the 3D annotation box will be completely invisible in the length direction, and the first confidence level in the length direction will be 0, indicating that it is completely unreliable.

[0109] Scenario 3: Conversely, if the area of ​​the first annotation box overlaps with the occluding object in the projected view, it indicates that the occluding object occludes the target object to a certain extent in the X-axis direction, i.e., the length direction.

[0110] The visibility of the first annotation box area = the first confidence level in the length direction = the first visible projection area / the first annotation box projection area.

[0111] like Figure 5d and Figure 5e As shown, the target object is car A, and the object occluding car A is car B. Other areas are projections of areas other than the projections of cars A and B in the projection view. Within the 3D annotation frame of car A, the area between dashed lines B and C is area a, and the area between dashed lines A and B is area b. Areas a, b, and c together form the complete 3D annotation frame of car A. Areas a and b together constitute the first batch of annotation frame areas for car A. At this time, the overall projection of car B coincides with area a, which is the occluded projection within the first batch of annotation frame areas for car A. Area b is the first visible projection area of ​​car A. Area a + area b = the first batch of annotation frame projection area for car A. The visibility of car A in the length direction (first confidence level) = area b / (area a + area b). After calculating the first confidence level, it is compared with a preset visibility threshold. If the first confidence level is not less than the preset visibility threshold, it indicates that the occlusion of the target object in the length direction is not severe, and the information of its 3D annotation frame in the length direction is reliable.

[0112] In other words, the first level of confidence can be divided into three categories based on the occlusion of the target object: completely trustworthy, trustworthy, and untrustworthy.

[0113] The 3D object annotation detection method provided in this invention selects a first annotation box region to eliminate the influence of other directions and focuses on calculating the confidence level of the 3D annotation box in the length direction. This avoids calculation errors caused by the influence of other directions, making the first confidence level of the determined 3D annotation box in the length direction more accurate. Only when there is no occlusion in the first annotation box region is the first confidence level of the target object's 3D annotation box in the length direction determined to be completely reliable, making the completely reliable 3D annotation box more meaningful.

[0114] The three-dimensional object annotation detection method provided in this embodiment of the invention compares the visibility of the first annotation box region with a preset visibility threshold and quantifies the visibility result of the first annotation box region, thereby enabling a more accurate judgment of the reliability of the three-dimensional annotation box in the length direction.

[0115] Similarly, the second confidence level in the width direction can be determined in the same way. In the preset sphere, the width direction corresponds to the Y-axis. When calculating the visibility of the target object's 3D bounding box in the width direction, a very small value is assigned to the length direction to obtain the second bounding box region. This second bounding box region is then projected onto the preset sphere. If the target object is fully visible in the width direction, the second bounding box region will appear in the projected view as having no overlap with the occluding object. In this case, the second confidence level is 1, meaning the 3D bounding box is completely reliable in the width direction.

[0116] Conversely, if the area of ​​the second annotation box overlaps with the occluding object in the projected view, it indicates that the occluding object occludes the target object to some extent in the Y-axis direction, i.e., the width direction.

[0117] The visibility of the second annotation box area = the second confidence level in the width direction = the second visible projection area / the second annotation box projection area.

[0118] The second visible projection area is the projection area of ​​the unobstructed area of ​​the second annotation box area on the preset projection surface, and the second annotation box projection area is the projection area of ​​the second annotation box area on the preset projection surface.

[0119] In another embodiment of the present invention, after calculating the second confidence level, the value of the second confidence level is compared with that of a preset visibility threshold. If the second confidence level is not less than the preset visibility threshold, it indicates that the occlusion of the target object in the width direction is not severe, and the information of its 3D bounding box in the width direction is reliable. If the second confidence level is less than the preset visibility threshold, it indicates that the occlusion of the target object in the width direction is relatively severe, and the information of its 3D bounding box in the width direction is unreliable.

[0120] In other words, the second confidence level can be divided into three cases based on the occlusion of the target object: completely trustworthy, trustworthy, and untrustworthy.

[0121] As can be seen from the above, the three-dimensional object annotation detection method provided in this embodiment of the invention obtains the visibility of the target object in the length and width directions respectively to obtain the first confidence and second confidence of the three-dimensional annotation box. The confidence of the three-dimensional annotation box obtained in this way can more completely reflect the credibility of the three-dimensional annotation box, thereby improving the accuracy of the detection results.

[0122] In one embodiment of the present invention, the visibility of the target object further includes visibility in the height direction, and the confidence of the 3D bounding box further includes a third confidence corresponding to the visibility in the height direction. When calculating the third confidence, there is no need to process the length and width directions; the occlusion in the height direction can be observed simply by projecting the 3D bounding box onto a preset projection plane. For example,

[0123] When the height direction (i.e., the third confidence level) is completely reliable, it means that there is no occlusion in the entire height direction when it is displayed on the projection map.

[0124] When there is occlusion in the height direction (for example, the tires at the bottom of the vehicle are not visible due to occlusion), a threshold can be set according to the proportion of occlusion. For example, if the occlusion proportion reaches 50%, the height direction is considered unreliable. When the occlusion proportion is less than 20%, the height direction is considered reliable, but not completely reliable.

[0125] The three-dimensional object annotation detection method provided in this embodiment of the invention introduces visibility in the height direction when determining the confidence of the three-dimensional annotation box, providing richer criteria for determining the confidence of the three-dimensional annotation box and further improving the accuracy of three-dimensional annotation box detection.

[0126] In one embodiment of the present invention, determining the confidence level of the 3D object bounding box based at least on the first confidence level and the second confidence level includes:

[0127] The confidence level of the 3D bounding box is determined based on the first confidence level, the second confidence level, and the third confidence level. For example, if the first confidence level, the second confidence level, and the third confidence level of the 3D bounding box of the target object are all completely confident, then the 3D bounding box is determined to be completely confident. That is, if there is no occlusion in the length, width, and height directions of the 3D bounding box, then the 3D bounding box is completely confident, and in this case, the 3D bounding box can be provided as a truth value with high confidence.

[0128] When at least one of the length, width, and height directions is considered reliable, and the other two are considered fully reliable, it indicates that further judgment is needed based on the 3D bounding box before it can be applied to subsequent operations, such as further road condition assessment. In this case, the target object corresponding to the 3D bounding box can be specially marked, and the changes in the 3D point cloud of the target object can be continuously observed over multiple frames to determine whether to accept the bounding box or return to re-label it.

[0129] If any of the length, width, or height directions are unreliable, the 3D bounding box of the target object is deemed unreliable, and a re-correction operation is required.

[0130] The 3D object annotation detection method provided in this invention can only provide a 3D annotation frame as a high-confidence true value when the 3D annotation frame of the target object is completely reliable in the length, width, and height directions. In other cases, manual intervention or other information assistance is required to further determine whether the annotation result is acceptable. This ensures the accuracy of the 3D annotation frame of the target object as much as possible and improves the accuracy of the target object annotation.

[0131] In one embodiment of the present invention, such as Figure 6 As shown, a schematic diagram of a three-dimensional object annotation detection device is also provided, including:

[0132] The annotation box acquisition module 601 is used to acquire the three-dimensional annotation box of the target object to be detected in the three-dimensional point cloud data;

[0133] The occlusion determination module 602 is used to project the three-dimensional annotation box onto a preset projection surface and determine whether there is occlusion in the three-dimensional annotation box. The origin of the preset projection surface is the origin of the sensor coordinate system corresponding to the three-dimensional point cloud data.

[0134] The confidence determination module 603 is used to determine the confidence of the three-dimensional annotation box based on the occlusion of the target object, according to the occlusion situation of the three-dimensional annotation box.

[0135] As can be seen from the above, the 3D object annotation detection device provided in this embodiment of the invention first acquires the 3D annotation box of the target object in the 3D point cloud data, and projects the 3D annotation box onto a preset projection surface. The origin of the preset projection surface is the origin of the sensor coordinate system corresponding to the 3D point cloud data. Then, it determines whether there is occlusion in the 3D annotation box. If there is, it obtains and determines the confidence level of the 3D annotation box based on the occlusion of the 3D annotation box and the visibility of the target object. This invention calculates the visibility of the target object by the occlusion of the 3D annotation box and then determines the confidence level of the 3D annotation box, quantifying the confidence level of the 3D annotation box. This allows the confidence level of the 3D annotation of the target object to be accurately detected and further judged, improving the accuracy of 3D annotation, providing higher reliability and applicability for the application of 3D annotation, and providing a criterion for the accuracy of manual annotation.

[0136] In one embodiment of the present invention, the visibility of the target object includes visibility in the length direction and the width direction, and the confidence determination module 603 includes:

[0137] The first confidence level determination submodule is used to obtain the first confidence level of the three-dimensional annotation box based on the occlusion of the three-dimensional annotation box and the visibility of the target object in the length direction.

[0138] The second confidence determination submodule is used to obtain the second confidence of the three-dimensional annotation box based on the occlusion of the three-dimensional annotation box and determine the second confidence of the three-dimensional annotation box based on the visibility of the target object in the width direction.

[0139] The confidence level determination submodule is used to determine the confidence level of the 3D bounding box based at least on the first confidence level and the second confidence level.

[0140] The three-dimensional object annotation detection device provided in this embodiment of the invention obtains the visibility of the target object in the length and width directions to obtain the first confidence and second confidence of the three-dimensional annotation box. The confidence of the three-dimensional annotation box obtained based on this can more completely reflect the credibility of the three-dimensional annotation box, thereby improving the accuracy of the detection results.

[0141] In one embodiment of the present invention, the first confidence level determination submodule includes:

[0142] A region projection unit is used to select a first annotation box region along the width direction of the three-dimensional annotation box on one side near the origin of the preset projection surface, and project the first annotation box region onto the preset projection surface.

[0143] The visibility calculation unit is used to obtain the occlusion status of the first annotation box area and calculate the visibility of the first annotation box area based on the occlusion status of the first annotation box area;

[0144] The confidence determination unit is used to determine the first confidence level of the three-dimensional annotation box of the target object in the length direction based on the visibility of the first annotation box area.

[0145] In one embodiment of the present invention, the confidence level determination unit is specifically used for:

[0146] If there is no occlusion in the first annotation box area, the first confidence level of the three-dimensional annotation box of the target object in the length direction is determined to be completely reliable.

[0147] The 3D object annotation detection device provided in this embodiment of the invention selects a first annotation box region, eliminates the influence that may be caused by other directions, and focuses on calculating the confidence level of the 3D annotation box in the length direction. This avoids calculation errors caused by the influence of other directions, making the first confidence level of the determined 3D annotation box in the length direction more accurate. Only when there is no occlusion in the first annotation box region is the first confidence level of the target object's 3D annotation box in the length direction determined to be completely reliable, making the completely reliable 3D annotation box more meaningful.

[0148] In one embodiment of the present invention, the visibility of the target object further includes visibility in the height direction, and the confidence of the three-dimensional bounding box further includes a third confidence corresponding to the visibility in the height direction. Therefore, the confidence determination submodule is specifically used for:

[0149] The confidence level of the 3D bounding box is determined based on the first confidence level, the second confidence level, and the third confidence level.

[0150] The three-dimensional object annotation detection device provided in this embodiment of the invention introduces visibility in the height direction when determining the confidence of the three-dimensional annotation box, providing richer criteria for determining the confidence of the three-dimensional annotation box and further improving the accuracy of three-dimensional annotation box detection.

[0151] In one embodiment of the present invention, the confidence level determination submodule is specifically used for:

[0152] When the first confidence level, the second confidence level, and the third confidence level of the 3D annotation box of the target object are all completely reliable, the 3D annotation box is determined to be completely reliable.

[0153] The 3D object annotation detection device provided in this embodiment of the invention can only provide a 3D annotation frame as a high-confidence true value when the 3D annotation frame of the target object is completely reliable in the length, width, and height directions. In other cases, manual intervention or other information assistance is required to further determine whether the annotation result is acceptable. This ensures the accuracy of the 3D annotation frame of the target object as much as possible, improving the accuracy of the target object annotation.

[0154] In one embodiment of the present invention, the confidence level determination unit is specifically used for:

[0155] If the visibility of the first annotation box area is not less than a preset visibility threshold, the three-dimensional annotation box of the target object is determined to be credible in the length direction; if the visibility of the first annotation box area is less than the preset visibility threshold, the three-dimensional annotation box of the target object is determined to be unreliable in the length direction.

[0156] The three-dimensional object annotation detection method provided in this embodiment of the invention compares the visibility of the first annotation box region with a preset visibility threshold and quantifies the visibility result of the first annotation box region, thereby enabling a more accurate judgment of the reliability of the three-dimensional annotation box in the length direction.

[0157] This invention also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0158] Memory 703 is used to store computer programs;

[0159] The processor 701, when executing the program stored in the memory 703, implements the method steps for detecting any of the above-mentioned three-dimensional object annotations.

[0160] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0161] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0162] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0163] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the detection method for any of the above-described three-dimensional object annotations.

[0165] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the three-dimensional object annotation detection methods described in the above embodiments.

[0166] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0167] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0169] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method of detecting three-dimensional object annotation, characterized by, The method comprises: obtaining a three-dimensional bounding box of a target object to be detected in three-dimensional point cloud data; projecting the three-dimensional bounding box onto a preset projection surface to determine whether there is an occlusion in the three-dimensional bounding box, wherein the preset projection surface comprises a preset sphere, and the origin of the preset sphere is the origin of a sensor coordinate system corresponding to the three-dimensional point cloud data; if there is, determining a first confidence of the three-dimensional bounding box based on the occlusion of the three-dimensional bounding box and the visibility of the target object in the length direction; determining a second confidence of the three-dimensional bounding box based on the occlusion of the three-dimensional bounding box and the visibility of the target object in the width direction, wherein the visibility of the target object is obtained based on the unoccluded part of the three-dimensional bounding box of the target object; determining the confidence of the three-dimensional bounding box based on at least the first confidence and the second confidence.

2. The method of claim 1, wherein, The determination of the first confidence of the three-dimensional bounding box based on the occlusion of the three-dimensional bounding box and the visibility of the target object in the length direction comprises: selecting a first bounding box region on the side close to the origin of the preset projection surface along the width direction in the three-dimensional bounding box, and projecting the first bounding box region onto the preset projection surface; obtaining the occlusion of the first bounding box region, and calculating the visibility of the first bounding box region based on the occlusion of the first bounding box region; determining the first confidence of the three-dimensional bounding box of the target object in the length direction based on the visibility of the first bounding box region.

3. The method of claim 2, wherein, The determination of the first confidence of the three-dimensional bounding box of the target object in the length direction based on the visibility of the first bounding box region comprises: in the case where there is no occlusion in the first bounding box region, determining that the first confidence of the three-dimensional bounding box of the target object in the length direction is completely reliable.

4. The method of claim 1, wherein, The visibility of the target object also includes the visibility in the height direction, and the confidence of the three-dimensional bounding box also includes a third confidence corresponding to the visibility in the height direction, and the determination of the confidence of the three-dimensional bounding box based on at least the first confidence and the second confidence comprises: determining the confidence of the three-dimensional bounding box based on the first confidence, the second confidence, and the third confidence.

5. The method of claim 4, wherein, The determination of the confidence of the three-dimensional bounding box based on the first confidence, the second confidence, and the third confidence comprises: when the first confidence, the second confidence, and the third confidence of the three-dimensional bounding box of the target object are all completely reliable, determining that the three-dimensional bounding box is completely reliable.

6. The method of claim 2, wherein, The determination of the first confidence of the three-dimensional bounding box of the target object in the length direction based on the visibility of the first bounding box region comprises: In a case where the visibility of the first bounding box region is not less than a preset visibility threshold, it is determined that the three-dimensional bounding box of the target object is reliable in the length direction; and in a case where the visibility of the first bounding box region is less than the preset visibility threshold, it is determined that the three-dimensional bounding box of the target object is unreliable in the length direction.

7. A three-dimensional object mark detection device, characterized by, The apparatus comprises: a bounding box obtaining module configured to obtain a three-dimensional bounding box of a target object to be detected in three-dimensional point cloud data; an occlusion condition judging module configured to project the three-dimensional bounding box onto a preset projection surface, and judge whether there is an occlusion condition in the three-dimensional bounding box, wherein the preset projection surface comprises a preset sphere, and an origin of the preset sphere is an origin of a sensor coordinate system corresponding to the three-dimensional point cloud data; a confidence degree determining module configured to, if there is an occlusion condition, determine a first confidence degree of the three-dimensional bounding box according to the occlusion condition of the three-dimensional bounding box and based on a visibility of the target object in a length direction; determine a second confidence degree of the three-dimensional bounding box according to the occlusion condition of the three-dimensional bounding box and based on a visibility of the target object in a width direction, wherein the visibility of the target object is obtained based on a part of the three-dimensional bounding box of the target object that is not occluded; determine a confidence degree of the three-dimensional bounding box according to at least the first confidence degree and the second confidence degree.

8. An electronic device, comprising: The apparatus comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory is configured to store a computer program; the processor is configured to execute the program stored on the memory, and implement the method steps of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-6.

Images