A point cloud detection optimization method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a point cloud detection optimization method and device, electronic equipment and storage medium, relates to the technical field of artificial intelligence, and particularly relates to the technical field of intelligent traffic, automatic driving, high-precision map and the like. The specific implementation scheme is: determining a first image related to a three-dimensional bounding box; wherein the three-dimensional bounding box is obtained through point cloud detection, and the first image is an image containing a target object corresponding to the three-dimensional bounding box; projecting the three-dimensional bounding box to the first image to obtain a projected bounding box; and performing image detection on the first image to obtain a two-dimensional bounding box; determining whether the three-dimensional bounding box is splittable by using the positional relationship between the projected bounding box and the two-dimensional bounding box. The present disclosure can improve the accuracy of point cloud detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to the technical field of intelligent transportation, autonomous driving, high-precision map, etc. BACKGROUND

[0002] Point cloud detection refers to obtaining point cloud data through laser radar scanning, and detecting the point cloud data to identify a target object. In point cloud detection, there is a situation that detection errors are caused due to the target objects being very close to each other; for example, due to the positions of two or more objects being very close to each other, it is difficult to distinguish the two or more objects through point cloud detection only, and the two or more objects can be misidentified as one object. SUMMARY

[0003] The present disclosure provides a point cloud detection optimization method and device, an electronic device, and a storage medium.

[0004] According to an aspect of the present disclosure, a point cloud detection optimization method is provided, comprising:

[0005] determining a first image related to a three-dimensional bounding box; wherein the three-dimensional bounding box is obtained through point cloud detection, and the first image is an image containing a target object corresponding to the three-dimensional bounding box;

[0006] projecting the three-dimensional bounding box to the first image to obtain a projected bounding box; and performing image detection on the first image to obtain a two-dimensional bounding box;

[0007] determining, by using a positional relationship between the projected bounding box and the two-dimensional bounding box, whether the three-dimensional bounding box is splittable.

[0008] According to another aspect of the present disclosure, a point cloud detection optimization device is provided, comprising:

[0009] a first image determination module configured to determine a first image related to a three-dimensional bounding box; wherein the three-dimensional bounding box is obtained through point cloud detection, and the first image is an image containing a target object corresponding to the three-dimensional bounding box;

[0010] a bounding box determination module configured to project the three-dimensional bounding box to the first image to obtain a projected bounding box; and perform image detection on the first image to obtain a two-dimensional bounding box;

[0011] a judgment module configured to determine, by using a positional relationship between the projected bounding box and the two-dimensional bounding box, whether the three-dimensional bounding box is splittable.

[0012] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein

[0015] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.

[0016] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method according to any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any embodiment of the present disclosure.

[0018] The point cloud detection optimization method proposed in the embodiments of the present disclosure determines whether the three-dimensional bounding box can be cut according to the position relationship of the projection bounding box obtained by the projection of the three-dimensional bounding box on the first image and the two-dimensional bounding box in the first image, which can improve the accuracy of point cloud detection.

[0019] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0021] Figure 1 is an application scenario diagram of the embodiments of the present disclosure;

[0022] Figure 2 is an implementation flowchart of the point cloud detection optimization method 200 according to an embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of the distance between the acquisition vehicle and the target object according to an embodiment of the present disclosure;

[0024] Figure 4 is a point cloud detection result optimization flowchart according to an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of the intersection relationship of the projection bounding box and the 2D bounding box according to an embodiment of the present disclosure;

[0026] Figure 6A is a schematic diagram of the first image 2D bounding box according to an embodiment of the present disclosure;

[0027] Figure 6B This is a schematic diagram of the 3D bounding box plane segmentation points of a first image according to an embodiment of the present disclosure;

[0028] Figure 7 This is a schematic diagram of a point cloud detection optimization method according to an embodiment of the present disclosure;

[0029] Figure 8 This is a schematic diagram of the structure of a point cloud detection optimization device 800 according to an embodiment of the present disclosure;

[0030] Figure 9 This is a schematic diagram of the structure of a point cloud detection optimization device 900 according to an embodiment of the present disclosure;

[0031] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0032] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] Point cloud detection refers to the process of obtaining point cloud data through LiDAR scanning and then detecting the point cloud data to identify target objects. In point cloud detection, errors can occur due to target objects being very close together; for example, if two or more objects are very close together, it may be difficult to distinguish them using point cloud detection alone, potentially leading to misidentification of two or more objects as a single object.

[0034] Point cloud detection is an important stage in the production process of high-precision maps. High-precision maps, also known as high-precision maps, have wide applications in the fields of autonomous driving, intelligent transportation, etc. High-precision maps have accurate vehicle position information and rich road element data information, which can help vehicles predict complex road information such as slope, curvature, heading, etc., and better avoid potential risks. High-precision map production is to describe map elements in the real physical world with the least amount of data, where each map element can be composed of geometric coordinates and attribute information. Therefore, one of the core demands of high-precision map production is to obtain the coordinates (such as coordinates in the world coordinate system) of each map element (such as lane lines, traffic lights, pedestrian crossings, etc.) in the physical world. Map elements can include ground elements, facade elements, etc.; common facade elements include signs, poles, traffic lights, etc., which play an important role in road information acquisition and autonomous driving positioning as important map elements. Map elements in high-precision maps can also be considered as target objects in a traffic scene.

[0035] In addition to the production process of high-precision maps, autonomous vehicles also need to detect target objects from point clouds.

[0036] In related technologies, laser radar is usually used for point cloud detection. For example, in the process of high-precision map making or the perception process of autonomous vehicles, laser radar is used to collect point cloud data, and relevant algorithms are used to detect the collected point cloud data to obtain the three-dimensional bounding box of the target object. In actual scenarios, there are cases where the positions of objects are very close, resulting in point cloud detection errors. For example, there are many traffic signs with close geometric positions or even adhesion. In this case, it is difficult for laser radar scanning point cloud data to distinguish independent signs, resulting in errors in point cloud detection of signs and affecting the quality of high-precision maps. Such detected erroneous signs can be referred to as adhesion point cloud signs.

[0037] Embodiments of the present disclosure propose a point cloud detection optimization method that can improve the optimization of point cloud detection results. Figure 1 is a schematic diagram of an application scenario of embodiments of the present disclosure. As Figure 1As shown, the network architecture based on which the embodiments of the present disclosure can include an image acquisition device 110, a laser radar 120, and a point cloud detection optimization apparatus 130 for implementing the point cloud detection optimization method. The image acquisition device 110 and the point cloud detection optimization apparatus 130 can be connected through a wired network or a wireless network. The image acquisition device 110 acquires images and provides the acquired images to the point cloud detection optimization apparatus 130 through the wired network or the wireless network. The laser radar 120 and the point cloud detection optimization apparatus 130 can be connected through a wired network or a wireless network. The laser radar 120 generates point cloud data and provides the point cloud data to the point cloud detection optimization apparatus 130 through the wired network or the wireless network. The point cloud detection optimization apparatus 130 performs point cloud detection using the point cloud data and optimizes the point cloud detection result using the image data. The network architecture based on which the embodiments of the present disclosure can also include a data server 140, which can be a cloud server or a server cluster and can be used for storing data. After completing the point cloud detection optimization, the point cloud detection optimization apparatus 130 can send the optimized point cloud detection result to the data server 140 for storage.

[0038] Figure 2 The implementation flowchart of the point cloud detection optimization method 200 provided by an embodiment of the present disclosure includes the following steps.

[0039] S210, determining a first image related to a three-dimensional bounding box; wherein the three-dimensional bounding box is obtained through point cloud detection, and the first image is an image containing a target object corresponding to the three-dimensional bounding box;

[0040] S220, projecting the three-dimensional bounding box to the first image to obtain a projected bounding box; and performing image detection on the first image to obtain a two-dimensional bounding box;

[0041] S230, determining whether the three-dimensional bounding box is splittable by using the positional relationship between the projected bounding box and the two-dimensional bounding box.

[0042] In some examples, point cloud data collected by a laser radar can be subjected to point cloud detection to obtain a three-dimensional (3D) bounding box of a target object. For example, the target object is a traffic sign, and the point cloud data is subjected to traffic sign detection to determine a three-dimensional bounding box containing the traffic sign. The point cloud detection method can be completed by using a deep learning model or by using a traditional algorithm.

[0043] In some examples, the first image can be an image taken under the condition that a lens is directed towards the three-dimensional bounding box and the distance between the lens and the three-dimensional bounding box is less than or equal to a preset threshold.

[0044] For example, an image acquisition device (such as a camera) and a LiDAR are installed on a data acquisition vehicle. The camera and LiDAR acquire data synchronously, with the camera acquiring video data and the LiDAR acquiring point cloud data. The position of the 3D bounding box can be considered the actual position of the target object. Therefore, the first image can be the image captured when the camera lens is facing the target object (i.e., the data acquisition vehicle is moving towards the target object) and the distance between the camera lens and the target object (i.e., the distance between the data acquisition vehicle and the target object) is less than or equal to a preset threshold. Figure 3 As shown, when the distance between the data collection vehicle and the target object reaches a preset threshold ( Figure 3 (represented by S in the text), and as the acquisition vehicle gradually approaches the target object, each frame of the video data it acquires can be used as the first image. For example... Figure 3 In this process, each frame of video data collected by the acquisition vehicle between position A and position B can be used as the first image; where position B is the location of the target object (i.e., the location of the 3D bounding box determined by point cloud detection), and the distance between position A and position B is a preset threshold. This preset threshold can be a pre-set value, for example, a preset threshold of 50 meters.

[0045] Compared to point cloud detection, image detection offers higher accuracy. For example, in facade feature detection scenarios, some traffic signs are located close together, and due to the influence of the external environment, point cloud detection might mistakenly identify nearby traffic signs as a single object, leading to recognition errors; while image detection can distinguish multiple objects. Figure 4 For example, in order from left to right, Figure 4 The first image in the diagram is a schematic diagram of image detection for a scene. Figure 4 In the first image, the two-dimensional (2D) bounding boxes of each traffic sign can be detected; Figure 4 The second image in the diagram is a schematic diagram of point cloud detection in this scene. Figure 4 In the second image, because the three traffic signs at the top are close together, point cloud detection made a recognition error, mistaking the three traffic signs for a single object and only defining one 3D bounding box. To address similar situations, the point cloud detection optimization method proposed in this embodiment can identify whether the 3D bounding box determined by point cloud detection is segmentable, and segment the segmentable 3D bounding box. Following the order from left to right... Figure 4 The third image in the example is a schematic diagram of the segmented 3D bounding box. Figure 4 The first image shows the video detection results, for Figure 4the point cloud detection result shown in the second image is identified and segmented. In addition, according to requirements, the point cloud detection optimization method proposed in the embodiments of the present disclosure can also use the image detection result to split an actual traffic sign into two or more (for example Figure 5 vertical sign) according to its content.

[0046] Regarding the determination of whether the 3D bounding box is splittable, the embodiments of the present disclosure at least include the following two effectiveness determination methods:

[0047] Method one, using a first image to determine:

[0048] Projecting the 3D bounding box to the first image to obtain a projection bounding box; and performing image detection on the first image to obtain a 2D bounding box;

[0049] Using the positional relationship between the projection bounding box and the 2D bounding box to determine whether the three-dimensional bounding box is splittable.

[0050] In some embodiments, projecting the three-dimensional bounding box to the first image can include: using the camera parameters corresponding to the first image to project the three-dimensional bounding box to the first image. Wherein, the camera parameters corresponding to the first image can include: the camera parameters of the image acquisition device when shooting the first image, which includes at least one of the internal parameters and the external parameters. For example, the external parameters can include the pose of the image acquisition device, including the three-dimensional coordinate information and the lens orientation information of the image acquisition device, etc.

[0051] Specifically, the three-dimensional coordinates of the three-dimensional bounding box can be determined; then, according to the parameters of the point cloud acquisition device (such as a laser radar) and the camera parameters corresponding to the first image, a calibration matrix between the point cloud acquisition device and the image acquisition device is determined; according to the three-dimensional coordinates of the three-dimensional bounding box and the calibration matrix, the two-dimensional coordinates of the three-dimensional bounding box in the image coordinate system are determined, thereby obtaining the projection bounding box obtained by projection.

[0052] In some examples, the three-dimensional bounding box can be determined to be splittable when the projection bounding box and the 2D bounding box have an intersection, the ratio of the area of the intersection to the area of the 2D bounding box is greater than a first threshold, and the ratio of the area of the intersection to the area of the projection bounding box is less than a second threshold;

[0053] Wherein, the first threshold and the second threshold are positive numbers preset in advance.

[0054] If the above conditions are not met, it can be considered that the 3D bounding box is not splittable.

[0055] For example, Box 3d_2dThis represents the projection of the 3D bounding box onto the first image, i.e., the projection bounding box; it can also represent the area of ​​the projection bounding box; the projection bounding box is two-dimensional.

[0056] box 2d It can represent a 2D bounding box, or it can represent the area of ​​the 2D bounding box;

[0057] I box2d =Box 3d_2d ∩box 2d , which represents the intersection of the projected bounding box and the 2D bounding box, or the area of ​​the intersection;

[0058] The 3D bounding box is considered valid if all of the above bounding boxes satisfy the following relationship:

[0059]

[0060] Where T1 represents the first threshold and T2 represents the second threshold; T1 and T2 are pre-set positive numbers, for example, T1 is 0.9 and T2 is 0.5.

[0061] by Figure 5 For example, Figure 6A In this diagram, rectangle ABCD represents the projected bounding box, and rectangles A'B'C'D' represent the 2D bounding box. When the two bounding boxes intersect, and this intersection occupies a large proportion of the 2D bounding box while occupying a small proportion of the projected bounding box, it indicates that the 2D bounding box is related to the 3D bounding box, and the target object corresponding to the 2D bounding box is part of the target object corresponding to the 3D bounding box. Therefore, it can be determined that the 3D bounding box is segmentable. In this way, the 3D bounding boxes of multiple very close target objects can be accurately identified.

[0062] Method 2:

[0063] Method 2 is similar to Method 1, except that it uses multiple first images to determine the separability of the 3D bounding box, and combines the results of these multiple determinations to determine whether the 3D bounding box is separable. When determining the separability of a single first image, the specific determination method is the same as in Method 1.

[0064] For example, identify at least two first images associated with the 3D bounding box;

[0065] For each of the at least two first images, the positional relationship between the projection bounding box and the two-dimensional bounding box is used to determine whether the three-dimensional bounding box can be segmented, so as to obtain the first judgment result for the at least two first images.

[0066] Based on the first judgment result for at least two first images, determine whether the 3D bounding box is segmentable.

[0067] In some embodiments, the first determination result is splittable or non-splittable.

[0068] According to the determination results of the at least two first images, it is determined whether the three-dimensional bounding box is splittable, which can include:

[0069] In a case where the ratio of the first quantity to the second quantity is greater than or equal to a preset proportion, it is determined that the three-dimensional bounding box is splittable; wherein,

[0070] The first quantity is the number of first determination results that are splittable.

[0071] The second quantity is the number of first determination results.

[0072] The second quantity is equal to the sum of the number of first determination results that are splittable and the number of first determination results that are non-splittable.

[0073] In some embodiments, the first image related to the 3D bounding box can refer to a plurality of continuous first images related to the 3D bounding box.

[0074] For example, 10 first images related to the 3D bounding box are obtained, and for each of the first images, a first determination result is obtained, which is splittable or non-splittable; if 9 of the first determination results are splittable and 1 of the first determination results is non-splittable, the ratio of the first quantity to the second quantity is 0.9; assuming that the preset proportion is 0.9, the ratio is equal to the preset proportion, and it can be determined that the 3D bounding box is splittable.

[0075] It can be seen that, compared with mode one, mode two uses a plurality of first images to determine the splittability of the 3D bounding box, which can improve the accuracy of the determination result.

[0076] After it is determined that the 3D bounding box is splittable, the 3D bounding box can be further split according to the embodiments of the present disclosure. The embodiments of the present disclosure include at least the following two splitting modes:

[0077] Mode one, splitting using one first image:

[0078] In a case where it is determined that the 3D bounding box is splittable, a 2D bounding box in the first image related to the 3D bounding box is projected onto a plane of the 3D bounding box to obtain a splitting point;

[0079] The 3D bounding box is split using the splitting point to obtain a split 3D bounding box.

[0080] The mode is particularly suitable for a scenario in which the target object is a flat shape, for example, the target object is a traffic sign. Such an object has a small thickness, and its 3D bounding box is usually approximated to a plane, which can be referred to as the plane of the 3D bounding box.

[0081] In some examples, when projecting the 2D bounding box in the first image to the plane of the 3D bounding box, the pixel point of the 2D bounding box in the first image can be projected to the plane of the 3D bounding box by using the camera parameters corresponding to the first image. For example, the two-dimensional coordinates of the pixel point of the 2D bounding box are determined; then, a calibration matrix between the image acquisition device and the point cloud acquisition device is determined according to the camera parameters corresponding to the first image and the parameters of the point cloud acquisition device (such as a laser radar); and the position of the pixel point of the 2D bounding box in the plane of the 3D bounding box is determined according to the two-dimensional coordinates of the pixel point of the 2D bounding box and the calibration matrix, so as to obtain the segmentation point.

[0082] It can be seen that, by using the image detection result, the 3D bounding box can be segmented by using the advantages of image detection, and the optimization of point cloud detection is realized.

[0083] Mode two, segmentation by using at least a first image:

[0084] Mode two is similar to mode one described above, and the difference lies in that mode two segments the 3D bounding box by using multiple first images. For the same pixel point, a projection point on the plane of the 3D bounding box is determined by using each first image; and the segmentation point is determined by using the projection points.

[0085] In some embodiments, at least two first images related to the 3D bounding box are acquired;

[0086] The same pixel point of the 2D bounding box in the acquired at least two first images is projected to the plane of the 3D bounding box to obtain at least two projection points.

[0087] The segmentation point is determined by using the at least two projection points.

[0088] Specifically, when projecting, for each of the at least two first images, the same pixel point in the first image is projected to the plane of the 3D bounding box by using the camera parameters corresponding to the first image.

[0089] Figure 6A And 6B is a schematic diagram for determining a segmentation point of a 3D bounding box according to an embodiment of the present disclosure. As shown in Figure 6B Three 2D bounding boxes in a first image are shown, and point A is a pixel point in one of the 2D bounding boxes; for each first image, there is a point A. Figure 6Ba plane of the 3D bounding box corresponding to the first image is displayed, and for a plurality of first images, a projection point is determined on the plane of the 3D bounding box for each point A on each first image (e.g. Figure 7 A plurality of points are scattered in the 3D bounding box, and the plurality of points are projection points of the plurality of first images. The plurality of projection points can be used to determine a split point. For example, the x and y coordinates of the plurality of projection points can be averaged respectively to obtain coordinate values (x', y'), which are the coordinates of the split point. In the same way, the same operation can be performed for other pixel points to determine other split points. After the split points are determined, the 3D bounding box can be split using the split points.

[0090] Obviously, compared with the first mode, the second mode uses a plurality of first images to determine the split point, which can reduce the position deviation between image detection and point cloud detection, and more accurately determine the position of the split point.

[0091] In addition, after the 3D bounding box is split, the attribute information corresponding to the 2D bounding box in the first image related to the 3D bounding box can be used to assign attributes to the split 3D bounding box. The attribute information can refer to the type (such as traffic lights, traffic signs, lane lines, etc.) and the information carried (such as "height limit 4.5 meters", "no pedestrian access", etc.).

[0092] For example, if the 3D bounding box is split by a 2D bounding box in the first image, and the attribute of the 2D bounding box is "traffic sign, no pedestrian access", the attribute of the split 3D bounding box can also be assigned as "traffic sign, no pedestrian access".

[0093] Point cloud detection is difficult to determine the attributes of the target object. For example, for a traffic sign, point cloud detection can determine its shape, but cannot determine the text on the traffic sign. Image detection can solve this problem well. Therefore, the attribute information obtained by image detection is used to assign attributes to the 3D bounding box, which can improve the detection effect.

[0094] The following describes an embodiment of the point cloud detection optimization of the present disclosure in the actual high-definition map production process, taking the point cloud detection optimization of the data collected by a single collection task as an example. Before target detection, in order to ensure the accuracy of point cloud element detection and three-dimensional image reconstruction, the pose data and other parameters of the laser radar and image collection device are first optimized.

[0095] Then, data is collected by the image collection device (such as a camera) and the laser radar respectively. The camera and the laser radar after parameter optimization can be installed on the collection vehicle, and the camera and the laser radar collect data synchronously as the collection vehicle moves, wherein the camera collects video data and the laser radar collects point cloud data. The video image and the point cloud data in the road can be collected synchronously or separately.

[0096] The collection task of one collection vehicle is referred to as a single collection task. For the video data collected for the single collection task, the video data can be divided into multiple subgraphs according to a predetermined division rule, and each subgraph includes continuous multiple frames of images. For example, the video data is divided into a subgraph every 10 minutes according to the time length of the video data. Alternatively, the video data collected by the collection vehicle is divided into a subgraph every 50 meters of driving distance.

[0097] Point cloud detection optimization is performed on each subgraph divided. Figure 7 is a schematic diagram of a point cloud detection optimization method according to an embodiment of the present disclosure, and Figure 8 In the example shown in FIG. 7, a vertical element of a high-definition map is taken as an example for introduction. The method includes the following steps.

[0098] S710: Point cloud detection is performed on the point cloud data to obtain a 3D bounding box.

[0099] Specifically, the point cloud data is detected, vertical elements are identified, and a 3D bounding box of the vertical elements is determined.

[0100] The vertical elements include a sign, a pole, a traffic light, and the like.

[0101] S720: Image detection is performed on the image to obtain an image detection result.

[0102] Specifically, vertical elements in the image are identified, and a 2D bounding box of the vertical elements is determined.

[0103] Steps S710 and S720 can be executed synchronously or sequentially, and the present embodiment does not limit the execution order of the two steps.

[0104] S730: Whether the 3D bounding box can be split is determined by using the detection results of steps S710 and S720, camera parameters such as a camera pose and internal parameter data corresponding to the image, and the like. If the 3D bounding box can be split, step S740 is executed.

[0105] S740: The 3D bounding box is split to obtain an optimized 3D bounding box (i.e., a split 3D bounding box).

[0106] The specific judgment and splitting manner has been described in the above embodiments, and will not be described here.

[0107] In addition, the embodiment of the present disclosure can also utilize the attribute of the target object determined by image detection to assign the attribute of the split 3D bounding box.

[0108] The embodiment of the present disclosure also proposes a point cloud detection optimization device, Figure 9 is a structural schematic diagram of a point cloud detection optimization device 800 according to an embodiment of the present disclosure, comprising:

[0109] A first image determination module 810 is configured to determine a first image related to a three-dimensional bounding box, wherein the three-dimensional bounding box is obtained by point cloud detection, and the first image is an image containing a target object corresponding to the three-dimensional bounding box;

[0110] A bounding box determination module 820 is configured to project the three-dimensional bounding box to the first image to obtain a projection bounding box, and perform image detection on the first image to obtain a two-dimensional bounding box;

[0111] A judgment module 830 is configured to determine whether the three-dimensional bounding box is splittable by using the positional relationship between the projection bounding box and the two-dimensional bounding box.

[0112] In some embodiments, the first image determination module 810 is configured to determine at least two frames of first images related to the three-dimensional bounding box;

[0113] The judgment module 830 comprises:

[0114] A first judgment sub-module 831 is configured to, for each frame of the at least two frames of first images, determine whether the three-dimensional bounding box is splittable by using the positional relationship between the projection bounding box and the two-dimensional bounding box, to obtain a first judgment result for the at least two frames of first images;

[0115] A second judgment sub-module 832 is configured to determine whether the three-dimensional bounding box is splittable according to the first judgment result for the at least two frames of first images.

[0116] In some embodiments, the first judgment result is splittable or non-splittable;

[0117] The second judgment sub-module 832 is configured to determine that the three-dimensional bounding box is splittable in a case where the ratio of the first quantity to the second quantity is greater than or equal to a preset proportion; wherein

[0118] The first quantity is the number of the first judgment results that are splittable;

[0119] The second quantity is the number of the first judgment results.

[0120] In some embodiments, the position relationship between the projection bounding box and the two-dimensional bounding box is used to determine whether the three-dimensional bounding box is splittable, including: in a case where the projection bounding box and the two-dimensional bounding box have an intersection, a ratio of an area of the intersection to an area of the two-dimensional bounding box is greater than a first threshold, and a ratio of the area of the intersection to an area of the projection bounding box is less than a second threshold, it is determined that the three-dimensional bounding box is splittable; wherein the first threshold and the second threshold are positive numbers preset in advance.

[0121] Figure 9 FIG. 9 is a structural schematic diagram of a point cloud detection optimization device 900 according to an embodiment of the present disclosure, as shown in some embodiments, the point cloud detection optimization device 900 proposed in the embodiments of the present disclosure further includes: Figure 10

[0122] A split point determination module 940 is configured to, in a case where it is determined that the three-dimensional bounding box is splittable, project the two-dimensional bounding box in the first image related to the three-dimensional bounding box to a plane of the three-dimensional bounding box to obtain a split point.

[0123] A split module 950 is configured to split the three-dimensional bounding box using the split point to obtain a split three-dimensional bounding box.

[0124] In some embodiments, the split point determination module 940 is configured to:

[0125] Obtain at least two first images related to the three-dimensional bounding box;

[0126] Project the same pixel points of the two-dimensional bounding boxes in the obtained at least two first images to the plane of the three-dimensional bounding box to obtain at least two projection points;

[0127] Determine the split point using the at least two projection points.

[0128] In some embodiments, the split point determination module 940 is configured to:

[0129] For each of the at least two first images, project the same pixel points in the first image to the plane of the three-dimensional bounding box using the camera parameters corresponding to the first image.

[0130] In some embodiments, the point cloud detection optimization device 900 further includes:

[0131] An attribute assignment module 960 is configured to assign attributes to the split three-dimensional bounding box using attribute information corresponding to the two-dimensional bounding box in the first image related to the three-dimensional bounding box.

[0132] In some embodiments, the bounding box determination module 820 is configured to project the three-dimensional bounding box to the first image using the camera parameters corresponding to the first image.​

[0133] In some embodiments, the camera parameters corresponding to the first image include camera parameters of the image acquisition device when the first image is captured, and the camera parameters include at least one of internal parameters and external parameters.

[0134] The specific functions and examples of the modules and sub-modules of the apparatus of the disclosed embodiments are described above in the related description of the corresponding steps in the method embodiments, and will not be described here.

[0135] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

[0136] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0137] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0138] As shown in ​ The device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0139] A number of the components in device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, a CD, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices over computer networks, such as the Internet, and / or various telecommunication networks.

[0140] The computing unit 1001 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the point cloud detection optimization method. For example, in some embodiments, the point cloud detection optimization method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the point cloud detection optimization method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the point cloud detection optimization method by any other appropriate means, such as by means of firmware.

[0141] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0142] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0145] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between a client and a server is one of client-server. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0147] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in sequence, or executed in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0148] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A point cloud detection optimization method, comprising: A first image associated with a 3D bounding box is determined; wherein the 3D bounding box is obtained by point cloud detection, and the first image is an image containing the target object corresponding to the 3D bounding box; The three-dimensional bounding box is projected onto the first image to obtain a projected bounding box; and image detection is performed on the first image to obtain a two-dimensional bounding box. By utilizing the positional relationship between the projection bounding box and the two-dimensional bounding box, it is determined whether the three-dimensional bounding box can be segmented; The step of determining the first image associated with the 3D bounding box includes: determining at least two first images associated with the 3D bounding box; Determining whether the 3D bounding box is separable based on the positional relationship between the projected bounding box and the 2D bounding box includes: For each of the at least two first images, the positional relationship between the projection bounding box and the two-dimensional bounding box is used to determine whether the three-dimensional bounding box can be segmented, so as to obtain a first judgment result for the at least two first images. Based on the first judgment result for the at least two first images, it is determined whether the three-dimensional bounding box can be segmented.

2. The method according to claim 1, wherein, The first determination result is either divisible or indivisible; The step of determining whether the 3D bounding box is segmentable based on the judgment results for the at least two frames of the first image includes: If the ratio of the first quantity to the second quantity is greater than or equal to a preset ratio, the three-dimensional bounding box is determined to be divisible; wherein, The first quantity is the number of segments that can be divided according to the first judgment result; The second quantity is the number of the first judgment results.

3. The method according to claim 1 or 2, wherein, The step of determining whether the 3D bounding box is separable based on the positional relationship between the projected bounding box and the 2D bounding box includes: If the projected bounding box and the two-dimensional bounding box intersect, the ratio of the area of ​​the intersection to the area of ​​the two-dimensional bounding box is greater than a first threshold, and the ratio of the area of ​​the intersection to the area of ​​the projected bounding box is less than a second threshold, then the three-dimensional bounding box is determined to be divisible. Wherein, the first threshold and the second threshold are preset positive numbers.

4. The method according to any one of claims 1-3, further comprising: If the three-dimensional bounding box is determined to be separable, the two-dimensional bounding box in the first image associated with the three-dimensional bounding box is projected onto the plane of the three-dimensional bounding box to obtain the cutting point; The three-dimensional bounding box is segmented using the segmentation points to obtain the segmented three-dimensional bounding box.

5. The method according to claim 4, wherein, The step of projecting the 2D bounding box in the first image associated with the 3D bounding box onto the plane of the 3D bounding box to obtain the cutting points includes: Acquire at least two first images associated with the 3D bounding box; Project the same pixels of the two-dimensional bounding boxes in at least two first images onto the plane of the three-dimensional bounding box to obtain at least two projection points; The cutting point is determined using the at least two projection points.

6. The method according to claim 5, wherein, The step of projecting the same pixels of the two-dimensional bounding boxes in at least two acquired first images onto the plane of the three-dimensional bounding box includes: For each of the at least two first images, using the camera parameters corresponding to the first image, the same pixels in the first image are projected onto the plane of the three-dimensional bounding box.

7. The method according to any one of claims 4-6, further comprising: The attributes of the segmented 3D bounding box are assigned values ​​using the attribute information corresponding to the 2D bounding box in the first image associated with the 3D bounding box.

8. The method according to any one of claims 1-6, wherein, The step of projecting the 3D bounding box onto the first image includes: Using the camera parameters corresponding to the first image, the 3D bounding box is projected onto the first image.

9. The method according to claim 6 or 8, wherein, The camera parameters corresponding to the first image include: the camera parameters of the image acquisition device when capturing the first image, and the camera parameters include at least one of internal parameters and external parameters.

10. A point cloud detection optimization device, comprising: The first image determination module is used to determine a first image related to a 3D bounding box; wherein the 3D bounding box is obtained by point cloud detection, and the first image is an image containing the target object corresponding to the 3D bounding box; A bounding box determination module is used to project the three-dimensional bounding box onto the first image to obtain a projected bounding box; and to perform image detection on the first image to obtain a two-dimensional bounding box. The judgment module is used to determine whether the three-dimensional bounding box can be segmented by using the positional relationship between the projection bounding box and the two-dimensional bounding box; The first image determination module is further configured to determine at least two first images associated with the three-dimensional bounding box; The judgment module includes: The first judgment submodule is used to determine whether the three-dimensional bounding box can be segmented for each of the at least two first images by using the positional relationship between the projection bounding box and the two-dimensional bounding box, so as to obtain a first judgment result for the at least two first images. The second judgment submodule is used to determine whether the three-dimensional bounding box can be segmented based on the first judgment result for the at least two frames of the first image.

11. The apparatus according to claim 10, wherein, The first determination result is either divisible or indivisible; The second judgment submodule is used to determine that the three-dimensional bounding box can be segmented if the ratio of the first quantity to the second quantity is greater than or equal to a preset ratio; wherein, The first quantity is the number of segments that can be divided according to the first judgment result; The second quantity is the number of the first judgment results.

12. The apparatus according to claim 10 or 11, wherein, The step of determining whether the three-dimensional bounding box can be segmented based on the positional relationship between the projected bounding box and the two-dimensional bounding box includes: determining that the three-dimensional bounding box can be segmented when the projected bounding box and the two-dimensional bounding box have an intersection, the ratio of the area of ​​the intersection to the area of ​​the two-dimensional bounding box is greater than a first threshold, and the ratio of the area of ​​the intersection to the area of ​​the projected bounding box is less than a second threshold; wherein the first threshold and the second threshold are preset positive numbers.

13. The apparatus according to any one of claims 10-12, further comprising: The segmentation point determination module is used to project the two-dimensional bounding box in the first image related to the three-dimensional bounding box onto the plane of the three-dimensional bounding box when it is determined that the three-dimensional bounding box can be segmented, so as to obtain the segmentation point. The segmentation module is used to segment the 3D bounding box using the segmentation points to obtain the segmented 3D bounding box.

14. The apparatus according to claim 13, wherein, The segmentation point determination module is used for: Acquire at least two first images associated with the 3D bounding box; Project the same pixels of the two-dimensional bounding boxes in at least two first images onto the plane of the three-dimensional bounding box to obtain at least two projection points; The cutting point is determined using the at least two projection points.

15. The apparatus according to claim 14, wherein, The segmentation point determination module is used for: For each of the at least two first images, using the camera parameters corresponding to the first image, the same pixels in the first image are projected onto the plane of the three-dimensional bounding box.

16. The apparatus according to any one of claims 13-15, further comprising: The attribute assignment module is used to assign values ​​to the attributes of the segmented 3D bounding box using the attribute information corresponding to the 2D bounding box in the first image related to the 3D bounding box.

17. The apparatus according to any one of claims 10-15, wherein, The bounding box determination module is used to project the 3D bounding box onto the first image using the camera parameters corresponding to the first image.

18. The apparatus according to claim 15 or 17, wherein, The camera parameters corresponding to the first image include: the camera parameters of the image acquisition device when capturing the first image, and the camera parameters include at least one of internal parameters and external parameters.

19. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

21. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images