Object singulation method, electronic device and three-dimensional scanning and reconstruction device

Abstract

This application relates to the field of three-dimensional reconstruction technology and discloses a method for object singulation, an electronic device, and a three-dimensional scanning and reconstruction device. The method includes: acquiring 3D point cloud data and multiple frames of images of a three-dimensional scene; constructing a voxel map based on the 3D point cloud data; performing target detection on each of the multiple frames of images, projecting the detected target objects onto the voxel map, and obtaining a voxel set for each target object; constructing an adjacency relationship between voxels based on the voxel map and the voxel set to generate a weighted undirected graph; and dividing the voxels in the weighted undirected graph into multiple clusters, wherein each cluster corresponds to a singulated target object. This application improves the accuracy of object singulation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of three-dimensional reconstruction technology, and specifically to an object singulation method, an electronic device, and a three-dimensional scanning and reconstruction device. Background Art

[0002] With the rapid development of autonomous driving, robotic navigation, smart security and other fields, three-dimensional object detection and segmentation technology has been widely used and has become a key research direction. Currently, most mainstream three-dimensional object detection and segmentation methods are based on single-frame images, and achieve the individualization of three-dimensional objects through object detection, mask segmentation, and 2D to 3D projection. However, when performing three-dimensional object detection based on single-frame images, due to its heavy reliance on single-frame images, its detection and projection results are easily affected by noise, lighting changes, and occlusion. The detection results are obtained from a top-down view, which is difficult to deal with the overlap and occlusion problems in complex scenes such as placing a computer on a desktop or a chair under the table, resulting in inaccurate detection results.

[0003] Related technologies also include multi-frame image fusion, voxel representation, and segmentation methods based on graph structures and deep learning. For example, methods based on multi-frame image and multimodal data fusion for 3D detection also require converting the data into a bird's-eye view first, and are unable to solve problems such as overlap and occlusion in complex scenes. Another example is voxel clustering and supervoxel clustering point cloud segmentation methods. These methods perform voxel division and supervoxel clustering on point cloud data, and iteratively merge them using the adjacency relationship between supervoxels to output the target point cloud area. This clustering method is relatively rough and only relies on the features of the point cloud network or the normal features of the point cloud. In scenes with dense targets and severe occlusion, the segmentation results are not accurate enough. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide an object singulation method, an electronic device, and a three-dimensional scanning and reconstruction device to solve the problem of inaccurate object detection in the prior art.

[0005] According to one aspect of an embodiment of the present application, a method for singulating an object is provided, the method comprising:

[0006] Acquire 3D point cloud data and multi-frame images of three-dimensional scenes;

[0007] constructing a voxel map based on the 3D point cloud data;

[0008] Performing target detection on each of the multiple frames of image, projecting the detected target objects onto the voxel map, and obtaining a voxel set of each target object;

[0009] Constructing an adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph;

[0010] The voxels in the weighted undirected graph are divided into a plurality of clusters, wherein each cluster corresponds to a singularized target object.

[0011] Optionally, dividing the voxels in the weighted undirected graph into a plurality of clusters comprises:

[0012] A community detection algorithm is used to partition the voxels in the weighted undirected graph into a plurality of clusters.

[0013] Optionally, performing target detection on the multiple frames of images respectively, projecting the detected target objects onto the voxel map, and obtaining a voxel set of each target object includes:

[0014] Perform target detection on each frame of image to obtain the 2D bounding box of the target object in each image;

[0015] Obtain a pixel-level mask of the target object for each 2D border area;

[0016] The pixel-level mask in each image is projected onto the voxel map according to the parameters of the camera that captured the image, so as to obtain a voxel set of each target object, wherein the parameters of the camera include intrinsic parameters and extrinsic parameters.

[0017] Optionally, after dividing the voxels in the weighted undirected graph into a plurality of clusters, the method further comprises:

[0018] Back-projecting the plurality of clusters onto the plurality of frames of image to obtain a back-projected 2D bounding box of each cluster in the image;

[0019] Determining whether the back-projected 2D bounding box matches at least one of the 2D bounding boxes;

[0020] If so, retain the cluster corresponding to the back-projected 2D bounding box;

[0021] If not, the cluster corresponding to the back-projected 2D bounding box is removed.

[0022] Optionally, determining whether the back-projected 2D bounding box matches at least one of the 2D bounding boxes includes:

[0023] Calculate the IoU between the back-projected 2D bounding box and each of the 2D bounding boxes respectively;

[0024] If the IoU between the back-projected 2D bounding box and a certain 2D bounding box is greater than a preset threshold, determine that the back-projected 2D bounding box matches the 2D bounding box;

[0025] If the IoU between the back-projected 2D bounding box and a certain 2D bounding box is less than or equal to a preset threshold, it is determined that the back-projected 2D bounding box does not match the 2D bounding box.

[0026] Optionally, back-projecting the plurality of clusters onto the plurality of frames of images to obtain a back-projected 2D bounding box of each cluster in the image includes:

[0027] Back-project each cluster onto each frame according to the extrinsic parameters of the camera that captured each image;

[0028] The minimum enclosing rectangle of each cluster in multiple back projections on the multiple frames of images is calculated to obtain the back-projected 2D bounding box of the cluster.

[0029] Optionally, constructing an adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph includes:

[0030] Add edges between any two voxels in each voxel set corresponding to each target object, and add a fixed weight to each edge;

[0031] Add up the weights of the same edges.

[0032] Optionally, the 3D point cloud data is 3D point cloud data obtained by three-dimensionally reconstructing laser point cloud data and the multiple frames of images, and the multiple frames of images include images from different perspectives.

[0033] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the object singulation method as described above.

[0034] According to another aspect of an embodiment of the present application, a three-dimensional scanning and reconstruction device is provided, the three-dimensional scanning and reconstruction device comprising:

[0035] LiDAR, used to collect laser point cloud data of three-dimensional scenes;

[0036] A camera, configured to acquire multiple frames of images of the three-dimensional scene;

[0037] A computing unit is configured to acquire the laser point cloud data and the multiple frames of images, perform three-dimensional reconstruction on the laser point cloud data and the multiple frames of images to obtain 3D point cloud data, and execute the object singulation method as described above.

[0038] The embodiment of the present application constructs a voxel map based on 3D point cloud data of a three-dimensional scene, and detects target objects from multiple frames of images of the three-dimensional scene, projects the detected target objects onto the voxel map, and obtains a voxel set of each target object. The adjacency relationship between voxels is constructed based on the voxel map and the voxel set, and a weighted undirected graph is generated. Since the weights of the edges between voxels of the same target object in the weighted undirected graph are the same, voxel clustering can be performed based on the weighted undirected graph to obtain multiple clusters, each cluster corresponding to a single target object, thereby achieving object singulation. By fusing multi-frame image information, the error and noise effects caused by single-frame detection are reduced, making the detection results more stable; a detailed three-dimensional scene map is established using voxel representation, overcoming the limitations of detection in complex occlusion scenes caused by only using a top-down view, thereby improving detection accuracy; by constructing a weighted undirected graph, clusters are divided based on the weights of the edges between voxels, achieving efficient segmentation and fine clustering of targets, and accurately separating densely arranged and partially occluded objects.

[0039] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the embodiments of the present application, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present application. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0041] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown;

[0042] Figure 2 A flowchart of the object singulation method provided in an embodiment of the present application is shown;

[0043] Figure 3 A flowchart of another object singulation method provided in an embodiment of the present application is shown;

[0044] Figure 4 A schematic structural diagram of an object singulation device provided in an embodiment of the present application is shown;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0046] Figure 6 A schematic structural diagram of a three-dimensional scanning and reconstruction device provided in an embodiment of the present application is shown;

[0047] Figure 7A diagram showing the object singulation effect of an embodiment of the present application is shown;

[0048] Figure 8 Another object singulation effect diagram of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0050] With the rapid development of autonomous driving, robotic navigation, smart security and other fields, three-dimensional object detection and segmentation technology has been widely used and has become a key research direction. Currently, most mainstream three-dimensional object detection and segmentation methods are based on single-frame images, and achieve the individualization of three-dimensional objects through object detection, mask segmentation, and 2D to 3D projection. However, when performing three-dimensional object detection based on single-frame images, due to its heavy reliance on single-frame images, its detection and projection results are easily affected by noise, lighting changes, and occlusion. The detection results are obtained from a top-down view, which is difficult to deal with the overlap and occlusion problems in complex scenes such as placing a computer on a desktop or a chair under the table, resulting in inaccurate detection results.

[0051] In this application, 2D (two-dimensional) refers to a two-dimensional plane with length and width, which is used to describe the image plane content of objects and scenes. 3D (three-dimensional) refers to a three-dimensional space with length, width, and height, which is used to describe the three-dimensional geometry of objects and scenes.

[0052] Related technologies also include multi-frame image fusion, voxel representation, and segmentation methods based on graph structures and deep learning. For example, methods based on multi-frame image and multimodal data fusion for 3D detection also require converting the data into a bird's-eye view, which cannot solve problems such as overlap and occlusion in complex scenes.

[0053] A voxel (Volume Element) is the smallest unit of volume in three-dimensional space, similar to a pixel in a two-dimensional image. Voxelization facilitates the structured representation of point cloud data. Voxel-based point cloud segmentation methods, such as voxel clustering and supervoxel clustering, are used. These methods perform voxel-based segmentation and supervoxel clustering on the point cloud data, then iteratively merge the supervoxels using the adjacency relationship between them to output the target point cloud region. This clustering method is relatively crude, relying solely on the features of the point cloud network or the point cloud normal phase. Consequently, segmentation results may be inaccurate in scenes with densely populated objects and severe occlusion.

[0054] The present application provides a method for object singulation, which constructs a voxel map based on 3D point cloud data of a three-dimensional scene, detects target objects from multiple frames of the three-dimensional scene, and projects the detected target objects onto the voxel map to obtain a voxel set of each target object. The adjacency relationship between voxels is constructed based on the voxel map and the voxel set to generate a weighted undirected graph. Since the weights of the edges between voxels of the same target object in the weighted undirected graph are the same, voxel clustering can be performed based on the weighted undirected graph to obtain multiple clusters, each cluster corresponding to a singulated target object, thereby achieving object singulation. By fusing multi-frame image information, the error and noise effects caused by single-frame detection are reduced, making the detection results more stable; a detailed three-dimensional scene map is established using voxel representation, overcoming the limitations of detection only through top-view in complex occlusion scenes, thereby improving detection accuracy; by constructing a weighted undirected graph, clusters are divided based on the weights of the edges between voxels, achieving efficient segmentation and fine clustering of targets, and accurately separating densely arranged and partially occluded objects.

[0055] Figure 1 A schematic diagram of an application scenario of an embodiment of the present application is shown. As shown in the figure, in a three-dimensional scene, the three-dimensional scene is scanned by a three-dimensional scanning and reconstruction device 1 to obtain point cloud data and image data. For example, the three-dimensional scanning and reconstruction device 1 can be provided with a laser radar, a camera, and a computing unit, wherein the laser radar collects laser point cloud data of the three-dimensional scene, the camera obtains images of the three-dimensional scene from different perspectives, and the computing unit fuses the laser point cloud data and the image to achieve three-dimensional reconstruction. The three-dimensional scanning and reconstruction device 1 can be a handheld device or a non-handheld device. If the three-dimensional scanning and reconstruction device 1 is a handheld device, the user holds the three-dimensional scanning and reconstruction device 1 and moves it in the three-dimensional scene to collect point cloud data and image data of the three-dimensional scene. If the three-dimensional scanning and reconstruction device 1 is a non-handheld device, the three-dimensional scanning and reconstruction device 1 can be moved autonomously in the three-dimensional scene to collect point cloud data and image data of the three-dimensional scene.

[0056] Terminal device 2 is communicatively connected to 3D scanning and reconstruction device 1, receiving and displaying the 3D reconstruction results obtained by 3D scanning and reconstruction device 1 after 3D reconstruction of the 3D scene. Alternatively, 3D scanning and reconstruction device 1 may simply collect raw point cloud data and image data and transmit them to terminal device 2, which then fuses the data for 3D reconstruction and displays the 3D reconstruction results. Terminal device 2 may be a mobile phone, tablet computer, laptop computer, desktop computer, server, or other device, with a mobile phone being used as an example in the figure. Terminal device 2 and 3D scanning and reconstruction device 1 may be connected via wired or wireless communication.

[0057] In other scenarios, the lidar and camera can also be separate devices and not integrated into the three-dimensional scanning and reconstruction device 1. They send the data collected by each of them to the terminal device 2, which performs data fusion processing to obtain and display the three-dimensional reconstruction results.

[0058] Figure 2 The flowchart of the object singulation method provided by the embodiment of the present application is shown, and the method is executed by an electronic device. The electronic device may be Figure 1 The three-dimensional scanning and reconstruction device 1 or terminal device 2 in the embodiment. Figure 2 As shown, the method includes the following steps:

[0059] S110 , acquiring 3D point cloud data and multiple frames of images of a three-dimensional scene.

[0060] In a 3D reconstruction scenario, laser point cloud data of the 3D scene can be collected using a lidar (lidar), and images of the 3D scene can be captured using a camera. A computing unit or terminal device then fuses the laser point cloud data and images to perform 3D reconstruction and generate 3D point cloud data. To obtain complete 3D information about the scene, a camera typically captures images of the scene from different perspectives, generating multiple frames. Images from different perspectives provide information from multiple aspects of the scene, helping to calculate the depth, shape, and spatial position of objects in the 3D scene, thereby determining the spatial position of the objects. Images from multiple perspectives also help to fill in details in occluded areas. Areas obscured from some perspectives may become visible from other perspectives, resulting in a more complete reconstruction of the scene. The camera can be a monocular camera, a binocular camera, a panoramic camera, or other similar cameras.

[0061] In addition to lidar, point cloud data can also be collected through other devices, such as structured light scanners, time-of-flight (ToF) sensors, ultrasonic sensors, stereo vision cameras (such as binocular cameras), etc.

[0062] S120: Construct a voxel map based on the 3D point cloud data.

[0063] This step discretizes the 3D point cloud data and simplifies it into a voxel map. A voxel map is a method for representing an environment in three-dimensional space, similar to the concept of pixels in two-dimensional images. A voxel map divides the three-dimensional space into multiple voxels, each of which is a small cubic area of ​​a fixed volume. It represents a specific area in space and can store various information about that area, such as the presence of objects and properties such as density.

[0064] When constructing a voxel map, the 3D scene (i.e., 3D space) containing the 3D point cloud data is first divided into voxels. Then, based on the 3D point cloud data, each voxel is determined to be occupied. If a point cloud data point exists within the voxel, the voxel is considered occupied; otherwise, it is considered empty.

[0065] When constructing a voxel map, an appropriate voxel size can be set to make the voxel resolution sufficiently refined to accommodate the fine segmentation of targets in complex scenes. The voxel size d matches the sampling density of the scene point cloud and the minimum geometric feature size s of the target. For example, the voxel size d is less than 1 / 2 of the minimum geometric feature size s. The minimum geometric feature size s is the minimum feature size of the target to be singulated (such as the smallest protrusion, gap, etc.). For fine indoor scenes, the voxel size d can range from 5mm to 20mm; for macro outdoor scenes, the voxel size d can range from 20mm to 100mm.

[0066] By simplifying complex 3D scene point cloud data into voxel maps, the data can be processed and analyzed more efficiently. At the same time, by setting the appropriate voxel size, sufficient details are guaranteed to accommodate the fine segmentation of targets in complex scenes.

[0067] S130 , performing target detection on the multiple frames of images respectively, projecting the detected target objects onto a voxel map, and obtaining a voxel set of each target object.

[0068] This step can use a known image object detection algorithm to detect the target and determine the location of the target object in the image. The image object detection algorithm can be a detector based on a deep neural network, and the detector network architecture can be R-CNN, Fast R-CNN, Faster R-CNN, YOLO series networks, etc.

[0069] The camera that captures the image has its own parameters, which include intrinsic parameters and extrinsic parameters. Among them, the intrinsic parameters are parameters that describe the internal imaging characteristics of the camera, such as focal length, principal point coordinates, lens distortion, etc., which are mainly related to the hardware structure of the camera itself and are inherent properties of the camera itself. The extrinsic parameters are parameters that describe the position and posture (i.e., posture information) of the camera in the world coordinate system, including the rotation matrix and translation vector, which reflect the external geometric relationship of the camera relative to the world coordinate system. Each frame of image captured by the camera contains the intrinsic and extrinsic parameters of the camera. Based on the intrinsic and extrinsic parameters of the camera, the detected target object can be projected into a voxel map, which is equivalent to projecting a two-dimensional target into 3D space to obtain a voxel set of each target object.

[0070] The same target object will appear in multiple frames of images at different perspectives. Therefore, after detecting and projecting the multiple frames of images, multiple voxel sets will be obtained for the same target object, that is, each target object corresponds to multiple voxel sets.

[0071] S140 , constructing adjacency relationships between voxels based on the voxel map and the voxel set to generate a weighted undirected graph.

[0072] Based on the voxel map and projection to obtain the voxel set of each target object, we can establish the connections (i.e., edges) between the voxels within each voxel set in the voxel map, and assign weights to each edge to obtain a weighted undirected graph. When clustering the target objects subsequently, we can quickly and accurately cluster them based on the weights of the edges within the weighted undirected graph.

[0073] Step S140 can be implemented by the following steps:

[0074] S141 , adding an edge between any two voxels in each voxel set corresponding to each target object, and adding a fixed weight to each edge.

[0075] S142, superimpose the weights of the same edges.

[0076] Since each target object corresponds to multiple voxel sets, after adding an edge between any two voxels in each voxel set corresponding to the same target object and increasing the fixed weight of the edge, edges will be added repeatedly between the same two voxels. This article refers to such edges as "identical edges." For identical edges, their weights are added together to obtain the total weight of the edge. For example, if target 1 is detected in all three image frames, three voxel sets corresponding to target 1 are obtained. Edge 1 between voxel 1 and voxel 2 is established for all three voxel sets, and a fixed weight X is added to edge 1 three times. The total weight of edge 1 in the resulting weighted undirected graph is 3X.

[0077] Due to viewing angle issues, the voxel set obtained by performing target detection and projection on a certain frame image may include voxels inside the real target object and voxels outside the real target object. Therefore, the edges added between voxels in the voxel set include both edges between voxels inside the real target object and edges between voxels inside the real target object and voxels outside the real target object. However, since step S130 performs target detection and projection on multiple frames of images, the voxels in the voxel set obtained based on other frame images are usually voxels inside the real target object. The weighted undirected graph combines the edges and weights obtained based on all frame images, that is, the weight of each edge in the final weighted undirected graph is the total weight after the weights are superimposed. Therefore, the total weight of the edges between voxels inside the real target object in the weighted undirected graph will be higher than the total weight of the edges between voxels inside the real target object and voxels outside the real target object. For example, in a 3D scene, there is a chair and a table next to it. The chair is detected in all three frames. The voxels in the voxel set derived from detection and projection in two frames (images 1 and 2) are all actual chair voxels. However, due to perspective, the voxel set derived from detection and projection in the other frame (image 3) includes not only actual chair voxels but also table voxels. When adding edges to voxels within the voxel set, the edges derived from images 1 and 2 are all edges between voxels within the actual chair, while the edges derived from image 3 include not only edges between voxels within the actual chair (such edges are called first-class edges) but also edges between voxels in the chair and voxels on the table (such edges are called second-class edges). If a fixed weight X is added to each edge, the total weight of the first-class edges is 3X, and the total weight of the second-class edges is X, based on the results from the three frames. Therefore, the total weight of the first-class edges is higher than that of the second-class edges. Finally, when performing voxel clustering based on the weighted undirected graph, only voxels connected by the first-class edges are clustered together. For a target detected in an image, the edges between any two voxels in its corresponding voxel set are assigned the same fixed weight to form a local connection relationship within the target, strengthening the correlation between voxels within the target, and making the voxels of the same target more closely connected in the undirected graph. Since the voxels in a single voxel set belong to the same target and have the same importance or degree of correlation, when constructing the internal connection relationship of the target object, no matter where the two voxels are in the target, as long as they belong to the voxel set corresponding to the same target, they will be assigned the same fixed weight to strengthen the connection between them.

[0078] In the above manner, detection is performed based on multiple frames of images, and edges are added between voxels in each voxel set and weights are increased for the edges. Even though detection errors are sometimes inevitable, resulting in added weights for voxels within the real object and voxels outside the real object, the weighted undirected graph finally obtained is the total weight after weight aggregation. Therefore, the weight of the edge between voxels within the real object is ultimately higher than the weight of the edge between voxels outside the real object, thereby improving the accuracy of clustering.

[0079] In addition, only adding edges between voxels within a voxel set and increasing fixed weights can also simplify calculations and improve efficiency.

[0080] S150 , dividing the voxels in the weighted undirected graph into a plurality of clusters, wherein each cluster corresponds to a single target object.

[0081] This step segments and clusters the target in the weighted undirected graph, resulting in one or more individual target objects. Community detection algorithms (such as the Louvain algorithm) can be used to partition the voxels in the weighted undirected graph into multiple clusters. Community detection algorithms are graph-theoretic clustering methods used to identify densely connected subregions within a graph structure.

[0082] Specifically, the community detection algorithm determines the degree of voxel aggregation based on the connection weights between voxels in a weighted undirected graph (that is, the weights of the edges). Since the connection weights of voxels within the same target object are higher and the connection weights between voxels of different target objects are smaller, the community detection algorithm can automatically cluster voxels into multiple objects based on the connection weights.

[0083] If the same object is detected in multiple frames, multiple voxel sets of the object will be obtained, but such multiple voxel sets are usually adjacent or close in 3D space. The above algorithm will divide such multiple voxel sets into a cluster. For example, for two frames of images with different perspectives, the same chair is detected in these two frames, corresponding to two voxel sets, then the two voxel sets are likely to overlap (for example, 90%), and the voxels in the two voxel sets will be clustered into one cluster. For two frames of images with different perspectives, a chair is detected in each of these two frames, and these two chairs are different chairs, corresponding to two voxel sets, then the two voxel sets are likely to have no intersection and will not be clustered into one cluster. The graph-based approach can merge the voxel sets of the same object in multiple frames into one, making full use of the information of multiple frames, thereby achieving complete and accurate segmentation of the target object in the scene, and improving the accuracy and completeness of target segmentation.

[0084] Community detection algorithms can solve the problem of object singulation in complex scenarios. Other clustering algorithms can also be used to achieve semantic singulation of target objects, such as traditional region growing or density-based clustering algorithms (such as DBSCAN), but this application does not limit these methods.

[0085] Figure 3 FIG. 1 shows a flow chart of another object singulation method provided by an embodiment of the present application, which is executed by an electronic device. Figure 3 As shown, the method includes the following steps:

[0086] S210: Acquire 3D point cloud data and multiple frames of images of a three-dimensional scene.

[0087] S220: Construct a voxel map based on the 3D point cloud data.

[0088] Steps S210 and S220 are similar to the aforementioned steps S110 and S120 and will not be repeated here.

[0089] S231 , performing target detection on each frame of image to obtain a 2D bounding box of the target object in each image.

[0090] After detecting objects in each image frame using an image object detection algorithm, the detected object's category and the object's 2D bounding box on the image plane are obtained. A 2D bounding box is typically represented by four coordinates, such as (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the bounding box's top-left corner and (x2, y2) represents the coordinates of its bottom-right corner. Based on these coordinates, a 2D bounding box can be displayed in the image.

[0091] S232: Obtain a pixel-level mask of the target object for each 2D border area.

[0092] The 2D bounding box only represents the location of the target object and is coarse-grained positioning information. To more accurately segment the boundary and shape of the target object, the 2D bounding box can be further used to determine the exact pixel range of the target object, resulting in a fine-grained segmentation result called a pixel-level mask of the target object.

[0093] This step uses a semantic segmentation algorithm to obtain a pixel-level mask of the target object. The semantic segmentation algorithm can classify each pixel in the image as belonging to a specific category, such as a person, a car, a dog, etc. For a target object whose 2D bounding box has been obtained by an image target detection algorithm, the result of semantic segmentation can be used to obtain a pixel-level mask of the target object. For example, in the scenario of a city street scene image, the 2D bounding box of the car is first obtained by an image target detection algorithm. Then, the semantic segmentation algorithm is used to perform pixel-level classification on the entire image and mark out the pixels belonging to the car category. Finally, combined with the 2D bounding box of the car obtained by target detection, the car-type pixels located within the 2D bounding box in the semantic segmentation result are extracted to obtain a pixel-level mask of the car. A pixel value of 1 in the mask indicates that it belongs to the target object, and 0 indicates background. The semantic segmentation algorithm can provide comprehensive pixel-level classification information, and the segmentation results are highly accurate.

[0094] In some embodiments, the target detection of step S231 and the pixel-level segmentation of step S232 can also be performed simultaneously by an instance segmentation algorithm, and an algorithm can be used to output not only the 2D bounding box of each target object, but also the pixel-level mask of each target object directly. For example, the Mask R-CNN algorithm is used, which is improved on the basis of Faster R-CNN and adds a parallel branch for predicting the segmentation mask corresponding to each target instance. The algorithm first uses a region proposal network (RPN) to generate candidate target regions (similar to candidate regions of 2D bounding boxes), and then processes the feature map through the ROI (Region of Interest) Align layer to avoid the loss of pixel information caused by the pooling operation during the feature extraction process. Finally, a special mask prediction branch is used to predict the segmentation mask of each target instance, thereby directly obtaining the pixel-level mask of the target object.

[0095] The above is a method for obtaining a pixel-level mask based on a segmentation algorithm. In other embodiments, a pixel-level mask may also be obtained by a method based on boundary extension and thinning or other methods.

[0096] S233 , projecting the pixel-level mask in each image onto a voxel map according to the parameters of the camera that captured the image, to obtain a voxel set of each target object, wherein the parameters of the camera include intrinsic parameters and extrinsic parameters.

[0097] This step converts the pixel-level mask information extracted from the image into a set of voxels in 3D space, providing a data basis for the subsequent construction of a weighted undirected graph.

[0098] During the projection process, the pixel coordinates in the target object's pixel-level mask are first converted to the normalized camera coordinate system. For example, for each pixel point (u, v) in the target object's pixel-level mask, the camera intrinsic parameters and the depth information of the point are used to convert it to the camera coordinate system's coordinates (x, y, z). The depth information of the pixel point is provided by the lidar. Then, based on the camera's extrinsic parameters, the coordinates of each pixel point in the camera coordinate system are converted to the world coordinate system (the voxel map's coordinate system is consistent with the world coordinate system), obtaining the pixel point's world coordinate coordinates (X, Y, Z). Finally, these coordinates are voxelized. Based on the voxel map's origin and voxel size, the pixel point's world coordinate coordinates (X, Y, Z) are converted to voxel coordinates (i, j, k). In this way, the pixels in each target object's pixel-level mask are traversed to obtain the corresponding voxel coordinates. By collecting the voxel coordinates of each target object's pixel point, the voxel set in the voxel map is formed. For example, after the above processing, the voxel coordinates of a target object may be (15, 23, 37), (16, 23, 37), (15, 24, 37), etc. These voxel coordinates constitute the voxel set of the target object.

[0099] S240 , constructing an adjacency relationship between voxels based on the voxel map and the voxel set to generate a weighted undirected graph.

[0100] S250 , dividing the voxels in the weighted undirected graph into a plurality of clusters, wherein each cluster corresponds to a single target object.

[0101] Steps S240 and S250 are similar to the aforementioned steps S140 and S150 and will not be repeated here.

[0102] S260 , back-projecting the multiple clusters onto multiple frames of images to obtain a back-projected 2D bounding box of each cluster in the image.

[0103] This step includes the following steps:

[0104] S2611 , back-projecting each cluster onto each frame of image based on the extrinsic parameters of the camera that captured each image.

[0105] The back-projection process is the opposite of the projection process in step S233. Each cluster includes multiple voxels, and the voxel coordinates (i, j, k) of each voxel can be converted to world coordinates (X, Y, Z) using the voxel size and the voxel map origin, for example:

[0106] X=origin x +i×voxel size

[0107] Y=origin y +j×voxel size

[0108] Z=origin z +k×voxel size

[0109] Among them, origin x 、origin y 、origin z is the coordinate of the origin of the voxel map, voxel size is the size of the voxel.

[0110] For each voxel in the cluster, the world coordinates (X, Y, Z) can be converted to the camera coordinate system using the camera's external parameters. The conversion formula is:

[0111] X camera =R -1 ×(X world -t)

[0112] Among them, X camera Is the coordinate of the point in the camera coordinate system after coordinate transformation, X world is the point coordinate in the world coordinate system, R is the rotation matrix, R -1 It is the inverse of the rotation matrix R, that is, the transposed matrix R T , t is the translation vector. In the camera coordinate system, the coordinates of a point can be expressed as (x, y, z). To project it onto the image plane, the camera's intrinsic parameters can be used for projection. The camera's intrinsic parameters include the focal length (f), the principal point coordinates (cx, cy), etc. The projection formula is as follows:

[0113] u=f×x / z+cx

[0114] v=f×y / z+cy

[0115] Among them, (u, v) is the pixel coordinate of the point after being projected onto the image plane.

[0116] Each voxel in each cluster is projected onto the image captured by the corresponding camera according to the above steps to obtain the back projection of each cluster on each frame of the image.

[0117] S2612 , calculating the minimum enclosing rectangle of each cluster in multiple back-projections on multiple frames of images to obtain the back-projected 2D bounding box of the cluster.

[0118] In multiple frames, the back-projected pixels of each cluster may be distributed in different locations. In order to obtain a compact 2D representation of the cluster in the image, the minimum bounding rectangle of each cluster in multiple back-projections on multiple frames can be calculated to obtain the back-projected 2D bounding box of the cluster.

[0119] S270 , determining whether the back-projected 2D bounding box matches at least one 2D bounding box; if so, executing step S280 ; otherwise, executing step S290 .

[0120] In this step, the back-projected 2D bounding box obtained by back-projection is matched with the multiple 2D bounding boxes obtained in the aforementioned step S231 , thereby verifying whether the individualized cluster obtained in step S250 accurately corresponds to the real target object.

[0121] When determining whether the back-projected 2D bounding box matches at least one 2D bounding box, the IoU (Intersection over Union) between the back-projected 2D bounding box and each 2D bounding box can be calculated separately. IoU is an indicator that measures the degree of overlap between two areas. It is calculated by dividing the area of ​​the intersection of the two areas by the area of ​​the union, and is used to verify and match the detection results. The method of calculating the IoU between the back-projected 2D bounding box and each 2D bounding box is to calculate the ratio of the area of ​​the overlapping part of the back-projected 2D bounding box and the 2D bounding box to the area of ​​the union of the two areas. If the IoU between the back-projected 2D bounding box and a certain 2D bounding box is greater than a preset threshold, it is determined that the back-projected 2D bounding box matches the 2D bounding box; if the IoU between the back-projected 2D bounding box and a certain 2D bounding box is less than or equal to the preset threshold, it is determined that the back-projected 2D bounding box does not match the 2D bounding box.

[0122] Generally, the larger the IoU value, the more overlapped the back-projected 2D bounding box and the real target object’s 2D bounding box is, and the cluster corresponding to the back-projected 2D bounding box is the real target object, with a good individualization effect. If the IoU value is low, it means that the position or size of the back-projected 2D bounding box is significantly different from the real target object’s 2D bounding box, and the cluster corresponding to the back-projected 2D bounding box is not the real target object.

[0123] Regarding the preset threshold, when the preset threshold is set to 0.3~0.4, loose verification is achieved, allowing for some projection deviations or small detection boxes to avoid misjudging the target as a false detection. In most cases, the preset threshold can be set to 0.5, taking into account both false detection filtering and true target retention. When the preset threshold is set to 0.6~0.7, strict verification is achieved, false detection filtering is strengthened, and only clusters whose back-projection boundaries and detection boxes are highly overlapped are retained. In some embodiments, three preset thresholds can be provided, and users can select the most appropriate preset threshold based on business needs.

[0124] S280, retain the cluster corresponding to the back-projected 2D bounding box.

[0125] If the back-projected 2D bounding box matches at least one 2D bounding box, it means that the cluster corresponding to the back-projected 2D bounding box corresponds to the real target object, and the cluster is retained.

[0126] S290: Remove the cluster corresponding to the back-projected 2D border.

[0127] If no bounding box matching the back-projected 2D bounding box is found among all 2D bounding boxes, it means that the cluster corresponding to the back-projected 2D bounding box is a false detection and is not a real target object, and the cluster is removed.

[0128] By back-projecting each cluster onto each frame image in the above manner, the IoU between it and the 2D bounding box obtained by the original detection is calculated to verify whether each cluster accurately corresponds to the real detection target, filter out the clusters that are falsely detected, ensure that the final individualized clusters have high accuracy, and improve the robustness of the method of the embodiment of this application. In general, using the IoU matching between the 2D detection and 3D back-projection results as an auxiliary verification mechanism can effectively filter out false detections and noisy point clouds, and improve the credibility of the final individualized results.

[0129] The embodiment of the present application detects targets based on multi-frame images through voxelization, projects the targets identified based on multi-frame images onto a voxel map, and constructs the adjacency relationship between voxels within the same target object in order to cluster and obtain clusters representing single target objects. The above process makes full use of the information of the entire three-dimensional scene, rather than relying solely on a certain perspective (such as a bird's-eye view), which can capture the structure and relationship of objects more comprehensively and avoid perspective dependence problems. If it is based only on a certain perspective (such as a bird's-eye view), the connection relationship of the object in other directions may be ignored, resulting in missegmentation.

[0130] Suppose in a scene, there is a cup and a school bag on the table. The cup and the school bag are separated in three-dimensional space, but from a bird's-eye view, part of the school bag blocks part of the cup, making some voxels of the cup appear to be separated.

[0131] Traditional segmentation methods might mistakenly interpret the cup body and handle as two different objects because they appear separate in a top-down view. However, in this embodiment, for a single target (e.g., a cup), edges are added between voxels within it and assigned high weights. A community detection algorithm analyzes the connectivity (connection weights) between voxels, and the algorithm prioritizes clustering voxels with high connection weights into a single group.

[0132] The voxels of the cup body and the voxels of the cup handle are adjacent in three-dimensional space. The edge between the cup body and the cup handle is assigned a high weight (for example, 1) because they belong to the same target (the cup). The edge between the cup body and the backpack is assigned a low weight because they are not the same target. Therefore, even if the cup handle is blocked by the backpack from a bird's-eye view, the cup handle and the cup body are still adjacent in three-dimensional space, so the weight between them is high. The community detection algorithm prioritizes high-weight connections to ensure that the cup body and the cup handle are clustered as a whole. The connection weight between the backpack and the cup is low, and the algorithm will not mistakenly regard them as a whole.

[0133] For example, a chair is tucked under a table. If only a traditional algorithm is used to detect the voxels under a top-down view, the voxels of the table and chair appear to belong to the table object. However, the method of the embodiment of the present application can be used to divide the voxels in this area into two clusters (i.e., two sub-regions) through multiple frame perspectives, such as a front view or an oblique view, representing the table and the chair respectively.

[0134] Figure 4 FIG. 1 shows a schematic diagram of the structure of the object singulation device provided in an embodiment of the present application. Figure 4 As shown, the object singulation device 400 includes:

[0135] An acquisition module 410 is configured to acquire 3D point cloud data and multiple frames of images of a three-dimensional scene;

[0136] A construction module 420 is used to construct a voxel map based on the 3D point cloud data;

[0137] The detection module 430 is used to perform target detection on multiple frames of images, project the detected target objects onto a voxel map, and obtain a voxel set of each target object;

[0138] A generating module 440 is configured to construct an adjacency relationship between voxels based on the voxel map and the voxel set to generate a weighted undirected graph;

[0139] The partitioning module 450 is configured to partition the voxels in the weighted undirected graph into a plurality of clusters, wherein each cluster corresponds to a singularized target object.

[0140] The object singulation device 400 of the embodiment of the present application also includes other modules for executing the steps of the above method embodiment, which will not be described one by one here.

[0141] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0142] like Figure 5As shown, the electronic device 300 may include a processor 302 and a memory 304 .

[0143] The memory 304 is used to store a computer program 306. The memory 304 may include a high-speed RAM memory, or may also include a non-volatile memory, such as at least one disk memory. The computer program 306 may include computer-executable instructions.

[0144] The processor 302 is configured to execute the computer program 306 to implement the above-mentioned object singulation method embodiment.

[0145] The processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the electronic device 300 may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0146] Figure 6 The following is a schematic diagram showing the structure of a three-dimensional scanning and reconstruction device provided in an embodiment of the present application. The three-dimensional scanning and reconstruction device 1 includes:

[0147] LiDAR 10, used to collect laser point cloud data of three-dimensional scenes;

[0148] Camera 20, used to acquire multiple frames of images of a three-dimensional scene;

[0149] The computing unit 30 is configured to acquire laser point cloud data and multiple image frames, perform three-dimensional reconstruction of the laser point cloud data and multiple image frames to obtain 3D point cloud data, and execute the object singulation method of the above embodiment. The computing unit 30 can be a small computer or controller, such as a single-chip microcomputer, a microcontroller unit (MCU), a programmable logic controller (PLC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a system on a chip (SoC), etc.

[0150] An embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the embodiment of the object singulation method is implemented.

[0151] An embodiment of the present application provides a computer program that can be executed by a processor to implement the above-mentioned object singulation method embodiment.

[0152] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned object singulation method embodiment.

[0153] In the several embodiments provided in this application, if any function is implemented in the form of a software function module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of this application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server or other electronic device) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store computer program code.

[0154] Figure 7 The object singulation effect diagram of the embodiment of the present application is shown. Figure 8 Another object singulation effect diagram of an embodiment of the present application is shown. As can be seen from the diagram, the embodiment of the present application can accurately segment multiple objects in a three-dimensional scene.

[0155] The algorithm or demonstration provided here are not inherently relevant to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the present application.

[0156] It should be noted that the above embodiments illustrate rather than limit the present application, and that a person skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In claims that list several means, several units or modules of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

[0157] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for singulating an object, characterized in that: The method comprises: Acquire 3D point cloud data and multi-frame images of three-dimensional scenes; constructing a voxel map based on the 3D point cloud data; Performing target detection on each of the multiple image frames, projecting the detected target objects onto the voxel map to obtain a voxel set for each target object, wherein if the same target object is detected in multiple image frames, the target object corresponds to multiple voxel sets; Constructing an adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph; Dividing the voxels in the weighted undirected graph into a plurality of clusters, wherein each cluster corresponds to a singularized target object, and voxels in a plurality of voxel sets corresponding to the same target object are divided into one cluster; The performing target detection on the multiple frames of images respectively, projecting the detected target objects onto the voxel map, and obtaining a voxel set of each target object, includes: Perform target detection on each frame of image to obtain the 2D bounding box of the target object in each image; Obtain a pixel-level mask of the target object for each 2D border area; Projecting the pixel-level mask in each image onto the voxel map according to the parameters of the camera that captured the image, to obtain a voxel set of each target object, wherein the parameters of the camera include intrinsic parameters and extrinsic parameters; After dividing the voxels in the weighted undirected graph into a plurality of clusters, the method further includes: Back-projecting the plurality of clusters onto the plurality of frames of image to obtain a back-projected 2D bounding box of each cluster in the image; Determining whether the back-projected 2D bounding box matches at least one of the 2D bounding boxes; If so, retain the cluster corresponding to the back-projected 2D bounding box; If not, the cluster corresponding to the back-projected 2D bounding box is removed.

2. The method according to claim 1, characterized in that The step of dividing the voxels in the weighted undirected graph into a plurality of clusters comprises: A community detection algorithm is used to partition the voxels in the weighted undirected graph into a plurality of clusters.

3. The method according to claim 1, characterized in that The determining whether the back-projected 2D bounding box matches at least one of the 2D bounding boxes includes: Calculate the IoU between the back-projected 2D bounding box and each of the 2D bounding boxes respectively; If the IoU between the back-projected 2D bounding box and a certain 2D bounding box is greater than a preset threshold, determine that the back-projected 2D bounding box matches the 2D bounding box; If the IoU between the back-projected 2D bounding box and a certain 2D bounding box is less than or equal to a preset threshold, it is determined that the back-projected 2D bounding box does not match the 2D bounding box.

4. The method according to claim 1, wherein Back-projecting the plurality of clusters onto the plurality of frames of images to obtain a back-projected 2D bounding box of each cluster in the image includes: Back-project each cluster onto each frame according to the extrinsic parameters of the camera that captured each image; The minimum enclosing rectangle of each cluster in multiple back projections on the multiple frames of images is calculated to obtain the back-projected 2D bounding box of the cluster.

5. The method according to claim 1, wherein The step of constructing an adjacency relationship between voxels based on the voxel map and the voxel set to generate a weighted undirected graph includes: Add edges between any two voxels in each voxel set corresponding to each target object, and add a fixed weight to each edge; Add up the weights of the same edges.

6. The method according to claim 1, characterized in that The 3D point cloud data is 3D point cloud data obtained by three-dimensionally reconstructing the laser point cloud data and the multiple frames of images, and the multiple frames of images include images from different perspectives.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the object singulation method according to any one of claims 1 to 6.

8. A three-dimensional scanning and reconstruction device, characterized in that: The three-dimensional scanning and reconstruction device includes: LiDAR, used to collect laser point cloud data of three-dimensional scenes; A camera, configured to acquire multiple frames of images of the three-dimensional scene; A computing unit, configured to acquire the laser point cloud data and the multiple frames of images, perform three-dimensional reconstruction on the laser point cloud data and the multiple frames of images to obtain 3D point cloud data, and execute the object singulation method according to any one of claims 1 to 6.

Images