Object monomer method, electronic equipment and three-dimensional scanning reconstruction equipment

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and discloses an object monomer method, electronic equipment and three-dimensional scanning reconstruction equipment. The method comprises the following steps: acquiring 3D point cloud data and multi-frame images of a three-dimensional scene; constructing a voxel map according to the 3D point cloud data; target detection is carried out on the multiple frames of images, detected target objects are projected to a voxel map, and a voxel set of each target object is obtained; constructing an adjacency relation between voxels according to the voxel map and the voxel set, and generating a weighted undirected graph; voxels in the weighted undirected graph are divided into a plurality of clusters, and each cluster corresponds to one monomeric target object. According to the invention, the object monomer accuracy is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of three-dimensional reconstruction, and in particular, to a method for object monomerization, an electronic device, and a three-dimensional scanning and reconstruction device. Background Art

[0002] With the rapid development of fields such as autonomous driving, robot navigation, and intelligent security, three-dimensional object detection and segmentation technologies have been widely applied and become a key research direction. Currently, most mainstream three-dimensional object detection and segmentation methods are based on single-frame images, and the monomerization of three-dimensional objects is achieved through object detection, mask segmentation, and 2D to 3D projection. However, when performing three-dimensional object detection based on a single-frame image, due to the heavy dependence on the single-frame image, its detection and projection results are easily affected by noise, illumination changes, and occlusion interference; the detection result is the detection result obtained using the top view, and it is difficult to handle the overlapping and occlusion problems in complex scenarios such as a computer placed on a table or a chair blocked under the table, resulting in inaccurate detection results.

[0003] In related technologies, there have also emerged multi-frame image fusion, voxel representation, and segmentation methods based on graph structures and deep learning. For example, a method for 3D detection based on the fusion of multi-frame images and multi-modal data. Such methods also need to first convert the data into a top view and cannot solve the overlapping, occlusion, and other problems in complex scenarios. Another example is the voxel clustering and supervoxel clustering point cloud segmentation methods. These methods perform voxel division and supervoxel clustering on the point cloud data, and use the adjacency relationship between supervoxels for iterative merging to output the target point cloud region. This clustering method is relatively rough and only relies on the features of the point cloud network or the phase features of the point cloud. In a scene where the targets are dense and severely occluded, the segmentation result is not accurate enough. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application provide a method for object monomerization, an electronic device, and a three-dimensional scanning and reconstruction device, which are used to solve the problem of inaccurate object detection existing in the prior art.

[0005] According to one aspect of the embodiments of the present application, a method for object monomerization is provided. The method includes: Obtaining 3D point cloud data and multiple frames of images of a three-dimensional scene; Constructing a voxel map according to the 3D point cloud data; Performing object detection on each of the multiple frames of images, and projecting the detected target objects onto the voxel map to obtain a voxel set of each target object; Constructing an adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph; Dividing the voxels in the weighted undirected graph into multiple clusters, where each cluster corresponds to a monomerized target object.

[0006] Optionally, dividing the voxels in the weighted undirected graph into multiple clusters includes: Dividing the voxels in the weighted undirected graph into multiple clusters using a community detection algorithm.

[0007] Optionally, performing object detection on each of the multiple frames of images, and projecting the detected target objects onto the voxel map to obtain a voxel set for each target object, includes: Performing object detection on each frame of image to obtain 2D bounding boxes of target objects in each image; Obtaining a pixel-level mask of the target object for each 2D bounding box region; Projecting the pixel-level masks in each image onto the voxel map according to the parameters of the camera that captured the image to obtain a voxel set for each target object, where the parameters of the camera include internal parameters and external parameters.

[0008] Optionally, after dividing the voxels in the weighted undirected graph into multiple clusters, the method further includes: Back-projecting the multiple clusters onto the multiple frames of images to obtain back-projected 2D bounding boxes of each cluster in the images; Determining whether the back-projected 2D bounding boxes match at least one of the 2D bounding boxes; If so, retaining the cluster corresponding to the back-projected 2D bounding box; If not, removing the cluster corresponding to the back-projected 2D bounding box.

[0009] Optionally, determining whether the back-projected 2D bounding boxes match at least one of the 2D bounding boxes includes: Calculating the IoU of the back-projected 2D bounding box and each of the 2D bounding boxes respectively; If the IoU of the back-projected 2D bounding box and a certain 2D bounding box is greater than a preset threshold, determining that the back-projected 2D bounding box matches the 2D bounding box; If the IoU of the back-projected 2D bounding box and a certain 2D bounding box is less than or equal to the preset threshold, determining that the back-projected 2D bounding box does not match the 2D bounding box.

[0010] Optionally, back-projecting the multiple clusters onto the multiple frames of images to obtain back-projected 2D bounding boxes of each cluster in the images includes: Back-projecting each cluster onto each frame of image according to the external parameters of the camera that captured each image; Calculating the minimum bounding rectangle of the multiple back-projections of each cluster on the multiple frames of images to obtain the back-projected 2D bounding box of the cluster.

[0011] Optionally, constructing the adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph includes: Adding an edge between any two voxels in each voxel set corresponding to each target object, and adding a fixed weight to each edge; Superimposing the weights of the same edges.

[0012] Optionally, the 3D point cloud data is 3D point cloud data obtained by three-dimensional reconstruction of the lidar point cloud data and the multi-frame images, and the multi-frame images include images from different perspectives.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the object monomerization method as described above.

[0014] According to another aspect of the embodiments of the present application, a three-dimensional scanning and reconstruction device is provided, and the three-dimensional scanning and reconstruction device includes: A lidar for collecting lidar point cloud data of a three-dimensional scene; A camera for acquiring multi-frame images of the three-dimensional scene; A calculation unit for acquiring the lidar point cloud data and the multi-frame images, performing three-dimensional reconstruction on the lidar point cloud data and the multi-frame images to obtain 3D point cloud data, and executing the object monomerization method as described above.

[0015] In the embodiments of the present application, a voxel map is constructed according to the 3D point cloud data of the three-dimensional scene, and target objects are detected from multi-frame images of the three-dimensional scene, and the detected target objects are projected onto the voxel map to obtain a voxel set of each target object. The adjacency relationship between voxels is constructed according to the voxel map and the voxel set to generate a weighted undirected graph. Since the weights of the edges between the voxels of the same target object in the weighted undirected graph are the same, voxel clustering can be performed based on the weighted undirected graph to obtain multiple clusters, and each cluster corresponds to a monomerized target object, thereby realizing object monomerization. Through the fusion of multi-frame image information, the errors and noise effects caused by single-frame detection are reduced, and the detection result is more stable; a detailed three-dimensional scene map is established by using voxel representation, overcoming the limitations in complex occlusion scenes caused by detecting only through the top view, and improving the detection accuracy; by constructing a weighted undirected graph and dividing clusters based on the weights of the edges between voxels, efficient segmentation and fine clustering of the target are realized, and objects arranged densely and partially occluded can be accurately separated.

[0016] The above description is only an overview of the technical solution of the embodiments of the present application. In order to better understand the technical means of the embodiments of the present application, it can be implemented according to the content of the description. And in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and understandable, the following specifically illustrates the specific implementation manners of the present application. Description of the Drawings

[0017] The drawings are only used to illustrate the embodiments and are not considered as a limitation to the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 The schematic diagram of the application scenario of the embodiments of the present application is shown; Figure 2 The flowchart of the object monomerization method provided by the embodiments of the present application is shown; Figure 3 The flowchart of another object monomerization method provided by the embodiments of the present application is shown; Figure 4 The schematic structural diagram of the object monomerization device provided by the embodiments of the present application is shown; Figure 5 The schematic structural diagram of the electronic device provided by the embodiments of the present application is shown; Figure 6 The schematic structural diagram of the three-dimensional scanning and reconstruction device provided by the embodiments of the present application is shown; Figure 7 The object monomerization effect diagram provided by the embodiments of the present application is shown; Figure 8 Another object monomerization effect diagram provided by the embodiments of the present application is shown. Detailed Description of the Embodiments

[0018] The exemplary embodiments of the present application will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein.

[0019] With the rapid development of fields such as autonomous driving, robot navigation, and intelligent security, three-dimensional object detection and segmentation technology has been widely applied and has become a key research direction. Currently, most of the mainstream three-dimensional object detection and segmentation methods are based on single-frame images, and the monomerization of three-dimensional objects is achieved through object detection, mask segmentation, and 2D to 3D projection. However, when performing three-dimensional object detection based on a single-frame image, due to the heavy dependence on the single-frame image, its detection and projection results are easily affected by noise, illumination changes, and occlusion interference; the detection result is the detection result obtained using the top view, and it is difficult to handle the overlapping and occlusion problems in complex scenarios such as a computer placed on a table or a chair blocked under the table, resulting in inaccurate detection results.

[0020] In this application, 2D (Two-Dimensional) represents a two-dimensional plane with length and width, and is used to describe the image plane content of objects and scenes. 3D (Three-Dimensional) represents a three-dimensional space with length, width, and height, and is used to describe the solid geometric structure of objects and scenes.

[0021] In related technologies, multi-frame image fusion, voxel representation, and segmentation methods based on graph structures and deep learning have also emerged. For example, methods for 3D detection based on the fusion of multi-frame images and multi-modal data also need to first convert the data into a top view, and cannot solve problems such as overlap and occlusion in complex scenes.

[0022] A voxel (Volume Element) is the smallest volume unit in three-dimensional space, similar to a pixel in a two-dimensional image. Voxelization helps to structurally represent point cloud data. Based on voxels, voxel clustering and supervoxel clustering point cloud segmentation methods can be performed. By dividing the point cloud data into voxels and performing supervoxel clustering, and using the adjacency relationship between supervoxels for iterative merging, the target point cloud region is output. This clustering method is relatively rough and only relies on the features of the point cloud network or the phase features of the point cloud. In scenes where the targets are dense and severely occluded, the segmentation results are not accurate enough.

[0023] This application provides an object monomerization method. A voxel map is constructed based on the 3D point cloud data of a three-dimensional scene, and target objects are detected from multiple frames of images of the three-dimensional scene. The detected target objects are projected onto the voxel map to obtain the voxel set of each target object. The adjacency relationship between voxels is constructed based on the voxel map and the voxel set to generate a weighted undirected graph. Since the weights of the edges between the voxels of the same target object in the weighted undirected graph are the same, voxel clustering can be performed based on this weighted undirected graph to obtain multiple clusters, and each cluster corresponds to a monomerized target object, thereby realizing object monomerization. Through the fusion of multi-frame image information, the errors and noise effects brought by single-frame detection are reduced, making the detection results more stable; a detailed three-dimensional scene map is established using voxel representation, overcoming the limitations in complex occlusion scenes caused by only detecting through the top view, and improving the detection accuracy; by constructing a weighted undirected graph and dividing clusters based on the weights of the edges between voxels, efficient segmentation and fine clustering of the target are achieved, and objects arranged densely and partially occluded can be accurately separated.

[0024] Figure 1The figure shows a schematic diagram of the application scenario of the embodiment of the present application. As shown in the figure, in a three-dimensional scene, point cloud data and image data are obtained by scanning the three-dimensional scene with a three-dimensional scanning and reconstruction device 1. For example, the three-dimensional scanning and reconstruction device 1 can be provided with a lidar, a camera, and a computing unit. Among them, the lidar collects the laser point cloud data of the three-dimensional scene, the camera obtains images from different perspectives of the three-dimensional scene, and the computing unit performs fusion processing on the laser point cloud data and the images to achieve three-dimensional reconstruction. The three-dimensional scanning and reconstruction device 1 can be a handheld device or a non-handheld device. If the three-dimensional scanning and reconstruction device 1 is a handheld device, the user holds the three-dimensional scanning and reconstruction device 1 and moves it in the three-dimensional scene to collect the point cloud data and image data of the three-dimensional scene. If the three-dimensional scanning and reconstruction device 1 is a non-handheld device, the three-dimensional scanning and reconstruction device 1 can move autonomously in the three-dimensional scene to collect the point cloud data and image data of the three-dimensional scene.

[0025] The terminal device 2 is communicatively connected to the three-dimensional scanning and reconstruction device 1, receives the three-dimensional reconstruction result obtained after the three-dimensional scanning and reconstruction device 1 performs three-dimensional reconstruction on the three-dimensional scene, and displays it. Of course, the three-dimensional scanning and reconstruction device 1 can also only collect the original point cloud data and image data and send them to the terminal device 2. The terminal device 2 performs fusion processing on the data to perform three-dimensional reconstruction and displays the three-dimensional reconstruction result. The terminal device 2 can be a device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a server, etc. In the figure, a mobile phone is used as an example. The terminal device 2 and the three-dimensional scanning and reconstruction device 1 can be connected by means of wired communication or wireless communication.

[0026] In some other scenarios, the lidar and the camera can also be separate devices and not integrated into the three-dimensional scanning and reconstruction device 1. They send the data they collect to the terminal device 2, and the terminal device 2 performs data fusion processing to obtain the three-dimensional reconstruction result and display it.

[0027] Figure 2 The figure shows a flowchart of the object monomerization method provided by the embodiment of the present application. This method is executed by an electronic device. The electronic device can be Figure 1 the three-dimensional scanning and reconstruction device 1 or the terminal device 2 in Figure 2 the figure. As shown, the method includes the following steps:

[0028] In a 3D reconstruction scenario, lidar can be used to collect lidar point cloud data of a 3D scene, and a camera can be used to collect images of the 3D scene. A computing unit or a terminal device can fuse the lidar point cloud data and the images to perform 3D reconstruction to obtain 3D point cloud data. To obtain the complete 3D information of the scene, the camera usually collects images of the scene from different perspectives to obtain multiple frames of images. Images from different perspectives can provide multiple side information of the scene to help calculate the depth information, shape, and spatial position relationship of objects in the 3D scene, so as to determine the spatial position of the objects. At the same time, images from multiple perspectives also help to supplement the details of the occluded areas. Parts that are occluded from some perspectives may be revealed from other perspectives, thus reconstructing the scene more completely. The camera can be a monocular camera, a binocular camera, a panoramic camera, etc.

[0029] In addition to lidar, point cloud data can also be collected by other devices, such as structured light scanners, time-of-flight (ToF) sensors, ultrasonic sensors, stereo vision cameras (such as binocular cameras), etc.

[0030] S120, construct a voxel map based on the 3D point cloud data.

[0031] This step performs discretization processing on the 3D point cloud data and simplifies the point cloud data into a voxel map (Voxel Map). A voxel map is a method of representing the environment in 3D space, similar to the pixel concept in 2D images. The voxel map divides the 3D space into multiple voxels, each voxel being a small cube region of a fixed volume, used to represent a specific region in space, and various information within this region, such as whether there is an object or attributes such as density values, can be stored.

[0032] When constructing the voxel map, first divide the 3D scene (i.e., 3D space) where the 3D point cloud data is located into individual voxels. Then, determine whether each voxel is occupied by the point cloud according to the 3D point cloud data. If there is a point cloud data point within the voxel, it is considered that the voxel is occupied, otherwise it is empty.

[0033] When constructing the voxel map, an appropriate voxel size can be set to make the voxel resolution fine enough to adapt to the fine segmentation of targets in complex scenes. The voxel size d matches the sampling density of the scene point cloud and the minimum geometric feature size s of the target. For example, the voxel size d is less than 1 / 2 of the minimum geometric feature size s. Among them, the minimum geometric feature size s is the minimum feature size of the target to be monomerized (such as the smallest protrusion, gap, etc.). For indoor fine scenes, the value range of the voxel size d can be 5mm - 20mm; for outdoor macroscopic scenes, the value range of the voxel size d can be 20mm - 100mm.

[0034] By simplifying the complex three-dimensional scene point cloud data into a voxel map, subsequent data processing and analysis can be more efficient. Meanwhile, by setting an appropriate voxel size, sufficient details are ensured to adapt to the fine segmentation of targets in complex scenes.

[0035] S130. Perform object detection on multiple frames of images respectively, project the detected target objects onto the voxel map, and obtain the voxel sets of each target object.

[0036] In this step, known image object detection algorithms can be used to perform object detection to obtain the positions of target objects in the images. The image object detection algorithm can be a detector based on a deep neural network, and the network architecture of the detector can be an R-CNN network, a Fast R-CNN network, a Faster R-CNN network, a YOLO series network, etc.

[0037] The camera that captures the images has its own parameters, including internal parameters and external parameters. Among them, the internal parameters are the parameters describing the internal imaging characteristics of the camera, such as focal length, principal point coordinates, lens distortion, etc., which are mainly related to the hardware structure of the camera itself and are the inherent attributes of the camera. The external parameters are the parameters describing the position and pose (i.e., the pose information) of the camera in the world coordinate system, including the rotation matrix and the translation vector, which reflect the external geometric relationship of the camera relative to the world coordinate system. Each frame of image captured by the camera contains the internal and external parameters of the camera. Based on the internal and external parameters of the camera, the detected target objects can be projected onto the voxel map, which is equivalent to projecting two-dimensional targets into 3D space, and the voxel sets of each target object are obtained.

[0038] The same target object will appear in multiple frames of images from different perspectives. Therefore, after detecting and projecting multiple frames of images, multiple voxel sets will be obtained for the same target object, that is, each target object corresponds to multiple voxel sets.

[0039] S140. Construct the adjacency relationship between voxels based on the voxel map and the voxel sets, and generate a weighted undirected graph.

[0040] Based on the voxel map and the voxel sets projected for each target object, the connection lines (i.e., edges) between the voxels within each voxel set in the voxel map can be established, and weights are assigned to each edge, thereby obtaining a weighted undirected graph. When clustering the target objects subsequently, fast and accurate clustering can be directly performed based on the weights of the edges in the weighted undirected graph.

[0041] Step S140 can be implemented through the following steps: S141. Add edges between any two voxels within each voxel set corresponding to each target object, and add a fixed weight to each edge.

[0042] S142. Superimpose the weights of the same edges.

[0043] Since each target object corresponds to multiple voxel sets, after adding edges and increasing the fixed weights of the edges between any two voxels within each voxel set corresponding to the same target object, the same two voxels will have edges added to them repeatedly. In this paper, such edges are called "same edges". For the same edges, superimpose their weights to obtain the total weight of the edge. For example, if target 1 is detected in all three frames of images, three voxel sets corresponding to target 1 are obtained. For the three voxel sets, an edge 1 is established between voxel 1 and voxel 2, and the fixed weight X is added to edge 1 three times. Then the total weight of edge 1 in the final weighted undirected graph is 3X.

[0044] Due to perspective problems, the voxel set obtained by performing object detection and projection on a certain frame of image may include voxels inside the real target object and voxels outside the real target object. Therefore, the edges added between the voxels in this voxel set include both the edges between the voxels inside the real target object and the edges between the voxels inside the real target object and the external voxels. However, since step S130 performs object detection and projection on multiple frames of images, the voxels in the voxel set obtained based on other frames of images are usually all voxels inside the real target object. The weighted undirected graph combines the edges and weights obtained based on all frames of images, that is, the weight of each edge in the finally obtained weighted undirected graph is the total weight after weight superposition. Therefore, the total weight of the edges between the voxels inside the real target object in the weighted undirected graph will be higher than the total weight of the edges between the voxels inside the real target object and the external voxels. For example, in a three-dimensional scene, there is a chair and a table beside the chair. The chair is detected in all three frames of images. The voxels in the voxel set detected and projected based on two of the frames (image 1 and image 2) are all real chair voxels. However, due to perspective problems, the voxel set detected and projected based on the other frame (image 3) includes not only the real chair voxels but also the voxels of the table. When adding edges between the voxels in the voxel set, the edges obtained based on image 1 and image 2 are all the edges between the voxels inside the real chair. In addition to the edges between the voxels inside the real chair (such edges are called the first type of edges) in the edges obtained based on image 3, there are also edges between the chair voxels and the table voxels (such edges are called the second type of edges). If a fixed weight X is added to each established edge, and the results of the three frames of images are combined, the total weight of the first type of edges is 3X, and the total weight of the second type of edges is X. Therefore, the total weight of the first type of edges is higher than the total weight of the second type of edges. Finally, when performing voxel clustering based on the weighted undirected graph, only the voxels connected by the first type of edges are clustered into one category. For a certain target detected in the image, the same and fixed weight is assigned to the edges between any two voxels in the corresponding voxel set to form the local connection relationship inside the target, strengthen the relevance of the voxels inside the target, and make the voxels of the same target have a closer connection in the undirected graph. Since the voxels in a single voxel set belong to the same target and have the same importance or degree of association, when constructing the internal connection relationship of the target object, no matter which part of the target these two voxels are in, as long as they belong to the voxel set corresponding to the same target, the same fixed weight will be assigned to strengthen their connection.

[0045] In the above manner, detection is performed based on multiple frames of images, and edges are added between the voxels within each voxel set and weights are added to the edges. Even if detection errors inevitably occur sometimes, resulting in weights being added to the edges between the voxels inside the real object and the voxels outside the real object, since the final weighted undirected graph is the total weight after weight aggregation, the weight of the edges between the voxels inside the real object is ultimately higher than the weight of the edges between the voxels outside the real object, improving the accuracy of clustering.

[0046] In addition, only adding edges and increasing fixed weights between the voxels within the voxel set can also simplify the calculation and improve the efficiency.

[0047] S150, divide the voxels in the weighted undirected graph into multiple clusters, where each cluster corresponds to a single-mesh target object.

[0048] This step realizes the segmentation and clustering of the target in the weighted undirected graph to obtain one or more single-mesh target objects. The community detection algorithm (such as the Louvain algorithm) can be used to divide the voxels in the weighted undirected graph into multiple clusters. The community detection algorithm is a clustering method based on graph theory for identifying tightly connected sub-regions from the graph structure.

[0049] Specifically, the community detection algorithm judges the aggregation degree of the voxels according to the connection weights between the voxels in the weighted undirected graph (i.e., the weights of the edges). Since the connection weights of the voxels inside the same target object are higher and the connection weights between the voxels of different target objects are smaller, the community detection algorithm can automatically cluster the voxels into multiple objects according to the connection weights.

[0050] If the same object is detected in multiple frames of images, multiple voxel sets of this object will be obtained. However, such multiple voxel sets are usually adjacent or close in 3D space. Through the above algorithm, such multiple voxel sets will be divided into one cluster. For example, for two frames of images from different perspectives, if the same chair is detected in these two frames of images, corresponding to two voxel sets, then these two voxel sets are very likely (e.g., 90%) to overlap, and the voxels in the two voxel sets will be clustered into one cluster. For two frames of images from different perspectives, if a chair is detected in each of these two frames of images, and these two chairs are different chairs, corresponding to two voxel sets, then these two voxel sets are very likely to have no intersection and will not be clustered into one cluster. The graph-based method can merge the voxel sets of the same object in multiple frames of images into one, making full use of the information of multiple frames of images, thereby realizing the complete and accurate segmentation of the target object in the scene and improving the accuracy and integrity of target segmentation.

[0051] Community detection algorithms can solve the problem of target monomerization in complex scenarios. It is also possible to perform clustering through other algorithms to achieve semantic monomerization of target objects. For example, traditional region growing or density-based clustering algorithms (such as DBSCAN) can be used for clustering, and this application does not limit this.

[0052] Figure 3 The flowchart of another object monomerization method provided by the embodiments of this application is shown, and this method is executed by an electronic device. As Figure 3 shown, this method includes the following steps: S210, obtain the 3D point cloud data and multiple frames of images of the three-dimensional scene.

[0053] S220, construct a voxel map according to the 3D point cloud data.

[0054] Steps S210 and S220 are similar to the aforementioned steps S110 and S120, and will not be elaborated here.

[0055] S231, perform object detection on each frame of image to obtain the 2D bounding boxes of the target objects in each image.

[0056] After performing object detection on each frame of image through an image object detection algorithm, the category of each detected object and the 2D bounding box of the object on the image plane are obtained. The 2D bounding box is usually represented by four coordinate values, for example, in the form of (x1, y1, x2, y2), where (x1, y1) are the coordinates of the upper left corner of the bounding box, and (x2, y2) are the coordinates of the lower right corner of the bounding box. Based on the above coordinate values, the 2D bounding box can be displayed in the image.

[0057] S232, obtain the pixel-level mask of the target object for each 2D bounding box area.

[0058] The 2D bounding box only represents the position range of the target object and is coarse-grained positioning information. In order to more accurately segment the boundary and shape of the target object, the precise pixel range of the target object can be further determined using the 2D bounding box to obtain the pixel-level mask of the target object, which is a fine-grained segmentation result.

[0059] In this step, a semantic segmentation algorithm is used to obtain the pixel-level mask of the target object. The semantic segmentation algorithm can classify each pixel in the image into a specific category, such as a person, a car, a dog, etc. For a target object whose 2D bounding box has been obtained through an image object detection algorithm, the semantic segmentation result can be used to obtain the pixel-level mask of the target object. For example, in the scenario of a city street view image, first use the image object detection algorithm to obtain the 2D bounding box of a car. Then, use the semantic segmentation algorithm to perform pixel-level classification on the entire image and mark the pixels belonging to the car category. Finally, combined with the 2D bounding box of the car obtained by object detection, the car pixels within the 2D bounding box in the semantic segmentation result are extracted to obtain the pixel-level mask of the car. The pixel value of 1 in the mask indicates belonging to the target object, and 0 indicates the background. The semantic segmentation algorithm can provide comprehensive pixel-level classification information with high segmentation accuracy.

[0060] In some embodiments, the object detection in step S231 and the pixel-level segmentation in step S232 can also be performed simultaneously through an instance segmentation algorithm. Through one algorithm, not only the 2D bounding box of each target object can be output, but also the pixel-level mask of each target object can be directly output. For example, the Mask R-CNN algorithm is used, which is an improvement based on Faster R-CNN and adds a parallel branch for predicting the segmentation mask corresponding to each target instance. The algorithm first uses the Region Proposal Network (RPN) to generate candidate target regions (similar to candidate regions of 2D bounding boxes), then processes the feature map through the ROI (Region of Interest) Align layer to avoid pixel information loss caused by pooling operations during feature extraction, and finally predicts the segmentation mask of each target instance through a dedicated mask prediction branch to directly obtain the pixel-level mask of the target object.

[0061] The above is the method for obtaining the pixel-level mask based on the segmentation algorithm. In some other embodiments, the pixel-level mask can also be obtained through a method based on boundary expansion and refinement or other methods.

[0062] S233, project the pixel-level masks in each image onto the voxel map according to the parameters of the camera that captured the image to obtain the voxel set of each target object, where the parameters of the camera include internal parameters and external parameters.

[0063] This step converts the pixel-level mask information extracted from the image into a voxel set in 3D space, providing a data basis for subsequent construction of the weighted undirected graph.

[0064] During the projection process, the pixel coordinates in the pixel-level mask of the target object are first converted to the normalized camera coordinate system. For example, for each pixel point (u, v) in the pixel-level mask of the target object, the camera intrinsic parameters and the depth information of the point are used to convert it to the coordinates (x, y, z) in the camera coordinate system, where the depth information of the pixel point is provided by the lidar. Then, according to the external parameters of the camera, the coordinates of each pixel point in the camera coordinate system of the target object are converted to the world coordinate system (the coordinate system of the voxel map is consistent with the world coordinate system), and the coordinates of the pixel point of the target object in the world coordinate system (X, Y, Z) are obtained. Finally, the coordinates are voxelized, and the coordinates of the pixel point in the world coordinate system (X, Y, Z) are converted to voxel coordinates (i, j, k) according to the origin and voxel size of the voxel map. In the above way, the pixel points in the pixel-level mask of each target object are traversed to obtain the voxel coordinates corresponding to each pixel point, and the voxel coordinates of each pixel point of the target object are collected to form the voxel set of the target object in the voxel map. For example, after the above processing, the voxel coordinates of a target object may be (15, 23, 37), (16, 23, 37), (15, 24,37), etc. These voxel coordinates constitute the voxel set of the target object.

[0065] S240 , constructing an adjacency relationship between voxels according to the voxel map and the voxel set to generate a weighted undirected graph.

[0066] S250 , dividing the voxels in the weighted undirected graph into a plurality of clusters, wherein each cluster corresponds to a monomerized target object.

[0067] Steps S240 and S250 are similar to the aforementioned steps S140 and S150 and will not be described in detail here.

[0068] S260 , back-projecting the multiple clusters to multiple frames of images to obtain a back-projected 2D bounding box of each cluster in the image.

[0069] This step includes the following steps: S2611, back-projecting each cluster to each frame of image according to the external parameters of the camera that captured each image.

[0070] The back-projection process is the opposite of the projection process in the aforementioned step S233. Each cluster includes multiple voxels, and the voxel coordinates (i, j, k) of each voxel can be converted to world coordinates (X, Y, Z) by the voxel size and the voxel map origin, for example: X=origin x +i×voxel size Y=origin y +j×voxelsize Z = origin z + k × voxel size where origin x 、origin y 、origin z are the coordinates of the origin of the voxel map, and voxel size is the size of the voxel.

[0071] For the world coordinates (X, Y, Z) of each voxel in the cluster, they can be transformed to the camera coordinate system using the extrinsic parameters of the camera. The transformation formula is: X camera = R -1 × (X world - t) where X camera is the point coordinate in the camera coordinate system after coordinate transformation, X world is the point coordinate in the world coordinate system, R is the rotation matrix, R -1 is the inverse of the rotation matrix R, that is, the transposed matrix R T , and t is the translation vector. In the camera coordinate system, the coordinates of a point can be expressed as (x, y, z). To project it onto the image plane, the intrinsic parameters of the camera can be used for projection. The intrinsic parameters of the camera include the focal length (f), the principal point coordinates (cx, cy), etc. The projection formula is as follows: u = f × x / z + cx v = f × y / z + cy where (u, v) are the pixel coordinates of the point after projection onto the image plane.

[0072] For each voxel in each cluster, it is projected onto the image captured by the corresponding camera according to the above steps, and the back-projection of each cluster on each frame of the image is obtained.

[0073] S2612, calculate the minimum bounding rectangle of the multiple back-projections of each cluster on multiple frames of images to obtain the back-projected 2D border of the cluster.

[0074] In multiple frames of images, the pixel points after back-projection of each cluster may be distributed at different positions. To obtain a compact 2D representation of the cluster in the image, the minimum bounding rectangle of the multiple back-projections of each cluster on multiple frames of images can be calculated, thereby obtaining the back-projected 2D border of the cluster.

[0075] S270, determine whether the back-projected 2D border matches at least one 2D border. If so, execute step S280; otherwise, execute step S290.

[0076] This step matches the back-projected 2D bounding box obtained by back-projection with the multiple 2D bounding boxes obtained in the previous step S231, so as to verify whether the monomerized clusters obtained in step S250 accurately correspond to the real target objects.

[0077] When determining whether the back-projected 2D bounding box matches at least one 2D bounding box, the IoU (Intersection over Union) of the back-projected 2D bounding box and each 2D bounding box can be calculated respectively. IoU is an index to measure the overlapping degree of two regions, and the calculation method is the area of the intersection of the two regions divided by the area of the union, which is used for the verification and matching of detection results. The method for calculating the IoU of the back-projected 2D bounding box and each 2D bounding box is: calculate the ratio of the area of the overlapping part of the back-projected 2D bounding box and the 2D bounding box to the area of the union of these two regions. If the IoU of the back-projected 2D bounding box and a certain 2D bounding box is greater than the preset threshold, it is determined that the back-projected 2D bounding box matches the 2D bounding box; if the IoU of the back-projected 2D bounding box and a certain 2D bounding box is less than or equal to the preset threshold, it is determined that the back-projected 2D bounding box does not match the 2D bounding box.

[0078] Generally, the larger the IoU value, the higher the degree of overlap between the back-projected 2D bounding box and the 2D bounding box of the real target object. The cluster corresponding to this back-projected 2D bounding box is the real target object, and the monomerization effect is very good; if the IoU value is low, it means that the position or size of the back-projected 2D bounding box is quite different from the 2D bounding box of the real target object, and the cluster corresponding to this back-projected 2D bounding box is not the real target object.

[0079] Regarding the preset threshold, when the preset threshold takes a value of 0.3 - 0.4, loose verification is realized, allowing some projection deviations or the situation where the detection box is on the small side, and avoiding misjudging the target as a false detection. In most cases, the preset threshold can be set to 0.5, taking into account both false detection filtering and the retention of real targets. When the preset threshold takes a value of 0.6 - 0.7, strict verification is realized, strengthening false detection filtering, and only retaining the clusters where the back-projected boundary and the detection box overlap highly. In some embodiments, three preset thresholds can be provided, and the user can select the most suitable preset threshold according to business requirements.

[0080] S280, retain the clusters corresponding to the back-projected 2D bounding box.

[0081] If the back-projected 2D bounding box matches at least one 2D bounding box, it means that the cluster corresponding to this back-projected 2D bounding box corresponds to the real target object, then retain this cluster.

[0082] S290, remove the clusters corresponding to the back-projected 2D bounding box.

[0083] If no matching bounding box can be found in all the 2D bounding boxes for the back-projected 2D bounding box, it means that the cluster corresponding to this back-projected 2D bounding box is a false detection and not a real target object, then remove this cluster.

[0084] By back-projecting each cluster onto each frame of the image in the above manner, the IoU between it and the 2D bounding box obtained from the original detection is calculated to verify whether each cluster accurately corresponds to the true detection target, filtering out the misdetected clusters, ensuring that the finally obtained monomerized clusters have high accuracy, and enhancing the robustness of the method of the embodiment of the present application. Generally speaking, using the IoU matching between the 2D detection and the 3D back-projection results as an auxiliary verification mechanism can effectively filter out misdetections and noisy point clouds and improve the credibility of the final monomerization result.

[0085] In the embodiment of the present application, by voxelizing and detecting the target based on multiple frames of images, the target recognized from the multiple frames of images is projected onto the voxel map, and the adjacency relationship between the voxels inside the same target object is constructed for clustering to obtain a cluster representing a single target object. The above process makes full use of the information of the entire three-dimensional scene, rather than relying solely on a certain perspective (such as the top-down perspective), and can capture the structure and relationship of the object more comprehensively, avoiding the problem of perspective dependence. If only based on a certain perspective (such as the top-down perspective), the connection relationship of the object in other directions may be ignored, resulting in missegmentation.

[0086] Suppose in a scene, there is a cup and a schoolbag on a table. The cup and the schoolbag are separated in three-dimensional space, but in the top-down perspective, a part of the schoolbag obscures a part of the cup, making some voxels of the cup appear to be separated.

[0087] Traditional segmentation methods may mistakenly consider the body and the handle of the cup as two different objects because they appear to be separated in the top view. However, in the embodiment of the present application, for the same target (such as the cup), edges are added between its internal voxels and high weights are assigned. By analyzing the connection relationship (connection weight) between the voxels through the community detection algorithm, the algorithm will preferentially cluster the voxels with high connection weights into one group.

[0088] The voxels of the cup body and the voxels of the cup handle are adjacent in three-dimensional space. The edge between the cup body and the cup handle is assigned a high weight (such as 1) because they belong to the same target (the cup). The edge between the cup body and the schoolbag is assigned a low weight because they are not the same target. Thus, even in the top-down perspective where the cup handle is obscured by the schoolbag, the cup handle and the cup body are still adjacent in three-dimensional space, so the weight between them is high. The community detection algorithm will give priority to high-weight connections, ensuring that the cup body and the cup handle are clustered into one whole. The connection weight between the schoolbag and the cup is low, and the algorithm will not mistake them for one whole.

[0089] For another example, a chair is placed under a table. If only traditional algorithms are used to detect from a top view, the voxels of the table and the chair both seem to belong to the object of the table. Through the method of the embodiments of the present application, through multiple frames of perspectives, such as a front view or an inclined view, the voxels in this area can be divided into two clusters (i.e., two sub-regions), representing the table and the chair respectively.

[0090] Figure 4 The structural schematic diagram of the object monomerization device provided by the embodiments of the present application is shown. As Figure 4 shown, the object monomerization device 400 includes: An acquisition module 410, configured to acquire 3D point cloud data and multiple frames of images of a three-dimensional scene; A construction module 420, configured to construct a voxel map according to the 3D point cloud data; A detection module 430, configured to perform target detection on multiple frames of images respectively, project the detected target objects onto the voxel map, and obtain a voxel set of each target object; A generation module 440, configured to construct an adjacency relationship between voxels according to the voxel map and the voxel set, and generate a weighted undirected graph; A division module 450, configured to divide the voxels in the weighted undirected graph into multiple clusters, where each cluster corresponds to a monomerized target object.

[0091] The object monomerization device 400 of the embodiments of the present application further includes other modules for performing the respective steps of the above method embodiments, which will not be elaborated here one by one.

[0092] Figure 5 The structural schematic diagram of the electronic device provided by the embodiments of the present application is shown. The specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0093] As Figure 5 shown, the electronic device 300 may include: a processor 302 and a memory 304.

[0094] Among them, the memory 304 is used to store a computer program 306. The memory 304 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The computer program 306 may include computer executable instructions.

[0095] The processor 302 is configured to execute the computer program 306 to implement the above object monomerization method embodiments.

[0096] The processor 302 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the electronic device 300 may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0097] Figure 6 The structural schematic diagram of a three-dimensional scanning and reconstruction device provided by an embodiment of the present application is shown. The three-dimensional scanning and reconstruction device 1 includes: A lidar 10 for collecting lidar point cloud data of a three-dimensional scene; A camera 20 for acquiring multiple frames of images of the three-dimensional scene; A calculation unit 30 for acquiring the lidar point cloud data and multiple frames of images, performing three-dimensional reconstruction on the lidar point cloud data and multiple frames of images to obtain 3D point cloud data, and executing the object monomerization method of the above embodiment. The calculation unit 30 may be a minicomputer or a controller, such as a single-chip microcomputer, a microcontroller unit (MCU), a programmable logic controller (PLC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a system on chip (SoC), etc.

[0098] An embodiment of the present application provides a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the object monomerization method embodiment is implemented.

[0099] An embodiment of the present application provides a computer program, and the computer program can be executed by a processor to implement the object monomerization method embodiment.

[0100] An embodiment of the present application provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the object monomerization method embodiment is implemented.

[0101] In several embodiments provided in the present application, if any function is implemented in the form of a software functional module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be an electronic device such as a personal computer, a server, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store computer program codes.

[0102] Figure 7 Shows the rendering effect of object monomerization in an embodiment of the present application. Figure 8 Shows another rendering effect of object monomerization in an embodiment of the present application. It can be seen from the figure that the embodiments of the present application can accurately segment multiple objects in a three-dimensional scene.

[0103] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. Based on the above description, the structure required to construct such a system is obvious. In addition, the embodiments of the present application are not directed to any particular programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the descriptions made above regarding specific languages are for the purpose of disclosing the best mode of the present application.

[0104] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a properly programmed computer. In the claims listing several devices, several of the units or modules in these devices can be embodied by the same hardware item. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0105] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for object monomerization, characterized in that, The method includes: Obtaining 3D point cloud data and multiple frames of images of a three-dimensional scene; Constructing a voxel map based on the 3D point cloud data; Performing object detection on each of the multiple frames of images, and projecting the detected target objects onto the voxel map to obtain a voxel set for each target object; Constructing an adjacency relationship between voxels based on the voxel map and the voxel sets, and generating a weighted undirected graph; Dividing the voxels in the weighted undirected graph into multiple clusters, where each cluster corresponds to a single-objectified target object.

2. The method according to claim 1, wherein The dividing the voxels in the weighted undirected graph into multiple clusters includes: Using a community detection algorithm to divide the voxels in the weighted undirected graph into multiple clusters.

3. The method according to claim 1, wherein The performing object detection on each of the multiple frames of images, and projecting the detected target objects onto the voxel map to obtain a voxel set for each target object includes: Performing object detection on each frame of image to obtain 2D bounding boxes of target objects in each image; Obtaining a pixel-level mask of the target object for each 2D bounding box region; Projecting the pixel-level masks in each image onto the voxel map according to the parameters of the camera that captured the image to obtain a voxel set for each target object, where the parameters of the camera include internal parameters and external parameters.

4. The method according to claim 3, wherein After the dividing the voxels in the weighted undirected graph into multiple clusters, the method further includes: Back-projecting the multiple clusters onto the multiple frames of images to obtain back-projected 2D bounding boxes of each cluster in the images; Determining whether the back-projected 2D bounding boxes match at least one of the 2D bounding boxes; If so, retaining the cluster corresponding to the back-projected 2D bounding box; If not, removing the cluster corresponding to the back-projected 2D bounding box.

5. The method according to claim 4, characterized in that, The determining whether the back-projected 2D bounding boxes match at least one of the 2D bounding boxes includes: Calculating the IoU between each back-projected 2D bounding box and each 2D bounding box; If the IoU between a back-projected 2D bounding box and a 2D bounding box is greater than a preset threshold, determining that the back-projected 2D bounding box matches the 2D bounding box; If the IoU between a back-projected 2D bounding box and a 2D bounding box is less than or equal to the preset threshold, determining that the back-projected 2D bounding box does not match the 2D bounding box.

6. The method according to claim 4, wherein The back-projecting the multiple clusters onto the multiple frames of images to obtain back-projected 2D bounding boxes of each cluster in the images includes: Back-projecting each cluster onto each frame of image according to the external parameters of the camera that captured each image; Calculating the minimum bounding rectangle of the multiple back-projections of each cluster on the multiple frames of images to obtain the back-projected 2D bounding box of the cluster.

7. The method according to claim 1, wherein The constructing an adjacency relationship between voxels based on the voxel map and the voxel sets, and generating a weighted undirected graph includes: Adding an edge between any two voxels within each voxel set corresponding to each target object, and adding a fixed weight to each edge; Superimposing the weights of the same edges.

8. The method according to claim 1, characterized in that, The 3D point cloud data is 3D point cloud data obtained by performing three-dimensional reconstruction on laser point cloud data and the multiple frames of images, and the multiple frames of images include images from different perspectives.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the object single-objectification method according to any one of claims 1 to 7.

10. A three-dimensional scanning and reconstruction device, characterized in that, The three-dimensional scanning and reconstruction device includes: A lidar, configured to collect lidar point cloud data of a three-dimensional scene; A camera, configured to obtain multiple frames of images of the three-dimensional scene; A computing unit, configured to obtain the lidar point cloud data and the multiple frames of images, perform three-dimensional reconstruction on the lidar point cloud data and the multiple frames of images to obtain 3D point cloud data, and execute the object monomerization method according to any one of claims 1 to 7.

Images