Method, device and equipment for real-time reconstruction of three-dimensional scene based on local fusion of meshes

Abstract

The application provides a three-dimensional scene real-time reconstruction method, device and equipment based on local fusion of a grid, comprising: if current frame point cloud data is received, dividing global point cloud data based on the current frame point cloud data to obtain old global point cloud data part and new global point cloud data part; determining the point type of each point in the global point cloud data according to the old global point cloud data part and the new global point cloud data part; and constructing a local grid according to the current frame point cloud data and the new global point cloud data part; and fusing the local grid into the global grid corresponding to the global point cloud data based on the point type of each point in the global point cloud data, to obtain new global point cloud data and the new global grid corresponding thereto, until the three-dimensional scene real-time reconstruction task is completed. The application can significantly reduce the calculation amount of three-dimensional scene real-time reconstruction, thereby effectively improving the efficiency of three-dimensional scene real-time reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of real-time three-dimensional scene reconstruction, and in particular to a real-time three-dimensional scene reconstruction method based on local mesh fusion, a real-time three-dimensional scene reconstruction device and a real-time three-dimensional scene reconstruction equipment. BACKGROUND

[0002] Real-time three-dimensional scene reconstruction technology is applied to many fields such as automatic driving, robots, medical treatment, etc. Generally, three-dimensional scene reconstruction and positioning are collectively referred to as SLAM (Simultaneous Localization and Mapping), but unlike real-time positioning in SLAM, real-time three-dimensional scene reconstruction technology focuses more on real-time modeling of the scene map. Three-dimensional scene reconstruction generally includes three reconstruction methods of point cloud data reconstruction based on a laser radar, depth map reconstruction based on an RGB-D depth camera and image reconstruction based on a normal monocular camera.

[0003] However, the current method for modeling based on RGB-D data is generally based on the principle of TSDF (Truncated Signed Distance Function), and on this basis, many real-time reconstruction frameworks such as KinectFusion, Kintinuous, ElasticFusion, InfiniTAM and BundleFusion are born. This kind of method adopts an incremental input mode, and as the input data increases, the calculation amount of real-time three-dimensional scene reconstruction will increase, resulting in low efficiency of real-time three-dimensional scene reconstruction. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a real-time three-dimensional scene reconstruction method based on local mesh fusion, a real-time three-dimensional scene reconstruction device and a real-time three-dimensional scene reconstruction equipment, which can significantly reduce the calculation amount of real-time three-dimensional scene reconstruction, thereby effectively improving the efficiency of real-time three-dimensional scene reconstruction.

[0005] In a first aspect, the present application provides a real-time three-dimensional scene reconstruction method based on local mesh fusion, comprising:

[0006] In the process of performing a real-time three-dimensional scene reconstruction task, if current frame point cloud data is received, the global point cloud data is divided based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part; wherein the global point cloud data is generated based on the initial frame point cloud data of the real-time three-dimensional scene reconstruction task to the previous frame point cloud data corresponding to the current frame point cloud data;

[0007] determine a point type of each point in the global point cloud data according to the old global point cloud data part and the new global point cloud data part; and construct a local mesh according to the current frame point cloud data and the new global point cloud data part; wherein the point type comprises an old point, a connecting point and a new point;

[0008] fuse the local mesh into a global mesh corresponding to the global point cloud data according to the point type of each point in the global point cloud data, to obtain new global point cloud data and a new global mesh corresponding to the new global point cloud data, until the real-time reconstruction task of the three-dimensional scene ends.

[0009] In an embodiment, the global point cloud data is divided based on the current frame point cloud data to obtain the old global point cloud data part and the new global point cloud data part, comprising:

[0010] taking an axis-aligned bounding box corresponding to the current frame point cloud data as a first bounding box, and expanding the first bounding box according to a preset resolution to obtain a second bounding box;

[0011] dividing point clouds in the global point cloud data outside the first bounding box to the old global point cloud data part, and dividing point clouds in the global point cloud data inside the second bounding box to the new global point cloud data part.

[0012] In an embodiment, the point type of each point in the global point cloud data is determined according to the old global point cloud data part and the new global point cloud data part, comprising:

[0013] determining an overlapping part between the old global point cloud data part and the new global point cloud data part;

[0014] marking points in the old global point cloud data part outside the overlapping part as old points, and marking points in the new global point cloud data part outside the overlapping part as new points, and marking points inside the overlapping part as connecting points.

[0015] In an embodiment, the local mesh is fused into the global mesh corresponding to the global point cloud data according to the point type of each point in the global point cloud data, to obtain new global point cloud data and a new global mesh corresponding to the new global point cloud data, comprising:

[0016] eliminating new points in the global mesh corresponding to the global point cloud data and triangular facets where the new points are located to obtain an eliminated global mesh, and updating a vertex identifier corresponding to each point in the eliminated global mesh;

[0017] updating a vertex identifier corresponding to each point in the local mesh based on the updated vertex identifier of each point in the eliminated global mesh;

[0018] determine a correspondence between the connection points in the global grid after culling and the local grid, and update the vertex identifiers corresponding to the connection points according to the correspondence;

[0019] fuse the local grid into the global grid after culling based on the updated vertex identifiers of the connection points, to obtain new global point cloud data and a new global grid corresponding thereto.

[0020] In an embodiment, the new points in the global grid corresponding to the global point cloud data are culled, and the new points are obtained, and the vertex identifiers corresponding to each point in the global grid after culling are updated, comprising:

[0021] setting an initial counter value to 0;

[0022] traversing the points in the global grid corresponding to the global point cloud data;

[0023] in a case where the current traversed point is not a new point, increasing the counter value by 1; in a case where the current traversed point is a new point, keeping the counter value unchanged; and replacing the vertex identifier corresponding to the current traversed point with the updated counter value;

[0024] constructing a mapping table based on the counter value; wherein the mapping table comprises the updated vertex identifiers of each point in the global grid;

[0025] according to the mapping table, culling the points with the vertex identifier being a specified numerical value and the triangular faces in which the points are located from the global grid, to obtain the global grid after culling and the updated vertex identifiers of each point in the global grid after culling.

[0026] In an embodiment, the vertex identifiers corresponding to each point in the local grid are updated based on the updated vertex identifiers of each point in the global grid after culling, comprising:

[0027] counting the number of points contained in the global grid after culling;

[0028] for any point in the local grid, taking the sum value between the vertex identifier corresponding to the point in the local grid and the number of points as the updated vertex identifier corresponding to the point in the local grid.

[0029] In an embodiment, the method further comprises:

[0030] In the process of performing the real-time reconstruction task of the three-dimensional scene, in a case where a loop is detected, the initial frame point cloud data of the real-time reconstruction task of the three-dimensional scene is fused to the current frame point cloud data, the fused point cloud data is subjected to voxel downsampling processing, and the point cloud triangular mesh is reconstructed from the point cloud data after the voxel downsampling processing, to realize loop correction.

[0031] In a second aspect, the present application also provides a device for real-time reconstruction of a three-dimensional scene based on local fusion of meshes, comprising:

[0032] a point cloud division module configured to, during execution of the real-time reconstruction task of the three-dimensional scene, if the current frame point cloud data is received, divide the global point cloud data based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part, wherein the global point cloud data is generated based on the initial frame point cloud data to the previous frame point cloud data corresponding to the current frame point cloud data of the real-time reconstruction task of the three-dimensional scene;

[0033] a point type and local mesh determination module configured to determine a point type to which each point in the global point cloud data belongs according to the old global point cloud data part and the new global point cloud data part, and construct a local mesh according to the current frame point cloud data and the new global point cloud data part, wherein the point type includes an old point, a connection point and a new point;

[0034] a mesh fusion module configured to fuse the local mesh into a global mesh corresponding to the global point cloud data based on the point type to which each point in the global point cloud data belongs, to obtain new global point cloud data and a new global mesh corresponding thereto, until the real-time reconstruction task of the three-dimensional scene is completed.

[0035] In a third aspect, the present application also provides an electronic device comprising a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.

[0036] In a fourth aspect, the present application also provides a computer readable storage medium storing computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method of any one of the first aspect.

[0037] The application provides a three-dimensional scene real-time reconstruction method, device and equipment based on local grid fusion.

[0038] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0039] So that the above and other objects, features and advantages of the present application can be clearly understood, preferred embodiments will be described in conjunction with the appended drawings. It should be noted, that the illustrations can not be to scale, and are generally provided by way of explanation of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 A flowchart of a three-dimensional scene real-time reconstruction method based on local grid fusion provided by an embodiment of the present application;

[0042] Figure 2 A schematic diagram of point cloud data provided for an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a mapping table provided for an embodiment of the present application;

[0044] Figure 4 A technical framework diagram of a three-dimensional scene real-time reconstruction based on local fusion of meshes provided for an embodiment of the present application;

[0045] Figure 5 A structural schematic diagram of a three-dimensional scene real-time reconstruction device based on local fusion of meshes provided for an embodiment of the present application;

[0046] Figure 6 A structural schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be described below in connection with embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0048] At present, such methods adopt an incremental input mode. With the increase of input data, not only is the efficiency of three-dimensional scene real-time reconstruction low, but also the model loop correction speed becomes slower and slower. Moreover, these methods have a common problem, i.e., no friendly real-time model processing interface is provided, resulting in that the object model presentation is subject to the original framework, and the framework engineering application is difficult.

[0049] Based on this, the present application provides a three-dimensional scene real-time reconstruction method, device and equipment based on local fusion of meshes, which can significantly reduce the calculation amount of three-dimensional scene real-time reconstruction, thereby effectively improving the efficiency of three-dimensional scene real-time reconstruction, solving the problems of fast loop and real-time scanning model output, reducing the influence of frame number on the reconstruction process loop correction, and enabling the reconstructed mesh model to be output in real time for other post-processing operations.

[0050] To facilitate the understanding of the present embodiment, first, a three-dimensional scene real-time reconstruction method based on local fusion of meshes disclosed by the present embodiment will be described in detail. The present embodiment mainly performs real-time modeling based on RGB-D image data. Referring to a flow schematic diagram of a three-dimensional scene real-time reconstruction method based on local fusion of meshes shown in Figure 1 The method mainly includes the following steps S102 to S106.

[0051] In step S102, in the process of performing the real-time reconstruction task of the three-dimensional scene, if the current frame point cloud data is received, the global point cloud data is divided based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part.

[0052] The current frame point cloud data is obtained based on the RGB-D data of the real-time scanning image received at the current moment; the global point cloud data is generated based on the initial frame point cloud data to the previous frame point cloud data corresponding to the current frame point cloud data in the real-time reconstruction task of the three-dimensional scene, for example, the initial frame point cloud data to the previous frame point cloud data is fused and voxel down-sampling processed, so that the global point cloud data is obtained; the old global point cloud data part is the point cloud in the global point cloud data located outside the axisymmetric bounding box of the current frame point cloud data, and the new global point cloud data part is the point cloud in the global point cloud data located within the specified range of the current frame point cloud data, and the specified range is positively correlated with the size of the axisymmetric bounding box.

[0053] In an example, when the RGB-D data of the new real-time scanning image is received, the corresponding current frame point cloud data and the axisymmetric bounding box thereof are generated, the first bounding box and the second bounding box are respectively constructed based on the axisymmetric bounding box, the first bounding box is also the axisymmetric bounding box, and the second bounding box is the bounding box after the axisymmetric bounding box is expanded by a specified multiple, which is equivalent to the aforementioned specified range; the global point cloud data is divided into the old global point cloud data part and the new global point cloud data part by using the first bounding box and the second bounding box.

[0054] In step S104, the point type to which each point in the global point cloud data belongs is determined according to the old global point cloud data part and the new global point cloud data part; and the local mesh is constructed according to the current frame point cloud data and the new global point cloud data part.

[0055] The point type includes an old point, a connection point and a new point, the new point can be understood as a point existing in the current frame point cloud data, the old point can be understood as a point not existing in the current frame point cloud data and not being in the same triangular patch as the new point, and the connection point can be understood as a point not existing in the current frame point cloud data and being in the same triangular patch as the new point. In an example, the point type to which each point in the global point cloud data belongs can be determined according to the overlapping part between the old global point cloud data part and the new global point cloud data part.

[0056] In an example, the current frame point cloud data can be merged with the new global point cloud data part, the updated point cloud can be obtained by sampling the merged point cloud data, and the local mesh can be obtained by performing triangular mesh reconstruction on the updated point cloud.

[0057] Step S106, based on the point type to which each point in the global point cloud data belongs, the local grid is fused into the global grid corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global grid until the three-dimensional scene real-time reconstruction task ends.

[0058] The global grid is a grid obtained by performing triangular mesh reconstruction on the global point cloud data. In an example, each point in the global point cloud data can be traversed to assign a corresponding vertex identifier to each point according to the point type to which each point belongs, so as to generate a mapping table corresponding to the global point cloud data. The new points and the triangular faces in which the new points are located in the global grid are removed according to the mapping table to obtain a global grid from which the new points and the triangular faces in which the new points are located are removed, referred to as a global grid after removal. The vertex identifier corresponding to each point in the local grid is determined again based on the vertex identifier corresponding to each point in the global grid after removal. The vertex identifier corresponding to the connection point is updated according to the correspondence between the connection point in the global grid after removal and the local grid. The global grid after removal and the local grid are spliced based on the updated vertex identifier of the connection point to obtain new global point cloud data and its corresponding new global grid until the three-dimensional scene real-time reconstruction task ends.

[0059] The three-dimensional scene real-time reconstruction method based on grid local fusion provided by the embodiments of the present application divides the global point cloud data into an old global point cloud data part and a new global point cloud data part by using the current frame point cloud data, determines the point type to which each point in the global point cloud data belongs, and constructs a local grid. Then, the local grid is fused with the global grid based on the point type to which each point in the global point cloud data belongs. Compared with the prior art method of first updating the global point cloud data using the current frame point cloud data, and then updating the global grid using the updated global point cloud data, the embodiments of the present application can significantly reduce the calculation amount of three-dimensional scene real-time reconstruction by fusing the local grid into the global grid, thereby effectively improving the efficiency of three-dimensional scene real-time reconstruction.

[0060] For ease of understanding, the embodiments of the present application provide a specific implementation manner of a three-dimensional scene real-time reconstruction method based on grid local fusion. The overall process of the method is as follows:

[0061] Step 1, RGB-D data of a real-time scanned image is collected by a device, and corresponding point cloud data is generated. The point cloud data of the i-th frame is recorded as curPointCloud[i]. The point cloud data of the i-th frame is also referred to as the point cloud data of the previous frame.

[0062] Step 2, the camera pose of the i-th frame point cloud data corresponding to the world coordinate system is recorded as curPose[i]. The camera pose of each frame is obtained by registering the point cloud data of the previous frame.

[0063] Step 3, traverse all point cloud frames from the first frame to the i-th frame, fuse them into a point cloud data in the world coordinate system, then sample the fused point cloud data at a specific resolution by a voxel downsampling algorithm to generate global point cloud data, and record it as globalPointCloud. The point cloud data of the first frame is also the initial frame point cloud data.

[0064] Step 4, perform point cloud triangular mesh reconstruction on the global point cloud data globalPointCloud to obtain a global mesh, and record it as globalMesh.

[0065] Step 5, based on the current frame point cloud data, divide the global point cloud data to obtain an old global point cloud data part and a new global point cloud data part. For specific implementation, refer to Figure 2 a schematic diagram of point cloud data, the circular black points in the middle on the left are global point cloud data, the triangular points are current frame (i+1-th frame) point cloud data, the gray dashed box is the first bounding box Box1, the black dashed box is the second bounding box Box2, and the distance from Box1 to Box2 is twice the global point cloud resolution; the circular black points in the middle on the right are old points, the circular dark gray points are connection points, and the circular light gray points are new points.

[0066] Specifically, the following steps 5.1 to 5.2 are included:

[0067] Step 5.1, take the axis-aligned bounding box corresponding to the current frame point cloud data as the first bounding box, and expand the first bounding box according to a preset resolution to obtain the second bounding box. In an example, when the current frame point cloud data, for example, the i+1-th frame point cloud data, is generated, the axis-aligned bounding box of the frame is calculated, the axis-aligned bounding box is taken as the first bounding box Box1, and the first bounding box Box1 is expanded by twice the point cloud resolution distance to obtain the second bounding box Box2, as shown in Figure 2 .

[0068] Step 5.2, divide the point cloud in the global point cloud data outside the first bounding box to the old global point cloud data part, and divide the point cloud in the global point cloud data inside the second bounding box to the new global point cloud data part. In an example, the global point cloud data globalPointCloud is filtered into two parts using the first bounding box Box1, and the part of point cloud outside the box is recorded as the old global point cloud data part old_globalPointCloud. Similarly, the global point cloud data globalPointCloud is filtered into two parts using the second bounding box Box2, and the part of point cloud inside the box is recorded as the new global point cloud data part new_globalPointCloud.

[0069] Step 6, determining the point type of each point in the global point cloud data according to the old global point cloud data part and the new global point cloud data part. Specifically, it includes the following steps 6.1 to 6.2:

[0070] Step 6.1, determining the overlapping part between the old global point cloud data part old_globalPointCloud and the new global point cloud data part new_globalPointCloud.

[0071] Step 6.2, marking the points in the old global point cloud data part outside the overlapping part as old points; and marking the points in the new global point cloud data part outside the overlapping part as new points; and marking the points in the overlapping part as connection points. Please continue to refer to Figure 2 , the old points are the points in the non-overlapping part of the old global point cloud data part old_globalPointCloud, the new points are the points in the non-overlapping part of the new global point cloud data part new_globalPointCloud, and the connection points are the points in the overlapping part between the old global point cloud data part old_globalPointCloud and the new global point cloud data part new_globalPointCloud.

[0072] Step 7, constructing a local mesh according to the current frame point cloud data and the new global point cloud data part. In an example, the i+1 frame point cloud data is merged with the new global point cloud data part new_globalPointCloud, and the new frame and the global frame update point cloud updatePointCloud is obtained by sampling at a certain resolution. Triangular mesh reconstruction is performed on the update point cloud updatePointCloud to obtain the local mesh updateMesh.

[0073] Step 8, based on the point type of each point in the global point cloud data, the local mesh is fused into the global mesh corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global mesh. Specifically, it includes the following steps 8.1 to 8.4:

[0074] Step 8.1, removing the new points and the triangular faces where the new points are located in the global mesh corresponding to the global point cloud data to obtain the removed global mesh, and updating the vertex identifier corresponding to each point in the removed global mesh. In specific implementation, please refer to the following (a) to (e):

[0075] (a) Set the initial counter value counter to 0.

[0076] (b) Traverse the points in the global mesh corresponding to the global point cloud data. In one example, you can use a pointer to traverse sequentially from the first point of the global point cloud data globalMesh to the last point.

[0077] (c) If the currently traversed point is not a new point, increment the counter value by 1; if the currently traversed point is a new point, keep the counter value unchanged. In one example, when the current point pointed to by the pointer is a reserved point, the counter value is incremented by one; otherwise, the counter value remains unchanged. Reserved points are also known as old points and connection points.

[0078] (d) Construct a mapping table based on the counter values; the mapping table includes the updated vertex identifier (id) for each point in the global mesh. In one example, the calculation logic of the mapping table is as follows: when the point pointed to by the pointer does not need to be deleted, the vertex id is counter; otherwise, the vertex id is -1. The mapping table is completed after the pointer completes one traversal.

[0079] (e) According to the mapping table, remove the points with vertex identifiers of the specified values ​​and the triangular faces containing those points from the global mesh to obtain the global mesh after removal and the updated vertex identifiers for each point within the global mesh. In one example, iterate through all triangular faces, update the vertex IDs according to the mapping table, and directly delete triangles with a value of -1 in the mapping table and a new point type. Note that at this point, the vertex IDs of all triangular faces for connecting points are incorrect and need to be recalculated in step 8.3.

[0080] For example, see Figure 3 The diagram illustrates a mapping table. Assume there is a triangular mesh (left column) consisting of five faces and five points. Points 2 and 4 need to be deleted. A mapping table is constructed (middle column). Based on the mapping table, points 2 and 4 are deleted, and the vertex IDs of the triangular faces are updated simultaneously, resulting in a new triangular mesh (right column). This new triangular mesh is also the global mesh after the deletion.

[0081] Step 8.2: Based on the updated vertex identifiers of each point in the global mesh after deletion, update the vertex identifiers corresponding to each point in the local mesh. In one example, count the number of points in the global mesh after deletion; for any point in the local mesh, use the sum of the vertex identifier corresponding to that point in the local mesh and the number of points as the updated vertex identifier corresponding to that point in the local mesh. For example, count the number of points remaining in the global mesh after deleting new points, denoted as n, and add n to the vertex IDs of all triangles in the local mesh updateMesh.

[0082] Step 8.3, determining the correspondence between the connection points in the global mesh after culling and the local mesh, and updating the vertex identification corresponding to the connection points according to the correspondence. In an example, all connection points in the global mesh globalMesh are traversed, the vertex id of the corresponding point in the local mesh updateMesh is found through coordinate matching, and the id of the point in the local mesh updateMesh is assigned to the point in the global mesh globalMesh, so as to complete the update of the id of the connection point.

[0083] Step 8.4, based on the updated vertex identification of the connection points, fusing the local mesh into the global mesh after culling to obtain new global point cloud data and a new global mesh corresponding thereto. In an example, the points of the local mesh updateMesh are directly pushed to the back of the points of the global mesh globalMesh after culling, and the triangular mesh of the local mesh updateMesh is added to the global mesh globalMesh after culling.

[0084] Step 9, in the process of executing the real-time reconstruction task of the three-dimensional scene, in the case of detecting loop closure, the initial frame point cloud data of the real-time reconstruction task of the three-dimensional scene is fused to the current frame point cloud data, the fused point cloud data is subjected to voxel downsampling processing, and the point cloud triangular mesh is reconstructed from the point cloud data subjected to the voxel downsampling processing, so as to realize loop closure correction. In an example, when the overall system is running, the global mesh globalMesh is displayed in real time, and a separate thread is established to detect loop closure. When loop closure is detected, the position of the corresponding frame is directly updated, and then steps 3 and 4 are used to complete loop closure correction. Since steps 3 and 4 only simply add and downsample the point cloud, the calculation cost is very low, and therefore the loop closure correction time of the system is basically not affected by the frame number.

[0085] In summary, the embodiments of the present application have at least the following characteristics: (1) the mapping table is used to delete the points and the patches of the triangular mesh. In general, deleting a point in the mesh will disturb the vertex ids of all the triangular meshes, resulting in the disorder of the mesh. However, the mapping table is used to delete the point and update the vertex id in the time complexity of O(n). (2) local fusion of the mesh. In general, the global mesh needs to be updated by first updating the global point cloud data and then recalculating the mesh of the global point cloud data. The calculation amount is very large. The local mesh fusion only needs to update the local point cloud and perform local meshing, and then splice the local mesh to the global mesh, and the calculation amount is very small.

[0086] Further, the embodiments of the present application also provide a method for reconstructing a three-dimensional scene in real time, as shown in the following. Figure 4A kind of technical framework diagram of three-dimensional scene real-time reconstruction based on grid local fusion shown, comprising: acquisition RGB-D data, generate current frame point cloud data and calculate point cloud position, save to the point cloud container that stores all frames;Based on the initial frame point cloud data in point cloud container storage to the previous frame point cloud data calculation global point cloud data and global grid, combine current frame point cloud data and output real-time grid for grid local fusion;In addition, based on current frame point cloud data loop detection and loop correction are carried out, and the point cloud data after loop correction is saved to point cloud container.The three-dimensional scene real-time reconstruction method based on grid local fusion provided in the embodiment of the application has obvious advantages in loop correction speed compared with the real-time reconstruction algorithm based on TSDF algorithm, is more flexible in reconstruction result presentation and post-processing, and is more beneficial to the engineering application landing of algorithm.

[0087] On the basis of the foregoing embodiment, the embodiment of the application provides a kind of three-dimensional scene real-time reconstruction device based on grid local fusion, refer to Figure 5 A kind of three-dimensional scene real-time reconstruction device based on grid local fusion structure diagram shown, the device mainly includes the following parts:

[0088] Point cloud division module 502, for in the process of executing three-dimensional scene real-time reconstruction task, if current frame point cloud data is received, then based on current frame point cloud data, global point cloud data is divided, and old global point cloud data part and new global point cloud data part are obtained;Wherein, global point cloud data is generated based on the previous frame point cloud data corresponding to the initial frame point cloud data to current frame point cloud data of three-dimensional scene real-time reconstruction task;

[0089] Point type and local grid determination module 504, for determining the point type of each point in global point cloud data according to old global point cloud data part and new global point cloud data part;And, according to current frame point cloud data and new global point cloud data part, local grid is structured;Wherein, point type includes old point, connecting point and new point;

[0090] Grid fusion module 506, for based on the point type of each point in global point cloud data, local grid is fused into the global grid corresponding to global point cloud data, to obtain new global point cloud data and its corresponding new global grid, until three-dimensional scene real-time reconstruction task ends.

[0091] The three-dimensional scene real-time reconstruction device based on local fusion of meshes provided by the embodiment of the application divides the global point cloud data into an old global point cloud data part and a new global point cloud data part by using the current frame point cloud data, thereby determining the point type of each point in the global point cloud data and constructing a local mesh, and then fusing the local mesh with the global mesh based on the point type of each point in the global point cloud data. Compared with the prior art in which the global point cloud data is first updated by using the current frame point cloud data, and then the global mesh is updated by using the updated global point cloud data, the embodiment of the application can significantly reduce the calculation amount of the three-dimensional scene real-time reconstruction by fusing the local mesh into the global mesh, thereby effectively improving the efficiency of the three-dimensional scene real-time reconstruction.

[0092] In an embodiment, the point cloud division module 502 is specifically configured to:

[0093] take the axis-aligned bounding box corresponding to the current frame point cloud data as a first bounding box, and perform expansion processing on the first bounding box according to a preset resolution to obtain a second bounding box;

[0094] divide the point cloud located outside the first bounding box in the global point cloud data to the old global point cloud data part, and divide the point cloud located inside the second bounding box in the global point cloud data to the new global point cloud data part.

[0095] In an embodiment, the point type and local mesh determination module 504 is specifically configured to:

[0096] determine an overlapping part between the old global point cloud data part and the new global point cloud data part;

[0097] mark the point located outside the overlapping part in the old global point cloud data part as an old point, mark the point located outside the overlapping part in the new global point cloud data part as a new point, and mark the point located inside the overlapping part as a connection point.

[0098] In an embodiment, the mesh fusion module 506 is specifically configured to:

[0099] remove the new point and the triangular facet where the new point is located in the global mesh corresponding to the global point cloud data to obtain a removed global mesh, and update the vertex identifier corresponding to each point in the removed global mesh;

[0100] update the vertex identifier corresponding to each point in the local mesh based on the updated vertex identifier of each point in the removed global mesh;

[0101] determine the corresponding relationship between the connection point and the removed global mesh and the local mesh, and update the vertex identifier corresponding to the connection point according to the corresponding relationship;

[0102] Based on the updated vertex identifier of the connection point, the local mesh is fused into the global mesh after the elimination to obtain new global point cloud data and a new global mesh corresponding thereto.

[0103] In an embodiment, the mesh fusion module 506 is specifically configured to:

[0104] Set an initial counter value to 0;

[0105] Iterate through the points in the global mesh corresponding to the global point cloud data;

[0106] In a case where the currently iterated point is not a new point, increase the counter value by 1; in a case where the currently iterated point is a new point, keep the counter value unchanged; and replace the vertex identifier corresponding to the currently iterated point with the updated counter value;

[0107] Construct a mapping table based on the counter value; wherein the mapping table comprises the updated vertex identifier of each point in the global mesh;

[0108] According to the mapping table, eliminate the points with the vertex identifier being a specified numerical value and the triangular facets where the points are located from the global mesh to obtain an eliminated global mesh and an updated vertex identifier of each point in the eliminated global mesh.

[0109] In an embodiment, the mesh fusion module 506 is specifically configured to:

[0110] Count the number of points contained in the eliminated global mesh;

[0111] For any point in the local mesh, take the sum value between the vertex identifier corresponding to the point in the local mesh and the number of points as the updated vertex identifier corresponding to the point in the local mesh.

[0112] In an embodiment, the method further comprises a loop correction module configured to:

[0113] In the process of performing the real-time reconstruction task of the three-dimensional scene, in a case where a loop is detected, fuse the initial frame point cloud data of the real-time reconstruction task of the three-dimensional scene to the current frame point cloud data, perform voxel down-sampling processing on the fused point cloud data, perform point cloud triangular mesh reconstruction on the point cloud data after the voxel down-sampling processing, to realize loop correction.

[0114] The device provided in the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments, and for brevity of description, the part not mentioned in the device embodiment part can be referred to the corresponding content in the foregoing method embodiments.

[0115] The electronic device provided by the embodiment of the present application, specifically, comprises a processor and a storage device; the storage device stores a computer program, and the computer program performs the method according to any one of the above embodiments when being run by the processor.

[0116] Figure 6 The structural schematic diagram of the electronic device provided by the embodiment of the present application, the electronic device 100 comprises a processor 60, a memory 61, a bus 62 and a communication interface 63, the processor 60, the communication interface 63 and the memory 61 are connected through the bus 62; the processor 60 is used for executing the executable module stored in the memory 61, for example, a computer program.

[0117] The memory 61 can contain a high-speed random access memory (RAM, Random Access Memory) and can also include a non-volatile memory, for example, at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0118] The bus 62 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 6 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0119] The memory 61 is used for storing a program, and the processor 60 executes the program after receiving an execution instruction. The method performed by the device defined by the flow process disclosed in any one of the above embodiments can be applied to the processor 60 or realized by the processor 60.

[0120] The processor 60 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 60 or by instructions in the form of software. The processor 60 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61, and combines the hardware to complete the steps of the above method.

[0121] The computer program product of the readable storage medium provided by the embodiments of the present application comprises a computer readable storage medium storing program codes, and the program codes comprise instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.

[0122] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0123] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for real-time reconstruction of 3D scenes based on local mesh fusion, characterized in that, include: During the execution of a real-time 3D scene reconstruction task, if current frame point cloud data is received, the global point cloud data is divided based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part; wherein, the global point cloud data is generated based on the point cloud data of the previous frame corresponding to the current frame point cloud data from the initial frame point cloud data of the real-time 3D scene reconstruction task. Based on the old global point cloud data portion and the new global point cloud data portion, determine the point type of each point in the global point cloud data; and construct a local mesh based on the current frame point cloud data and the new global point cloud data portion; wherein, the point type includes old points, connection points, and new points, the new points are points existing in the current frame point cloud data, the old points are points that do not exist in the current frame point cloud data and are not in the same triangular facet as the new points, and the connection points are points that do not exist in the current frame point cloud data and are in the same triangular facet as the new points; Based on the point type to which each point belongs in the global point cloud data, the local mesh is fused into the global mesh corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global mesh, until the real-time reconstruction task of the three-dimensional scene ends. Determining the point type of each point in the global point cloud data based on the old global point cloud data portion and the new global point cloud data portion includes: determining the overlapping portion between the old global point cloud data portion and the new global point cloud data portion; marking points in the old global point cloud data portion that are outside the overlapping portion as old points; marking points in the new global point cloud data portion that are outside the overlapping portion as new points; and marking points that are within the overlapping portion as connection points.

2. The real-time reconstruction method for 3D scenes based on local mesh fusion according to claim 1, characterized in that, Based on the current frame point cloud data, the global point cloud data is divided into an old global point cloud data portion and a new global point cloud data portion, including: The axis-aligned bounding box corresponding to the current frame point cloud data is used as the first bounding box, and the first bounding box is enlarged according to a preset resolution to obtain the second bounding box. The point clouds located outside the first bounding box in the global point cloud data are assigned to the old global point cloud data portion; and the point clouds located inside the second bounding box in the global point cloud data are assigned to the new global point cloud data portion.

3. The real-time reconstruction method for 3D scenes based on local mesh fusion according to claim 1, characterized in that, Based on the point type to which each point belongs in the global point cloud data, the local mesh is fused into the global mesh corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global mesh, including: The new point and the triangular facet containing the new point are removed from the global mesh corresponding to the global point cloud data to obtain the removed global mesh, and the vertex identifier corresponding to each point in the removed global mesh is updated. Based on the updated vertex identifier of each point in the global mesh after the removal, update the vertex identifier corresponding to each point in the local mesh; Determine the correspondence between the connection points and the culled global mesh and the local mesh, and update the vertex identifier corresponding to the connection points according to the correspondence; Based on the updated vertex identifiers of the connection points, the local mesh is merged into the culled global mesh to obtain new global point cloud data and its corresponding new global mesh.

4. The real-time reconstruction method for 3D scenes based on local mesh fusion according to claim 3, characterized in that, The process involves removing the new point and the triangular facet containing the new point from the global mesh corresponding to the global point cloud data to obtain a removed global mesh, and updating the vertex identifier corresponding to each point in the removed global mesh, including: Set the initial counter value to 0; The points in the global grid corresponding to the global point cloud data are traversed. If the currently traversed point is not the new point, increment the counter value by 1; if the currently traversed point is the new point, keep the counter value unchanged; replace the vertex identifier corresponding to the currently traversed point with the updated counter value. A mapping table is constructed based on the counter value; wherein the mapping table includes the updated vertex identifier for each point in the global grid; According to the mapping table, the points whose vertices are identified by a specified value and the triangular facets containing those points are removed from the global mesh to obtain the global mesh after removal and the updated vertex identifiers for each point in the global mesh after removal.

5. The real-time reconstruction method for 3D scenes based on local mesh fusion according to claim 3, characterized in that, Based on the updated vertex identifiers of each point in the global mesh after the removal of culled points, the vertex identifiers corresponding to each point in the local mesh are updated, including: Count the number of points contained in the global grid after the removal; For any point in the local mesh, the sum of the vertex identifier corresponding to that point in the local mesh and the number of points is used as the updated vertex identifier corresponding to that point in the local mesh.

6. The real-time reconstruction method for 3D scenes based on local mesh fusion according to claim 1, characterized in that, The method further includes: During the execution of a real-time 3D scene reconstruction task, if a loop closure is detected, the point cloud data of the initial frame of the real-time 3D scene reconstruction task is fused with the point cloud data of the current frame. The fused point cloud data is then subjected to voxel downsampling, and the voxel downsampling point cloud data is then reconstructed using point cloud triangular meshes to achieve loop closure correction.

7. A real-time reconstruction device for a 3D scene based on local mesh fusion, characterized in that, include: The point cloud partitioning module is used to partition the global point cloud data based on the current frame point cloud data when receiving the current frame point cloud data during the execution of the real-time reconstruction task of the 3D scene, to obtain an old global point cloud data part and a new global point cloud data part; wherein, the global point cloud data is generated based on the point cloud data of the initial frame of the real-time reconstruction task of the 3D scene to the previous frame point cloud data corresponding to the current frame point cloud data. The point type and local mesh determination module is used to determine the point type of each point in the global point cloud data based on the old global point cloud data portion and the new global point cloud data portion; and to construct a local mesh based on the current frame point cloud data and the new global point cloud data portion; wherein, the point type includes old points, connection points, and new points, the new points are points existing in the current frame point cloud data, the old points are points that do not exist in the current frame point cloud data and are not located in the same triangular facet as the new points, and the connection points are points that do not exist in the current frame point cloud data and are located in the same triangular facet as the new points; The mesh fusion module is used to fuse the local mesh into the global mesh corresponding to the global point cloud data based on the point type to which each point belongs in the global point cloud data, so as to obtain new global point cloud data and its corresponding new global mesh, until the real-time reconstruction task of the three-dimensional scene ends. The point type and local mesh determination module is specifically used to: determine the overlapping portion between the old global point cloud data portion and the new global point cloud data portion; mark points in the old global point cloud data portion that are outside the overlapping portion as old points; and mark points in the new global point cloud data portion that are outside the overlapping portion as new points; and mark points that are within the overlapping portion as connection points.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images