Three-dimensional scene real-time reconstruction method, device and equipment based on grid local fusion

Abstract

The invention provides a three-dimensional scene real-time reconstruction method, device and equipment based on grid local fusion, and the method comprises the steps: dividing global point cloud data based on the current frame point cloud data if the current frame point cloud data is received, and obtaining an old global point cloud data part and a new global point cloud data part; determining a point type to which each point in the global point cloud data belongs according to the old global point cloud data part and the new global point cloud data part; constructing a local grid according to the current frame point cloud data and the new global point cloud data part; and based on the point type to which each point in the global point cloud data belongs, fusing the local grid into a global grid corresponding to the global point cloud data to obtain new global point cloud data and a new global grid corresponding to the new global point cloud data until the real-time reconstruction task of the three-dimensional scene is finished. According to the method, the calculation amount of real-time reconstruction of the three-dimensional scene can be remarkably reduced, so that the real-time reconstruction efficiency of the three-dimensional scene is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time three-dimensional scene reconstruction, and in particular to a real-time three-dimensional scene reconstruction method, device and equipment based on grid local fusion. Background Art

[0002] The real-time three-dimensional scene reconstruction technology is applied to many fields such as autonomous driving, robotics, and medical treatment. Usually, three-dimensional scene reconstruction and positioning are collectively referred to as SLAM (Simultaneous Localization and Mapping), but different from real-time positioning in SLAM, the real-time three-dimensional scene reconstruction technology focuses more on the real-time modeling of the scene map. Three-dimensional scene reconstruction generally includes three reconstruction methods: point cloud data reconstruction based on lidar, depth map reconstruction based on RGB-D depth cameras, and image reconstruction based on ordinary monocular cameras.

[0003] However, currently, the methods for modeling based on RGB-D data generally follow the principle of TSDF (Truncated Signed Distance Function), and many real-time reconstruction frameworks such as KinectFusion, Kintinuous, ElasticFusion, InfiniTAM, and BundleFusion are born on this basis. These methods adopt an incremental input mode. As the input data increases, the computational amount of real-time three-dimensional scene reconstruction will increase, resulting in a low efficiency of real-time three-dimensional scene reconstruction. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a real-time three-dimensional scene reconstruction method, device and equipment based on grid local fusion, which can significantly reduce the computational amount of real-time three-dimensional scene reconstruction, thereby effectively improving the efficiency of real-time three-dimensional scene reconstruction.

[0005] In a first aspect, an embodiment of the present invention provides a real-time three-dimensional scene reconstruction method based on grid local fusion, including:

[0006] During the execution of the real-time three-dimensional scene reconstruction task, if the current frame of point cloud data is received, then based on the current frame of point cloud data, the global point cloud data is divided to obtain an old global point cloud data part and a new global point cloud data part; wherein, the global point cloud data is generated based on the initial frame of point cloud data of the real-time three-dimensional scene reconstruction task to the point cloud data of the previous frame corresponding to the current frame of point cloud data;

[0007] Determine the point type to which each point in the global point cloud data belongs according to the old global point cloud data part and the new global point cloud data part; and construct a local mesh based on the current frame point cloud data and the new global point cloud data part; wherein the point types include old points, connection points, and new points;

[0008] Based on the point type to which each point in the global point cloud data belongs, fuse the local mesh into the global mesh corresponding to the global point cloud data to obtain the new global point cloud data and its corresponding new global mesh until the three-dimensional scene real-time reconstruction task ends.

[0009] In one implementation, based on the current frame point cloud data, divide the global point cloud data to obtain the old global point cloud data part and the new global point cloud data part, including:

[0010] Use the axis-aligned bounding box corresponding to the current frame point cloud data as the first bounding box, and expand the first bounding box according to a preset resolution to obtain the second bounding box;

[0011] Divide the point cloud outside the first bounding box in the global point cloud data into the old global point cloud data part; and divide the point cloud inside the second bounding box in the global point cloud data into the new global point cloud data part.

[0012] In one implementation, according to the old global point cloud data part and the new global point cloud data part, determine the point type to which each point in the global point cloud data belongs, including:

[0013] Determine the overlapping part between the old global point cloud data part and the new global point cloud data part;

[0014] Mark the points outside the overlapping part in the old global point cloud data part as old points; and mark the points outside the overlapping part in the new global point cloud data part as new points; and mark the points inside the overlapping part as connection points.

[0015] In one implementation, based on the point type to which each point in the global point cloud data belongs, fuse the local mesh into the global mesh corresponding to the global point cloud data to obtain the new global point cloud data and its corresponding new global mesh, including:

[0016] Remove the new points and the triangular patches where the new points are located in the global mesh corresponding to the global point cloud data to obtain the global mesh after removal, and update the vertex identifiers corresponding to each point in the global mesh after removal;

[0017] Based on the updated vertex identifiers corresponding to each point in the global mesh after removal, update the vertex identifiers corresponding to each point in the local mesh;

[0018] Determine the correspondence between the connection points among the global mesh after culling and the local mesh, and update the vertex identifiers corresponding to the connection points according to the correspondence;

[0019] Based on the updated vertex identifiers of the connection points, fuse the local mesh into the global mesh after culling to obtain new global point cloud data and its corresponding new global mesh.

[0020] In one implementation, cull the new points and the triangular patches where the new points are located in the global mesh corresponding to the global point cloud data to obtain the global mesh after culling, and update the vertex identifier corresponding to each point in the global mesh after culling, including:

[0021] Set the initial counter value to 0;

[0022] Traverse the points in the global mesh corresponding to the global point cloud data;

[0023] When the currently traversed point is not a new point, increment the counter value by 1; when the currently traversed point is a new point, keep the counter value unchanged; replace the vertex identifier corresponding to the currently traversed point with the updated counter value;

[0024] Construct a mapping table based on the counter value; wherein, the mapping table includes the updated vertex identifiers of each point in the global mesh;

[0025] According to the mapping table, cull the points with the vertex identifier being the specified value and the triangular patches where the points are located from the global mesh to obtain the global mesh after culling and the updated vertex identifiers of each point in the global mesh after culling.

[0026] In one implementation, update the vertex identifiers corresponding to each point in the local mesh based on the updated vertex identifiers of each point in the global mesh after culling, including:

[0027] Count the number of points included in the global mesh after culling;

[0028] For any point in the local mesh, use the sum value between the vertex identifier corresponding to the point in the local mesh and the number of points as the updated vertex identifier corresponding to the point in the local mesh.

[0029] In one implementation, the method further includes:

[0030] During the execution of the three-dimensional scene real-time reconstruction task, when a loop is detected, fuse the point cloud data from the initial frame to the current frame of the three-dimensional scene real-time reconstruction task, perform voxel downsampling on the fused point cloud data, and perform point cloud triangular mesh reconstruction on the point cloud data after voxel downsampling to achieve loop correction.

[0031] Second aspect, the present invention further provides a three-dimensional scene real-time reconstruction device based on grid local fusion, including:

[0032] A point cloud division module, configured to divide the global point cloud data into an old global point cloud data part and a new global point cloud data part based on the current frame point cloud data during the execution of the three-dimensional scene real-time reconstruction task; wherein, the global point cloud data is generated based on the initial frame point cloud data of the three-dimensional scene real-time reconstruction task to the point cloud data of the previous frame corresponding to the current frame point cloud data;

[0033] A point type and local grid determination module, configured to determine the point type to which each point in the global point cloud data belongs according to the old global point cloud data part and the new global point cloud data part; and construct a local grid according to the current frame point cloud data and the new global point cloud data part; wherein, the point types include old points, connection points, and new points;

[0034] A grid fusion module, configured to fuse the local grid into the global grid corresponding to the global point cloud data based on the point type to which each point in the global point cloud data belongs, so as to obtain new global point cloud data and its corresponding new global grid until the three-dimensional scene real-time reconstruction task ends.

[0035] Third aspect, the present invention further provides an electronic device, including a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of the first aspect.

[0036] Fourth aspect, the present invention further provides a computer-readable storage medium, the computer-readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to implement the method according to any one of the first aspect.

[0037] A three-dimensional scene real-time reconstruction method, device and equipment based on grid local fusion provided by the present invention, in the process of performing the three-dimensional scene real-time reconstruction task, if the current frame point cloud data is received, the global point cloud data is divided based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part. The global point cloud data is generated based on the initial frame point cloud data of the three-dimensional scene real-time reconstruction task to the point cloud data of the previous frame corresponding to the current frame point cloud data; then, according to the old global point cloud data part and the new global point cloud data part, the point type to which each point in the global point cloud data belongs is determined, and the point types include old points, connection points and new points; then a local grid is constructed according to the current frame point cloud data and the new global point cloud data part; finally, based on the point type to which each point in the global point cloud data belongs, the local grid is fused into the global grid corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global grid until the three-dimensional scene real-time reconstruction task ends. The above method uses the current frame point cloud data to divide the global point cloud data into an old global point cloud data part and a new global point cloud data part, thereby determining the point type to which each point in the global point cloud data belongs and constructing a local grid, and then fusing the local grid with the global grid based on the point type to which each point in the global point cloud data belongs. Compared with the prior art method of first updating the global point cloud data using the current frame point cloud data and then updating the global grid using the updated global point cloud data, the present invention can significantly reduce the computational amount of three-dimensional scene real-time reconstruction by fusing the local grid into the global grid, thereby effectively improving the efficiency of three-dimensional scene real-time reconstruction.

[0038] Other features and advantages of the present invention will be described in the following specification, and in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention are achieved and obtained by the structures specifically pointed out in the specification, claims and drawings.

[0039] To make the above objectives, features and advantages of the present invention more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0041] Figure 1 It is a schematic flow chart of a three-dimensional scene real-time reconstruction method based on grid local fusion provided by an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of point cloud data provided by an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of a mapping table provided by an embodiment of the present invention;

[0044] Figure 4 Technical framework diagram of real-time 3D scene reconstruction based on grid local fusion provided by an embodiment of the present invention;

[0045] Figure 5 Schematic structural diagram of a device for real-time 3D scene reconstruction based on grid local fusion provided by an embodiment of the present invention;

[0046] Figure 6 Schematic structural diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0048] Currently, such methods adopt an incremental input mode. As the input data increases, not only does the efficiency of real-time 3D scene reconstruction become lower, but the speed of model loop correction also becomes slower and slower. Moreover, these methods also have a common problem that they do not provide a relatively friendly processing interface for real-time models, resulting in being restricted by the native framework in object model presentation and great difficulty in framework engineering applications.

[0049] Based on this, the embodiments of the present invention provide a method, device, and equipment for real-time 3D scene reconstruction based on grid local fusion, which can significantly reduce the computational amount of real-time 3D scene reconstruction, thereby effectively improving the efficiency of real-time 3D scene reconstruction. It also solves the problems of fast loop and output of real-time scan models, reduces the influence of the number of frames on loop correction during the reconstruction process, and the reconstructed grid model can be output in real time for other post-processing operations.

[0050] To facilitate the understanding of this embodiment, first, a method for real-time 3D scene reconstruction based on grid local fusion disclosed in the embodiments of the present invention will be introduced in detail. The embodiments of the present invention mainly perform real-time modeling based on RGB-D image data. Refer to Figure 1 The flowchart of a method for real-time 3D scene reconstruction based on grid local fusion shown, and this method mainly includes the following steps S102 to step S106:

[0051] Step S102, during the execution of the three-dimensional scene real-time reconstruction task, if the current frame point cloud data is received, based on the current frame point cloud data, the global point cloud data is divided to obtain an old global point cloud data part and a new global point cloud data part.

[0052] Among them, the current frame point cloud data is obtained based on the RGB-D data of the real-time scanned image received at the current moment; the global point cloud data is generated based on the initial frame point cloud data of the three-dimensional scene real-time reconstruction task to the previous frame point cloud data corresponding to the current frame point cloud data. For example, by performing fusion processing and voxel downsampling processing on the initial frame point cloud data to the previous frame point cloud data, the global point cloud data can be obtained; the old global point cloud data part is the point cloud outside the axisymmetric bounding box of the current frame point cloud data in the global point cloud data, and the new global point cloud data part is the point cloud within the specified range of the current frame point cloud data in the global point cloud data, and this specified range is positively correlated with the size of the axisymmetric bounding box.

[0053] In one example, when the RGB-D data of a new real-time scanned image is received, the corresponding current frame point cloud data and its axisymmetric bounding box are generated. Based on the axisymmetric bounding box, a first bounding box and a second bounding box are respectively constructed. The first bounding box is the axisymmetric bounding box, and the second bounding box is the bounding box after the axisymmetric bounding box is enlarged by a specified multiple, which is equivalent to the aforementioned specified range; the global point cloud data is divided into an old global point cloud data part and a new global point cloud data part by using the first bounding box and the second bounding box.

[0054] Step S104, according to the old global point cloud data part and the new global point cloud data part, determine the point type to which each point in the global point cloud data belongs; and, construct a local mesh according to the current frame point cloud data and the new global point cloud data part.

[0055] Among them, the point types include old points, connection points, and new points. New points can be understood as the points existing in the current frame point cloud data, old points can be understood as the points that do not exist in the current frame point cloud data and are not on the same triangular patch as the new points, and connection points can be understood as the points that do not exist in the current frame point cloud data and are on the same triangular patch as the new points. In one example, the point type to which each point in the global point cloud data belongs can be determined according to the overlapping part between the old global point cloud data part and the new global point cloud data part.

[0056] In one example, the current frame point cloud data can be merged with the new global point cloud data part, the merged point cloud data is sampled to obtain an updated point cloud, and the local mesh can be obtained by performing triangular mesh reconstruction on the updated point cloud.

[0057] Step S106: Based on the point type to which each point in the global point cloud data belongs, fuse the local grid into the global grid corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global grid until the real-time 3D scene reconstruction task ends.

[0058] Among them, the global grid is also the grid obtained by performing triangular mesh reconstruction on the global point cloud data. In one example, each point in the global point cloud data can be traversed to assign corresponding vertex identifiers to each point according to the point type to which each point belongs, so as to generate a mapping table corresponding to the global point cloud data. The new points in the global grid and the triangular patches where the new points are located are removed according to the mapping table to obtain the global grid after removing the new points and the triangular patches where the new points are located, simply referred to as the global grid after removal; based on the vertex identifiers corresponding to each point in the global grid after removal, re-determine the vertex identifiers corresponding to each point in the local grid; based on finding the corresponding relationship between the connection points in the global grid after removal and the local grid, update the vertex identifiers corresponding to the connection points according to this corresponding relationship; based on the updated vertex identifiers of the connection points, splice the global grid after removal and the local grid to obtain new global point cloud data and its corresponding new global grid until the real-time 3D scene reconstruction task ends.

[0059] The real-time 3D scene reconstruction method based on grid local fusion provided by the embodiments of the present invention divides the global point cloud data into an old global point cloud data part and a new global point cloud data part by using the current frame point cloud data, thereby determining the point type to which each point in the global point cloud data belongs and constructing a local grid. Furthermore, based on the point type to which each point in the global point cloud data belongs, the local grid is fused with the global grid. Compared with the prior art method of first updating the global point cloud data by using the current frame point cloud data and then updating the global grid by using the updated global point cloud data, the embodiments of the present invention can significantly reduce the computational amount of real-time 3D scene reconstruction by fusing the local grid into the global grid, thereby effectively improving the efficiency of real-time 3D scene reconstruction.

[0060] For the convenience of understanding, the embodiments of the present invention provide a specific implementation manner of a real-time 3D scene reconstruction method based on grid local fusion. The overall process of this method is as follows:

[0061] Step 1: Collect the RGB-D data of the real-time scanned image through the device and generate the corresponding point cloud data. The point cloud data of the i-th frame is recorded as curPointCloud[i]. Among them, the point cloud data of the i-th frame is also the point cloud data of the previous frame.

[0062] Step 2: The camera pose of the i-th frame point cloud data corresponding to the world coordinate system is recorded as curPose[i]. The camera pose of each frame is obtained by registering with the point cloud data of the previous frame.

[0063] Step 3: Traverse all point cloud frames from the 1st frame to the i-th frame, fuse them into a single point cloud data in the world coordinate system, and then sample the fused point cloud data according to a specific resolution using the voxel downsampling algorithm to generate the global point cloud data, which is recorded as globalPointCloud. Among them, the point cloud data of the first frame is also the initial frame point cloud data.

[0064] Step 4: Perform point cloud triangulation mesh reconstruction on the global point cloud data globalPointCloud to obtain the global mesh, which is recorded as globalMesh.

[0065] Step 5: Based on the current frame point cloud data, divide the global point cloud data into an old global point cloud data part and a new global point cloud data part. For the specific implementation, refer to Figure 2 the schematic diagram of a kind of point cloud data shown. In the left figure, the circular black dots are the global point cloud data, the triangular points are the point cloud data of the current frame (the (i + 1)-th frame), the gray dashed box is the first bounding box Box1, the black dashed box is the second bounding box Box2, and the distance from Box1 to Box2 is twice the global point cloud resolution; in the right figure, the circular black dots are the old points, the circular dark gray dots are the connection points, and the circular light gray dots are the new points.

[0066] Specifically, it includes the following steps 5.1 to 5.2:

[0067] Step 5.1: Take the axis-aligned bounding box corresponding to the current frame point cloud data as the first bounding box, and expand the first bounding box according to the preset resolution to obtain the second bounding box. In an example, when generating the current frame point cloud data, such as the (i + 1)-th frame point cloud data, calculate the axis-aligned bounding box of this frame, take the axis-aligned bounding box as the first bounding box Box1, and at the same time expand the first bounding box Box1 by a distance of twice the point cloud resolution to obtain the second bounding box Box2, as Figure 2 shown.

[0068] Step 5.2: Divide the point cloud data in the global point cloud data that is outside the first bounding box into the old global point cloud data part; and divide the point cloud data in the global point cloud data that is inside the second bounding box into the new global point cloud data part. In an example, use the first bounding box Box1 to filter the global point cloud data globalPointCloud into two parts, and take the point cloud outside the box as the old global point cloud data part old_globalPointCloud. Similarly, use the second bounding box Box2 to filter the global point cloud data globalPointCloud into two parts, and take the point cloud inside the box as the new global point cloud data part new_globalPointCloud.

[0069] Step 6. Determine the point type to which each point in the global point cloud data belongs based on the old global point cloud data part and the new global point cloud data part. Specifically, it includes the following steps 6.1 to 6.2:

[0070] Step 6.1. Determine the overlapping part between the old global point cloud data part old_globalPointCloud and the new global point cloud data part new_globalPointCloud.

[0071] Step 6.2. Mark the points in the old global point cloud data part that are outside the overlapping part as old points; and mark the points in the new global point cloud data part that are outside the overlapping part as new points; and mark the points within the overlapping part as connection points. Please continue to refer to Figure 2 , where the old points are the points within the non-overlapping part of the old global point cloud data part old_globalPointCloud, the new points are the points within the non-overlapping part of the new global point cloud data part new_globalPointCloud, and the connection points are the points within the overlapping part between the old global point cloud data part old_globalPointCloud and the new global point cloud data part new_globalPointCloud.

[0072] Step 7. Construct a local mesh based on the current frame point cloud data and the new global point cloud data part. In one example, merge the (i + 1)-th frame point cloud data with the new global point cloud data part new_globalPointCloud, and at the same time sample at a specific resolution to obtain the updated point cloud updatePointCloud of the new frame and the global frame. Perform triangular mesh reconstruction on the updated point cloud updatePointCloud to obtain the local mesh updateMesh.

[0073] Step 8. Based on the point type to which each point in the global point cloud data belongs, fuse the local mesh into the corresponding global mesh of the global point cloud data to obtain the new global point cloud data and its corresponding new global mesh. Specifically, it includes the following steps 8.1 to 8.4:

[0074] Step 8.1. Remove the new points and the triangular patches where the new points are located in the global mesh corresponding to the global point cloud data to obtain the global mesh after removal, and update the vertex identifiers corresponding to each point in the global mesh after removal. In specific implementation, the following (a) to (e) can be referred to:

[0075] (a) Set the initial counter value counter to 0.

[0076] (b) Traverse the points in the global mesh corresponding to the global point cloud data. In one example, the points in the global point cloud data globalMesh can be traversed in sequence from the first point to the last point through a pointer.

[0077] (c) When the currently traversed point is not a new point, increment the counter value by 1; when the currently traversed point is a new point, keep the counter value unchanged. In one example, when the current point pointed to by the pointer is a reserved point, the counter value counter is incremented by one, otherwise the counter value counter remains unchanged. Here, the reserved points are the old points and the connecting points.

[0078] (d) Construct a mapping table based on the counter value; the mapping table includes the updated vertex identifiers (ids) of each point in the global mesh. In one example, the calculation logic of the mapping table is: when the point pointed to by the pointer does not need to be deleted, the vertex id is counter, otherwise the vertex id is -1. When the pointer completes one traversal, the mapping table is constructed.

[0079] (e) According to the mapping table, remove the points with the vertex identifier being a specified value and the triangular patches where the points are located from the global mesh to obtain the global mesh after removal and the updated vertex identifiers of each point in the global mesh after removal. In one example, traverse all triangular patches, update the vertex ids according to the mapping table, and directly delete the triangles where the mapping table has -1 and the point type is a new point. Note that at this time, the vertex ids of all triangular patches for the connecting points are incorrect and need to be recalculated in step 8.3.

[0080] Exemplarily, refer to Figure 3 the schematic diagram of a mapping table as shown. Assume there is a triangular mesh (left column) composed of five faces and five points, and it is necessary to delete point 2 and point 4, construct the mapping table (middle column), and delete point 2 and 4 according to the mapping table and update the vertex ids of the triangular patches to obtain a new triangular mesh (right column). Here, the new triangular mesh is the global mesh after removal.

[0081] Step 8.2, based on the updated vertex identifiers of each point in the global mesh after removal, update the vertex identifiers corresponding to each point in the local mesh. In one example, count the number of points included in the global mesh after removal; for any point in the local mesh, use the sum value between the vertex identifier corresponding to the point in the local mesh and the number of points as the updated vertex identifier corresponding to the point in the local mesh. For example, count the number of remaining points after deleting the new points in the global mesh globalMesh, denoted as n, and add n to the vertex ids of all triangles in the local mesh updateMesh.

[0082] Step 8.3, determine the correspondence between the connection points among the global mesh and the local mesh after culling, so as to update the vertex identifiers corresponding to the connection points according to the correspondence. In one example, traverse all the connection points in the global mesh globalMesh, find the vertex id of the point in the local mesh updateMesh corresponding to it through coordinate matching, and assign the id of this point in the local mesh updateMesh to this point in the global mesh globalMesh to complete the update of the connection point id.

[0083] Step 8.4, based on the updated vertex identifiers of the connection points, fuse the local mesh into the global mesh after culling to obtain the new global point cloud data and its corresponding new global mesh. In one example, directly push the points of the local mesh updateMesh behind the points of the global mesh globalMesh after culling, and add the triangular mesh of the local mesh updateMesh to the global mesh globalMesh after culling.

[0084] Step 9, during the execution of the 3D scene real-time reconstruction task, in the case of detecting a loop, fuse the point cloud data from the initial frame to the current frame of the 3D scene real-time reconstruction task, perform voxel downsampling on the fused point cloud data, and perform point cloud triangular mesh reconstruction on the point cloud data after voxel downsampling to achieve loop correction. In one example, when the overall system is running, the global mesh globalMesh will be displayed in real time, and an independent thread will be established to detect loops. When a loop is detected, directly update the position of the corresponding frame, and then use Step 3 and Step 4 to complete loop correction. Since Step 3 and Step 4 only simply add and downsample the point cloud, the computational cost is very low, so the loop correction time of the system is basically not affected by the number of frames.

[0085] In summary, the embodiments of the present invention at least have the following characteristics: (1) Use a mapping table to delete the points and patches of the triangular mesh. Usually, deleting a point in the mesh will disrupt the vertex ids of all triangles, resulting in a disordered mesh. However, using the mapping table method, deleting points and updating vertex ids can be achieved with a time complexity of O(n). (2) Local fusion of the mesh. Usually, updating the global mesh requires first updating the global point cloud data and then recalculating the mesh of the global point cloud data, which has a very large amount of calculation. Local mesh fusion only needs to update the local point cloud and perform local meshing, and then splice the local mesh to the global mesh, with a very small amount of calculation.

[0086] Furthermore, the embodiments of the present invention also provide as Figure 4The technical framework diagram of a real-time 3D scene reconstruction based on grid local fusion is shown as follows, including: collecting RGB-D data, generating the point cloud data of the current frame and calculating the point cloud position, and saving it to the point cloud container storing all frames; calculating the global point cloud data and the global grid based on the initial frame point cloud data to the previous frame point cloud data stored in the point cloud container, performing grid local fusion in combination with the current frame point cloud data and outputting the real-time grid; in addition, performing loop detection and loop correction based on the current frame point cloud data, and saving the loop-corrected point cloud data to the point cloud container. Compared with the current real-time reconstruction algorithm based on the TSDF algorithm, the real-time 3D scene reconstruction method based on grid local fusion provided by the embodiments of the present invention has obvious advantages in the loop correction speed, and is more flexible in the presentation and post-processing of the reconstruction results, which is more conducive to the implementation of the algorithm in engineering applications.

[0087] On the basis of the foregoing embodiments, the embodiments of the present invention provide a real-time 3D scene reconstruction device based on grid local fusion, see Figure 5 The structural schematic diagram of a real-time 3D scene reconstruction device based on grid local fusion is shown as follows. The device mainly includes the following parts:

[0088] The point cloud division module 502 is used to divide the global point cloud data into an old global point cloud data part and a new global point cloud data part based on the current frame point cloud data during the execution of the real-time 3D scene reconstruction task; wherein, the global point cloud data is generated based on the initial frame point cloud data to the previous frame point cloud data corresponding to the current frame point cloud data of the 3D scene reconstruction task;

[0089] The point type and local grid determination module 504 is used to determine the point type to which each point in the global point cloud data belongs according to the old global point cloud data part and the new global point cloud data part; and construct a local grid according to the current frame point cloud data and the new global point cloud data part; wherein, the point types include old points, connection points and new points;

[0090] The grid fusion module 506 is used to fuse the local grid into the global grid corresponding to the global point cloud data based on the point type to which each point in the global point cloud data belongs, so as to obtain the new global point cloud data and its corresponding new global grid until the real-time 3D scene reconstruction task ends.

[0091] The three-dimensional scene real-time reconstruction device based on grid local fusion provided by the embodiment of the present invention divides the global point cloud data into an old global point cloud data part and a new global point cloud data part by using the current frame point cloud data, thereby determining the point type to which each point in the global point cloud data belongs and constructing a local grid, and then fusing the local grid with the global grid based on the point type to which each point in the global point cloud data belongs. Compared with the prior art method of first updating the global point cloud data by using the current frame point cloud data and then updating the global grid by using the updated global point cloud data, the embodiment of the present invention can significantly reduce the computational amount of three-dimensional scene real-time reconstruction by fusing the local grid into the global grid, thereby effectively improving the efficiency of three-dimensional scene real-time reconstruction.

[0092] In one implementation manner, the point cloud division module 502 is specifically configured to:

[0093] Use the axis-aligned bounding box corresponding to the current frame point cloud data as the first bounding box, and perform an expansion process on the first bounding box according to a preset resolution to obtain a second bounding box;

[0094] Divide the point cloud outside the first bounding box in the global point cloud data into the old global point cloud data part; and divide the point cloud inside the second bounding box in the global point cloud data into the new global point cloud data part.

[0095] In one implementation manner, the point type and local grid determination module 504 is specifically configured to:

[0096] Determine the overlapping part between the old global point cloud data part and the new global point cloud data part;

[0097] Mark the points outside the overlapping part in the old global point cloud data part as old points; and mark the points outside the overlapping part in the new global point cloud data part as new points; and mark the points inside the overlapping part as connection points.

[0098] In one implementation manner, the grid fusion module 506 is specifically configured to:

[0099] Remove the new points and the triangular patches where the new points are located in the global grid corresponding to the global point cloud data to obtain a global grid after removal, and update the vertex identifier corresponding to each point in the global grid after removal;

[0100] Update the vertex identifier corresponding to each point in the local grid based on the updated vertex identifier of each point in the global grid after removal;

[0101] Determine the corresponding relationship between the connection points in the global grid after removal and the local grid, so as to update the vertex identifier corresponding to the connection points according to the corresponding relationship;

[0102] Based on the vertex identifiers updated at the connection points, fuse the local meshes into the global mesh after culling to obtain new global point cloud data and its corresponding new global mesh.

[0103] In one implementation, the mesh fusion module 506 is specifically configured to:

[0104] Set the initial counter value to 0;

[0105] Traverse the points in the global mesh corresponding to the global point cloud data;

[0106] When the currently traversed point is not a new point, increment the counter value by 1; when the currently traversed point is a new point, keep the counter value unchanged; use the updated counter value to replace the vertex identifier corresponding to the currently traversed point;

[0107] Construct a mapping table based on the counter value; wherein, the mapping table includes the updated vertex identifiers of each point in the global mesh;

[0108] According to the mapping table, cull the points with the vertex identifier being a specified value and the triangular patches where the points are located from the global mesh to obtain the global mesh after culling and the updated vertex identifiers of each point in the global mesh after culling.

[0109] In one implementation, the mesh fusion module 506 is specifically configured to:

[0110] Count the number of points included in the global mesh after culling;

[0111] For any point in the local mesh, use the sum value between the vertex identifier corresponding to the point in the local mesh and the number of points as the updated vertex identifier corresponding to the point in the local mesh.

[0112] In one implementation, it further includes a loop closure correction module, which is used for:

[0113] During the execution of the three-dimensional scene real-time reconstruction task, when a loop closure is detected, fuse the initial frame point cloud data to the current frame point cloud data of the three-dimensional scene real-time reconstruction task, perform voxel downsampling on the fused point cloud data, and perform point cloud triangular mesh reconstruction on the point cloud data after voxel downsampling to achieve loop closure correction.

[0114] The device provided by the embodiments of the present invention has the same implementation principle and the same technical effects as those of the foregoing method embodiments. For the sake of brief description, for the parts not mentioned in the device embodiments, reference may be made to the corresponding content in the foregoing method embodiments.

[0115] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device; a computer program is stored on the storage device, and when the computer program is run by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0116] Figure 6 FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes: a processor 60, a memory 61, a bus 62, and a communication interface 63. The processor 60, the communication interface 63, and the memory 61 are connected through the bus 62; the processor 60 is configured to execute an executable module stored in the memory 61, such as a computer program.

[0117] Among them, the memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 63 (which may be wired or wireless), a communication connection between the system network element and at least one other network element is realized, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0118] The bus 62 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 6 only a bidirectional arrow is used in FIG. 4, but it does not mean that there is only one bus or one type of bus.

[0119] Among them, the memory 61 is used to store a program. After receiving an execution instruction, the processor 60 executes the program. The method executed by the device defined by the flow process disclosed in any one of the foregoing embodiments of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0120] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 60 or the instructions in the form of software. The above-mentioned processor 60 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP for short), an application specific integrated circuit (ASIC for short), a field-programmable gate array (FPGA for short), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present invention can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and combines its hardware to complete the steps of the above method.

[0121] The computer program product of the readable storage medium provided by the embodiments of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the foregoing method embodiments. For the specific implementation, reference can be made to the foregoing method embodiments and will not be elaborated herein.

[0122] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0123] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for real-time reconstruction of a three-dimensional scene based on local grid fusion, characterized in that: include: In the process of performing the three-dimensional scene real-time reconstruction task, if the current frame point cloud data is received, the global point cloud data is divided based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part; wherein the global point cloud data is generated based on the previous frame point cloud data corresponding to the initial frame point cloud data of the three-dimensional scene real-time reconstruction task to the current frame point cloud data; Determine the point type of each point in the global point cloud data according to the old global point cloud data portion and the new global point cloud data portion; and construct a local grid according to the current frame point cloud data and the new global point cloud data portion; wherein the point types include old points, connection points and new points; Based on the point type to which each point in the global point cloud data belongs, the local grid is fused into the global grid corresponding to the global point cloud data to obtain new global point cloud data and its corresponding new global grid until the real-time reconstruction task of the three-dimensional scene is completed.

2. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 1, characterized in that: Based on the current frame point cloud data, the global point cloud data is divided to obtain an old global point cloud data part and a new global point cloud data part, including: Taking the axis-aligned bounding box corresponding to the point cloud data of the current frame as the first bounding box, and enlarging the first bounding box according to a preset resolution to obtain a second bounding box; The point cloud located outside the first bounding box in the global point cloud data is divided into the old global point cloud data part; and the point cloud located within the second bounding box in the global point cloud data is divided into the new global point cloud data part.

3. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 1, characterized in that: Determining the point type of each point in the global point cloud data according to the old global point cloud data portion and the new global point cloud data portion, including: determining an overlap between the old global point cloud data portion and the new global point cloud data portion; The points in the old global point cloud data portion that are outside the overlapping portion are marked as old points; and the points in the new global point cloud data portion that are outside the overlapping portion are marked as new points; and the points in the overlapping portion are marked as connection points.

4. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 1, characterized in that: Based on the point type to which each point in the global point cloud data belongs, the local grid is merged into the global grid corresponding to the global point cloud data to obtain new global point cloud data and the new global grid corresponding thereto, including: Eliminate the new point and the triangular facet where the new point is located in the global mesh corresponding to the global point cloud data to obtain a global mesh after elimination, and update the vertex identifier corresponding to each point in the global mesh after elimination; Based on the updated vertex identifier of each point in the global mesh after the elimination, updating the vertex identifier corresponding to each point in the local mesh; Determine the correspondence between the connection point and the global mesh after the elimination, and update the vertex identifier corresponding to the connection point according to the correspondence; Based on the updated vertex identifiers of the connection points, the local grid is merged into the eliminated global grid to obtain new global point cloud data and its corresponding new global grid.

5. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 4, characterized in that: Eliminating the new point and the triangular facet where the new point is located in the global mesh corresponding to the global point cloud data to obtain a global mesh after elimination, and updating the vertex identifier corresponding to each point in the global mesh after elimination, including: Set the initial counter value to 0; Traversing points in a global grid corresponding to the global point cloud data; If the currently traversed point is not the new point, the counter value is increased by 1; if the currently traversed point is the new point, the counter value is kept unchanged; and the vertex identifier corresponding to the currently traversed point is replaced by the updated counter value; Constructing a mapping table based on the counter value; wherein the mapping table includes the updated vertex identifier of each point in the global grid; According to the mapping table, the points whose vertex identifiers are specified values ​​and the triangles where the points are located are removed from the global mesh to obtain a post-elimination global mesh and the updated vertex identifiers of each point in the post-elimination global mesh.

6. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 4, characterized in that: Based on the updated vertex identifier of each point in the global mesh after the elimination, updating the vertex identifier corresponding to each point in the local mesh comprises: Counting the number of points contained in the global grid after the elimination; For any point in the local grid, the sum of the vertex identifier corresponding to the point in the local grid and the number of points is used as the updated vertex identifier corresponding to the point in the local grid.

7. The method for real-time reconstruction of a three-dimensional scene based on local grid fusion according to claim 1, characterized in that: The method further comprises: In the process of performing a three-dimensional scene real-time reconstruction task, when a loop is detected, the initial frame point cloud data of the three-dimensional scene real-time reconstruction task is fused with the current frame point cloud data, voxel downsampling is performed on the fused point cloud data, and point cloud triangular mesh is reconstructed on the voxel downsampling point cloud data to achieve loop correction.

8. A three-dimensional scene real-time reconstruction device based on local grid fusion, characterized in that: include: A point cloud division module is used for dividing the global point cloud data based on the current frame point cloud data to obtain an old global point cloud data part and a new global point cloud data part when receiving the current frame point cloud data during the process of performing the real-time reconstruction task of the three-dimensional scene. The global point cloud data is generated based on the previous frame point cloud data corresponding to the initial frame point cloud data of the real-time reconstruction task of the three-dimensional scene to the current frame point cloud data. A point type and local grid determination module, used to determine the point type of each point in the global point cloud data according to the old global point cloud data part and the new global point cloud data part; and to construct a local grid according to the current frame point cloud data and the new global point cloud data part; wherein the point types include old points, connection points and new points; A grid fusion module is used to fuse the local grid into the global grid corresponding to the global point cloud data based on the point type to which each point in the global point cloud data belongs, so as to obtain new global point cloud data and its corresponding new global grid until the real-time reconstruction task of the three-dimensional scene is completed.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images