Dynamic Scene Real-Time Reconstruction Method, Navigation Method, Device, Equipment and Medium

Abstract

The present application discloses a dynamic scene real-time reconstruction method, a navigation method, a device, a device and a medium. The reconstruction method includes collecting point cloud data of a dynamic scene through a lidar; performing mesh reconstruction on the point cloud data to obtain single-frame mesh data; determining a single-frame distance field based on the single-frame mesh data, and updating a global distance field based on the single-frame distance field; determining scene space markers based on the updated global distance field, and removing dynamic occupied spaces based on the global distance field before updating and the single-frame distance field to obtain static space markers corresponding to the dynamic spaces; constructing a scene model based on the static space markers. The present application first performs single-frame reconstruction on the point cloud data of the dynamic scene collected by the lidar, then determines a single-frame distance field based on the single-frame reconstruction data, and fuses the single-frame distance fields for multi-frame reconstruction, making full use of the point cloud data captured by the lidar, and immediately converting the discrete points scanned by the lidar into continuous regions for surface reconstruction.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics, and particularly relates to a method and device for real-time reconstruction of dynamic scenes, a navigation method, a device, equipment and a medium. Background Art

[0002] In recent years, as one of the most widely used three-dimensional space sensors, Light Detection and Ranging (LiDAR) has received significant attention in outdoor scene acquisition. Compared with vision solutions, the LiDAR-alone solution has been proven to be more effective in environments with weak textures and low illumination, which can improve the reconstruction effect of outdoor scenes. However, in the existing methods for scene reconstruction using point cloud data collected by LiDAR, since real-time LiDAR scanning cannot directly present the spatial shape content, it is impossible to timely convert the discrete points scanned by LiDAR into continuous regions for surface reconstruction and three-dimensional mapping, and thus the timeliness of scene reconstruction cannot be guaranteed.

[0003] Therefore, the existing technology still needs to be improved. Summary of the Invention

[0004] The technical problem to be solved by the present application is to provide a method and device for real-time reconstruction of dynamic scenes, a navigation method, a device, equipment and a medium in view of the deficiencies of the existing technology.

[0005] To solve the above technical problem, in the first aspect of the present application, a method for real-time reconstruction of dynamic scenes is provided, wherein the method for real-time reconstruction of dynamic scenes specifically includes:

[0006] Collecting point cloud data of a dynamic scene by using a LiDAR;

[0007] Performing mesh reconstruction on the point cloud data to obtain single-frame mesh data corresponding to the point cloud data;

[0008] Determining a single-frame distance field based on the LiDAR line of sight based on the single-frame mesh data, and updating a global distance field based on the single-frame distance field, wherein the global distance field is initialized based on the single-frame distance field corresponding to the first-frame point cloud data;

[0009] Determining a spatial marker of the dynamic scene based on the updated global distance field, and removing the dynamically occupied space in the free space of the spatial marker based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene;

[0010] Constructing a scene model corresponding to the dynamic scene that does not include dynamic objects based on the free space of the static scene.

[0011] The described dynamic scene real-time reconstruction method, wherein the mesh reconstruction of the point cloud data to obtain the single-frame mesh data corresponding to the point cloud data specifically includes:

[0012] Construct a single-frame reconstruction mesh for the point cloud data;

[0013] Obtain the normal vector of each triangle in the single-frame reconstruction mesh, and determine the confidence weight of each triangular mesh based on the normal vector of each triangle;

[0014] Obtain multiple triangles corresponding to each point in the point cloud data, and determine the normal vector of each point in the point cloud data based on the normal vector and confidence weight of the obtained triangles;

[0015] Based on the normal vector of each point in the point cloud data and the single-frame reconstruction network, determine the single-frame mesh data corresponding to the point cloud data.

[0016] The described dynamic scene real-time reconstruction method, wherein the construction of the single-frame reconstruction mesh for the point cloud data specifically includes:

[0017] Divide the point cloud data into several non-overlapping sub-point cloud data;

[0018] Perform a radial transformation on each sub-point cloud data to obtain the reverse point set of each sub-point cloud data;

[0019] Calculate the convex hull of the union of the reverse point set of each sub-point cloud data and the viewing point of the point cloud data to obtain the single-frame reconstruction mesh of the point cloud data.

[0020] The described dynamic scene real-time reconstruction method, wherein the determination of the single-frame distance field based on the lidar line of sight based on the single-frame mesh data specifically includes:

[0021] Calculate the bounding box of the point cloud data, and construct an octree voxel set based on the bounding box;

[0022] Obtain the rays from the viewing point of the point cloud data to each point in the point cloud data, obtain all the voxels passing through the lidar line of sight in the octree voxel set, and obtain the intersections of the lidar line of sight and the single-frame meshes in the single-frame mesh data, wherein the voxels are initialized based on the single-frame mesh data, and the centroid, normal vector, and confidence weight of the triangle are recorded in the voxels;

[0023] Calculate the single-frame distance from each voxel to the single-frame mesh intersection, and determine that the perceptual space corresponding to the point cloud data is marked as free space, edge space, and unknown space based on the single-frame distance of the voxel to obtain the single-frame distance field based on the lidar line of sight.

[0024] The described real-time dynamic scene reconstruction method, wherein the specific process of updating the global distance field based on the single-frame distance field includes:

[0025] Initialize the global distance field based on the single-frame distance field of the first-frame point cloud data;

[0026] Search for the voxels in the single-frame distance field in the global distance field;

[0027] For the found voxels, update the global distance of the voxels in the global distance field based on the single-frame distance of the voxels in the single-frame distance field;

[0028] For the unfound voxels, add the voxels to the global distance field and use the single-frame distance of the voxels in the single-frame distance field as their global distance in the global distance field.

[0029] The described real-time dynamic scene reconstruction method, wherein the specific process of removing the dynamically occupied space in the free space of the spatial marker based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene includes:

[0030] Compare the voxels in the single-frame distance field with the voxels in the global distance field before update to select the dynamically occupied space;

[0031] Separate the selected dynamically occupied space from the free space in the spatial marker to obtain the free space of the static scene corresponding to the dynamic scene.

[0032] The second aspect of the present application provides a navigation method, which uses the real-time dynamic scene reconstruction method described above. The navigation method specifically includes:

[0033] Collect point cloud data through the lidar carried by the unmanned device, and reconstruct the scene model through the real-time dynamic scene reconstruction method;

[0034] Extract the free space in the scene model, and determine the path points of the unmanned device based on the free space to form the navigation path of the unmanned device.

[0035] The third aspect of the present application provides a real-time dynamic scene reconstruction device. The real-time dynamic scene reconstruction device specifically includes:

[0036] A collection module, used to collect point cloud data of a dynamic scene through a lidar;

[0037] A single-frame reconstruction module, used to perform grid reconstruction on the point cloud data to obtain the single-frame grid data corresponding to the point cloud data;

[0038] A multi-frame reconstruction module is configured to determine a single-frame distance field based on the lidar line of sight from the single-frame grid data, update a global distance field based on the single-frame distance field, determine spatial markings of the dynamic scene based on the updated global distance field, remove the dynamically occupied space in the free space of the spatial markings based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene, and construct a scene model corresponding to the dynamic scene without dynamic objects based on the free space of the static scene, wherein the global distance field is initialized based on the single-frame distance field corresponding to the first-frame point cloud data.

[0039] In a fourth aspect of the present application, there is provided a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps in any of the above-mentioned dynamic scene real-time reconstruction methods.

[0040] In a fifth aspect of the present application, there is provided a terminal device, which includes: a processor and a memory;

[0041] The memory stores a computer-readable program executable by the processor;

[0042] When the processor executes the computer-readable program, it implements the steps in any of the above-mentioned dynamic scene real-time reconstruction methods.

[0043] Advantageous effects: Compared with the prior art, the present application provides a dynamic scene real-time reconstruction method, a navigation method, a device, a device and a medium. The reconstruction method includes collecting point cloud data of a dynamic scene by a lidar; performing grid reconstruction on the point cloud data to obtain single-frame grid data corresponding to the point cloud data; determining a single-frame distance field based on the lidar line of sight from the single-frame grid data, and updating a global distance field based on the single-frame distance field; determining spatial markings of the dynamic scene based on the updated global distance field, and removing the dynamically occupied space in the free space of the spatial markings based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene; constructing a scene model corresponding to the dynamic scene without dynamic objects based on the free space of the static scene. The present application first performs single-frame reconstruction on the point cloud data of the dynamic scene collected by the lidar, and then fuses the single-frame distance field corresponding to the single-frame reconstruction data and the global distance field obtained by fusing all the forward point cloud data for multi-frame reconstruction, thus making full use of the point cloud data captured by the lidar, immediately converting the discrete points scanned by the lidar into a continuous region for surface reconstruction, and ensuring the timeliness of dynamic scene reconstruction. Description of the Drawings

[0044] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 It is a flowchart of the dynamic scene real-time reconstruction method provided by the embodiments of the present application.

[0046] Figure 2 It is a comparison chart of the normal vectors estimated by the embodiments of the present application and the normal vectors estimated by the existing methods.

[0047] Figure 3 It is a comparison chart of removing dynamic occupied voxels and not removing occupied voxels.

[0048] Figure 4 It is an example diagram of the reconstructed scene model.

[0049] Figure 5 It is an example diagram of the navigation path.

[0050] Figure 6 It is a principle block diagram of the dynamic scene real-time reconstruction device provided by the embodiments of the present application.

[0051] Figure 7 It is a principle block diagram of the terminal device provided by the embodiments of the present application. Specific embodiments

[0052] The embodiments of the present application provide a dynamic scene real-time reconstruction method, a navigation method, a device, a device and a medium. To make the purpose, technical solutions and effects of the present application clearer and more definite, the following further details the present application by way of examples with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of this application means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0054] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless specifically defined as here.

[0055] It should be understood that the sequence numbers and magnitudes of the steps in this embodiment do not mean the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0056] The following further illustrates the application content by describing the embodiments in conjunction with the accompanying drawings.

[0057] This embodiment provides a method for real-time reconstruction of a dynamic scene, as Figure 1 shown, the method includes:

[0058] S10. Collect point cloud data of the dynamic scene through a lidar.

[0059] Specifically, the point cloud data is obtained by scanning the dynamic scene at the current moment. For example, the lidar is mounted on a robot, and the robot scans the dynamic scene to obtain the point cloud data of the dynamic scene, etc. Among them, the lidar scans the dynamic scene by rotating the laser beam to capture three-dimensional point cloud data. Among them, the three-dimensional point cloud data is a point cloud in a ring structure, and the number of rings of the three-dimensional point cloud data corresponds to the number of laser beams emitted by the lidar.

[0060] S20. Perform mesh reconstruction on the point cloud data to obtain single-frame mesh data corresponding to the point cloud data;

[0061] Specifically, the single-frame grid data is obtained by performing network reconstruction on the point cloud data and is used to describe the internal space and boundaries of the point cloud data. Among them, the single-frame grid data includes a single-frame reconstruction network (i.e., a closed surface model) and normal vectors. That is to say, when performing grid reconstruction on the point cloud data, it is necessary to construct a single-frame reconstruction network of the point cloud data and the normal vector of each point in the point cloud data.

[0062] Based on this, the grid reconstruction of the point cloud data to obtain the single-frame grid data corresponding to the point cloud data specifically includes:

[0063] S21. Construct the single-frame reconstruction grid of the point cloud data;

[0064] S22. Obtain the normal vector of each triangle in the single-frame reconstruction grid, and determine the confidence weight of each triangular grid based on the normal vector of each triangle;

[0065] S23. Obtain multiple triangles corresponding to each point in the point cloud data, and determine the normal vector of each point in the point cloud data based on the normal vector and confidence weight of the obtained triangles.

[0066] Specifically, in step S21, first perform surface reconstruction on the point cloud data to obtain a single-frame reconstruction grid, that is, obtain a closed surface model of the point cloud data. Among them, the single-frame reconstruction grid can be obtained by performing surface reconstruction on the point cloud data. That is to say, after obtaining the point cloud data, surface reconstruction can be performed on the point cloud data to obtain the single-frame reconstruction grid of the point cloud data. Among them, when performing surface reconstruction on the point cloud data, the reverse point set of the point cloud data can be directly obtained, and then the convex hull of the union of the reverse point set and the viewing point can be calculated to perform surface reconstruction to obtain the single-frame reconstruction grid. Among them, the process of obtaining the reverse point set can be as follows:

[0067] For each point in the point cloud data , the point has coordinates in the local coordinate system centered on the viewing point (i.e., the lidar position) , which can be obtained through the odometer reading of the th frame, representing the three-dimensional coordinates of the viewing point. Then, place the viewing point at the center of the sphere and perform a radial transformation on the point cloud data . Through the kernel function, the point cloud data can be converted into a reverse point set . Among them, the reverse point of the reverse point set has the following expression:

[0068] ,

[0069] Among them, represents the reverse points of the point cloud , represents the viewing point of the lidar, which can be obtained through odometer readings, represents the radial transformation, represents the kernel function.

[0070] Furthermore, in the embodiments of the present application, taking as the origin of the local coordinate system of the point cloud , the expression of the reverse points is simplified to:

[0071] .

[0072] The kernel function can adopt a mirror mode, that is, the kernel function can be expressed as:

[0073] ,

[0074] Among them, represents the scaling factor, such as etc.

[0075] After calculating the reverse point set, based on the correspondence between the point cloud data and the reverse point set, a network connection of the point cloud data can be established based on the convex hull of the union of the reverse point set and the viewing point to construct a single-frame reconstruction grid of the point cloud data.

[0076] Furthermore, since the non-linear negativity of the convex hull calculation will slow down the reconstruction process as the number of points increases, which will in turn affect the real-time reconstruction speed of the dynamic scene. Based on this, in a typical implementation, when constructing the single-frame reconstruction grid of the point cloud data, the point cloud data can be divided into several non-overlapping sub-point cloud data, and the single-frame reconstruction grid of the point cloud data can be obtained by performing surface reconstruction on each sub-point cloud data in parallel. Specifically, the construction of the single-frame reconstruction grid of the point cloud data specifically includes:

[0077] Dividing the point cloud data into several non-overlapping sub-point cloud data;

[0078] Performing radial transformation on each sub-point cloud data to obtain the reverse point set of each sub-point cloud data;

[0079] Calculating the convex hull of the union of the reverse point set of each sub-point cloud data and the viewing point of the point cloud data to obtain the single-frame reconstruction grid of the point cloud data.

[0080] Specifically, each of a number of non-overlapping sub-point cloud data is included in the point cloud data, and the number of non-overlapping sub-point cloud data constitutes the point cloud data. For example, the point cloud data is divided into non-overlapping sub-point cloud data , where ， each covers a horizontal angle range . After dividing to obtain a number of non-overlapping sub-point cloud data, the reverse point set of each sub-point cloud data is determined by radial transformation. Among them, the acquisition process of the reverse point set corresponding to each sub-point cloud data is the same as the acquisition process of the reverse point set of the above-mentioned point cloud data, which will not be elaborated here.

[0081] After obtaining the reverse point set corresponding to each sub-point cloud data, the union convex hull of the reverse point set corresponding to each sub-point cloud data and the viewpoint can be calculated (that is, calculate the convex hull of ), and then a network connection of the point cloud data is established based on the convex hulls of the non-overlapping sub-point cloud data to obtain a single-frame reconstruction grid of the point cloud data. Among them, when calculating the convex hulls of the non-overlapping sub-point cloud data, parallel calculation can be performed through parallel threads, which can further improve the construction speed of the single-frame reconstruction grid of the point cloud data, and further improve the real-time reconstruction speed of the dynamic scene.

[0082] Furthermore, in steps S22 and S23, normal vector estimation is performed on the point cloud data to obtain the normal vector of each point in the point cloud data. Among them, when performing normal vector estimation, the normal vector of each triangle in the single-frame reconstruction grid can be obtained first, and then a confidence weight is assigned to each triangle, and then the normal vector of each point is calculated based on the normal vector and confidence weight of the triangle corresponding to each point. Among them, the triangle is determined when constructing the single-frame reconstruction network.

[0083] Since the lidar captures point cloud data by emitting laser beams to measure the reflection position and flight time, when the surface in the dynamic scene is parallel to the laser beam, the lidar cannot accurately capture the shape and position of the surface in the dynamic scene. Therefore, all reconstructed surfaces close to the planar laser beam are unreliable and need to be removed. It can be seen that when the triangle is perpendicular to the laser beam of the lidar (that is, the triangle faces the lidar), the confidence of the lidar is high. On the contrary, when the triangle is parallel to the laser beam of the lidar, the confidence of the lidar is 0. Based on this, the confidence weight of the triangle can be determined by the relationship between the points in the point cloud data included in the triangle and the viewpoint. For example, the confidence weight of the triangle can be expressed as:

[0084] ,

[0085] where Represents the confidence weight of a triangle, Represents the normal vector of a triangle, Represents the center point of a triangle, Represents the viewing point.

[0086] After obtaining the confidence weight of each triangle, obtain multiple triangles corresponding to each point in the point cloud data (such as selecting the triangle containing the point in the point cloud data, and then obtaining the adjacent triangles of the triangle to obtain the triangles corresponding to the point in the point cloud data, etc.). After obtaining the triangles corresponding to each point in the point cloud data, determine the normal vector of each point in the point cloud data by weighting the normal vectors of the triangles corresponding to each point in the point cloud data. Among them, the calculation formula for the normal vector of each point in the point cloud data can be:

[0087] ,

[0088] Among them, Represents the point in the point cloud data The normal vector of, Represents the th normal vector of a triangle, Represents the th confidence weight of a triangle, Represents the point The number of corresponding triangles.

[0089] In the embodiment of the present application, by using the normal vector and confidence weight of a triangle to estimate the normal vector of a point in the point cloud data, the noise in the normal vector estimation can be significantly reduced. And, compare the normal vector estimation performance of the normal vector estimation method in the present application with the PUMA, VDBFusion, and SHINE-Mapping methods on a set of highly uneven point clouds captured by a single frame: Top: synthetic scene, bottom: real scene. The accuracy of the estimated normal is evaluated using their cosine similarity with the ground truth normal. The comparison results are as Figure 2 shown. It can be seen from Figure 2 that the normal vector estimation method in the present application determines the normal vector more accurately.

[0090] S30. Determine a single-frame distance field based on the lidar line of sight based on the single-frame grid data, and update the global distance field based on the single-frame distance field.

[0091] Specifically, a single-frame distance field reflects the free space in a single-frame network data, where the free space refers to the area without any objects, and the laser beam can penetrate the free space and return to the lidar. To this end, when determining the single-frame distance field based on the lidar line of sight, the environment can be discretized into voxels, and the single-frame grid data can be used to initialize the voxels so that each voxel represents a local surface on a boundary grid, and the local surface can adopt three-dimensional barycentric coordinates pt ( q ), normal vector nt ( q ) and confidence weight wt ( q ) for approximate representation (i.e., the centroid, normal vector, and confidence weight of a triangle (i.e., triangular facet) are recorded in each voxel); calculate the distance value from each voxel to the object surface along the direction of the lidar scan line and record it in the voxel to obtain the single-frame distance field corresponding to the point cloud data.

[0092] Exemplarily, determining the single-frame distance field based on the lidar line of sight based on the single-frame grid data specifically includes:

[0093] Calculate the bounding box of the point cloud data and construct an octree voxel set based on the bounding box;

[0094] Obtain the rays from the viewpoint of the point cloud data to each point in the point cloud data, get all the voxels passing through the lidar line of sight in the octree voxel set, and obtain the intersection points of the lidar line of sight and the single-frame grids in the single-frame grid data, where the voxels are initialized based on the single-frame grid data, and the centroid, normal vector, and confidence weight of a triangle are recorded in the voxels;

[0095] Calculate the single-frame distance from each voxel to the single-frame grid intersection point, and determine the perception space corresponding to the point cloud data as free space, edge space, and unknown space based on the single-frame distance of the voxel to obtain the single-frame distance field based on the lidar line of sight.

[0096] Specifically, in the single-frame distance field based on the lidar line of sight (denoted as the LoS (Line of Sight) distance field), the object surface is located on the 0-value isosurface, the voxels between the object surface and the lidar have positive LoS single-frame distances, while the voxels behind the object surface have negative single-frame distances, where the free space represents the inside of the single-frame reconstruction), the edge space represents the vicinity of the surface of the single-frame reconstruction, and the unknown space represents the external space of the reconstruction. To this end, calculate the bounding box of the point cloud data and construct an octree voxel set (i.e., discretize the environment in the single-frame grid data into voxels), and then project a ray from the viewpoint of the lidar to the center of each voxel in the octree voxel set (i.e., the ray of the viewpoint of the point cloud data projected onto the voxel), and this ray serves as the lidar line of sight corresponding to the voxel.

[0097] After obtaining the lidar line of sight, calculate the grid intersection of the lidar line of sight and the single-frame reconstruction network in the single-frame grid data, and then determine the single-frame distance of the voxel based on this grid intersection. Among them, the expression of the single-frame distance can be:

[0098] ,

[0099] wherein, represents the single-frame distance field of the voxel centered on , represents the viewpoint, represents the grid intersection of the lidar line of sight and the single-frame reconstruction network in the single-frame grid data, represents the size of the voxel, represents the maximum value operation, and by it can ensure that negative single-frame distances are truncated to the minimum value.

[0100] In addition, due to visibility limitations, the single-frame point cloud data can only provide partial free space information in the scene. In order to accurately represent the complete free space of the entire scene, after obtaining the single-frame distance field, the single-frame distance field can be fused with the existing global distance field to obtain the free space of the dynamic scene. Based on this, after obtaining the single-frame distance field, the single-frame distance field will be fused into the existing global distance field, and this global distance field is initialized by the first-frame single-frame distance field. That is to say, the global distance field is the single-frame distance field that fuses all the point cloud data collected before this point cloud data, so that the global distance field corresponding to the point cloud data fuses the single-frame distance fields of all the point cloud data from the first-frame point cloud data to this point cloud data, thereby being able to more accurately represent the complete free space of the dynamic scene.

[0101] Exemplarily, the fusing of the single-frame distance field to obtain the global distance field specifically includes: The updating of the global distance field based on the single-frame distance field specifically includes:

[0102] Search for the voxels in the single-frame distance field in the global distance field;

[0103] For the found voxels, update the global distance of the voxels in the global distance field based on the single-frame distance of the voxels in the single-frame distance field;

[0104] For the unfound voxels, add the voxels to the global distance field and use the single-frame distance of the voxels in the single-frame distance field as their global distance in the global distance field.

[0105] Specifically, updating the global distance of a voxel in the global distance field based on the single-frame distance of the voxel in the single-frame distance field means weighting the single-frame distance of the voxel in the single-frame distance field and the global distance of the voxel in the existing global distance field (i.e., the global distance field before update) to obtain the updated global distance corresponding to the voxel. Among them, both the global weight value of the global distance and the single-frame weight value of the single-frame distance can be preset, and the global weight value and the single-frame weight value can be the same. For example, both the global weight value and the single-frame weight value are 1, etc. Of course, in practical applications, the global weight value and the single-frame weight value can also be set according to implementation requirements, and no specific restrictions are made here.

[0106] Furthermore, after obtaining the global weight value and the single-frame weight value, the updated global distance field can be obtained by weighting the single-frame distance and the existing global distance based on the global weight value and the single-frame weight value. The global distance of the voxel in the updated global distance field can be expressed as:

[0107] ,

[0108] where, represents the updated global distance, represents the single-frame distance, represents the global distance before update, represents the global weight value, represents the single-frame weight value.

[0109] By integrating the single-frame distance field into the global distance field to update the global distance field, the present application can continuously explore the scene with the unmanned device equipped with a lidar and continuously update the global distance field, making the global distance field gradually refined, so as to more accurately reconstruct the dynamic scene, and can more accurately represent the complete free space of the dynamic field nodes, providing more accurate scene information for navigation based on the real-time reconstruction model of the dynamic scene.

[0110] S40. Determine the spatial markers of the dynamic scene based on the updated global distance field, and remove the dynamically occupied spaces in the free space of the spatial markers based on the global distance field before update and the single-frame distance field to obtain the static scene free space corresponding to the dynamic scene;

[0111] Specifically, the global distance field helps to detect dynamic objects in a dynamic scene. In a scene with only static objects, the value of the LoS distance field monotonically increases as the lidar explores the scene from multiple perspectives, reflecting the expanding free space. However, when a dynamic object enters the previously marked free space, the value of the LoS distance field will decrease. Therefore, dynamic objects can be identified by comparing the single-frame distance field with the global distance field before the update, and they will be excluded when reconstructing the static scene in real time. Based on this, removing the dynamically occupied space in the free space of the spatial marker based on the global distance field before the update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene specifically includes:

[0112] Compare the voxels in the single-frame distance field with the voxels in the global distance field before the update to select the dynamically occupied space;

[0113] Separate the selected dynamically occupied space from the free space in the spatial marker to obtain the free space of the static scene corresponding to the dynamic scene.

[0114] Specifically, after obtaining the updated global distance field, the voxels in the updated global distance field with a global distance less than a preset distance threshold can be regarded as occupied voxels. Then, compare the single-frame distance of each occupied voxel in the single-frame distance field with the global distance in the global distance field before the update. If the global distance in the global distance field before the update is greater than the preset distance threshold and the single-frame distance is less than or equal to the preset distance threshold, then regard this occupied voxel as a dynamically occupied voxel to obtain the dynamically occupied space. Based on this, the dynamically occupied voxels need to meet the following conditions:

[0115] ,

[0116] wherein, represents the updated global distance, represents the single-frame distance, represents the global distance before the update.

[0117] By excluding the dynamically occupied voxels during the real-time reconstruction of the static scene in this application, significant errors caused by the points of dynamic objects during the real-time reconstruction can be avoided. For example, Figure 3 shows the reconstructed model after removing the dynamically occupied voxels and the reconstructed model without removing the occupied voxels. As can be seen from Figure 3 , the reconstructed model after removing the dynamically occupied voxels can represent the static scene more accurately.

[0118] S50. Construct a scene model corresponding to the dynamic scene without dynamic objects based on the free space of the static scene.

[0119] Specifically, after obtaining the free space of the static scene, all the points on all the static surfaces in the free space of the static scene are fused to reconstruct a high-quality dynamic scene. For example, a voxel-based signed distance function (SDF) framework is used for real-time online modeling to reconstruct a scene model without dynamic objects in real time. By excluding dynamic objects during the real-time reconstruction of the static scene in the embodiments of the present application, significant errors caused by the points of dynamic objects during the real-time reconstruction process can be avoided. For example, Figure 4 shows a scene of real-time autonomous exploration and scene reconstruction by a robot. During the autonomous navigation and exploration of the robot, the surface model of the dynamic scene gradually takes shape, and in Figure 4 Scenes 2 and 3 shown, the method provided by the embodiments of the present application effectively maintains the smoothness and accuracy of the model even when there are obstacles in the scene, highlighting its ability to handle different scenes and cope with the challenges of robot applications.

[0120] In summary, this embodiment provides a method for real-time reconstruction of a dynamic scene. The point cloud data of the dynamic scene is collected by a lidar; the point cloud data is subjected to mesh reconstruction to obtain single-frame mesh data corresponding to the point cloud data; a single-frame distance field based on the lidar line of sight is determined based on the single-frame mesh data, and the global distance field is updated based on the single-frame distance field; a spatial marker of the dynamic scene is determined based on the updated global distance field, and the dynamic occupied space in the free space of the spatial marker is removed based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene; a scene model of the dynamic scene without dynamic objects is constructed based on the free space of the static scene. The present application first performs single-frame reconstruction on the point cloud data of the dynamic scene collected by the lidar, and then fuses the single-frame distance field corresponding to the single-frame reconstruction data and the global distance field obtained by fusing all the forward point cloud data for multi-frame reconstruction. In this way, the point cloud data captured by the lidar is fully utilized, and the discrete points scanned by the lidar are immediately converted into a continuous region for surface reconstruction, ensuring the timeliness of the dynamic scene reconstruction and improving the accuracy of the dynamic scene reconstruction.

[0121] Based on the above method for real-time reconstruction of a dynamic scene, this embodiment provides a navigation method. Using the method for real-time reconstruction of a dynamic scene as described above, the navigation method specifically includes:

[0122] Collect point cloud data through a lidar carried by an unmanned device, and reconstruct a scene model through the method for real-time reconstruction of a dynamic scene;

[0123] Extract the free space in the scene model, and determine path points of the unmanned device based on the free space to form a navigation path of the unmanned device.

[0124] Specifically, the unmanned device can be a drone, a robot, etc. The unmanned device is equipped with a lidar, which collects point cloud data of the scene through the lidar, reconstructs the scene based on the collected point cloud data, and plans a navigation path based on the free space in the reconstructed dynamic scene, so as to ensure that the planned path avoids both dynamic obstacles and static obstacles. For example, Figure 5 shows the process of a robot extracting free space in an application scenario in real time and moving from an initial position to an unknown target position. From Figure 5 it can be seen that the robot gradually selects path points by using the free space, plans an open path to reach the target, and finally successfully reaches the end point without any knowledge of the scene.

[0125] Based on the above dynamic scene real-time reconstruction method, this embodiment provides a dynamic scene real-time reconstruction device, as Figure 6 shown, the dynamic scene real-time reconstruction device specifically includes:

[0126] An acquisition module 100, configured to collect point cloud data of a dynamic scene through a lidar;

[0127] A single-frame reconstruction module 200, configured to perform grid reconstruction on the point cloud data to obtain single-frame grid data corresponding to the point cloud data;

[0128] A multi-frame reconstruction module 300, configured to determine a single-frame distance field based on the lidar line of sight based on the single-frame grid data, update the global distance field based on the single-frame distance field, determine the spatial marker of the dynamic scene based on the updated global distance field, remove the dynamic occupied space in the free space of the spatial marker based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene, and construct a scene model corresponding to the dynamic scene without dynamic objects based on the free space of the static scene, where the global distance field is initialized based on the single-frame distance field corresponding to the first-frame point cloud data.

[0129] Based on the above dynamic scene real-time reconstruction method, this embodiment provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the dynamic scene real-time reconstruction method as described in the above embodiment.

[0130] Based on the above dynamic scene real-time reconstruction method, this application further provides a terminal device, as Figure 7As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22, and the communications interface 23 can complete mutual communication through the bus 24. The display screen 21 is set to display a user guidance interface preset in the initial setting mode. The communications interface 23 can transmit information. The processor 20 can call logic instructions in the memory 22 to execute the methods in the above embodiments.

[0131] In addition, when the logic instructions in the above-mentioned memory 22 can be implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0132] The memory 22, as a computer-readable storage medium, can be set to store software programs and computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, that is, to implement the methods in the above embodiments.

[0133] The memory 22 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, various media that can store program codes such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc can also be a transient storage medium.

[0134] In addition, the specific processes of loading and executing multiple instructions by the processor in the above storage medium and the terminal device have been described in detail in the above methods and will not be repeated here.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for real-time reconstruction of dynamic scenes, characterized in that, The specific steps of the dynamic scene real-time reconstruction method are as follows: Collect point cloud data of the dynamic scene through a lidar; Perform mesh reconstruction on the point cloud data to obtain single-frame mesh data corresponding to the point cloud data; Determine a single-frame distance field based on the lidar line of sight based on the single-frame mesh data, and update the global distance field based on the single-frame distance field, where the global distance field is initialized based on the single-frame distance field corresponding to the first-frame point cloud data; Determine the spatial markers of the dynamic scene based on the updated global distance field, and remove the dynamic occupied spaces in the free spaces in the spatial markers based on the global distance field before update and the single-frame distance field, so as to obtain the free space of the static scene corresponding to the dynamic scene; Construct a scene model corresponding to the dynamic scene that does not contain dynamic objects based on the free space of the static scene; The specific steps of determining a single-frame distance field based on the lidar line of sight based on the single-frame mesh data are as follows: Calculate the bounding box of the point cloud data, and construct an octree voxel set based on the bounding box; Obtain the rays from the viewpoint of the point cloud data to each point in the point cloud data, obtain all the voxels passing through the lidar line of sight in the octree voxel set, and obtain the intersections of the lidar line of sight with the single-frame meshes in the single-frame mesh data, where the voxels are initialized based on the single-frame mesh data, are used to represent the local surface on a boundary mesh, and the barycenter, normal vector, and confidence weight of the triangle are recorded in the voxels; Calculate the single-frame distance from each voxel to the single-frame mesh intersection, and determine that the perceptual spatial markers corresponding to the point cloud data are free space, edge space, and unknown space based on the single-frame distance of the voxels, so as to obtain a single-frame distance field based on the lidar line of sight; Wherein, the expression of the single-frame distance is: ； Among them, represents the single-frame distance field of the voxel centered on The voxel centered on represents the viewing point, represents the grid intersection of the lidar line of sight and the single-frame reconstruction network in the single-frame grid data, represents the size of the voxel, represents the maximum value operation.

2. The real-time reconstruction method for dynamic scenes according to claim 1, wherein The specific steps of performing mesh reconstruction on the point cloud data to obtain single-frame mesh data corresponding to the point cloud data are as follows: Construct a single-frame reconstruction mesh of the point cloud data; Obtain the normal vector of each triangle in the single-frame reconstruction mesh, and determine the confidence weight of each triangular mesh based on the normal vector of each triangle; Obtain multiple triangles corresponding to each point in the point cloud data, and determine the normal vector of each point in the point cloud data based on the normal vectors and confidence weights of the obtained triangles.

3. The real-time reconstruction method for dynamic scenes according to claim 2, wherein The specific steps of constructing a single-frame reconstruction mesh of the point cloud data are as follows: Divide the point cloud data into several non-overlapping sub-point cloud data; Perform radial transformation on each sub-point cloud data to obtain the reverse point set of each sub-point cloud data; Calculate the convex hull of the union of the reverse point set of each sub-point cloud data and the viewpoint of the point cloud data to obtain the single-frame reconstruction mesh of the point cloud data.

4. The real-time reconstruction method for dynamic scenes according to claim 1, wherein The specific steps of updating the global distance field based on the single-frame distance field are as follows: Search for the voxels in the single-frame distance field in the global distance field; For the found voxels, update the global distance of the voxels in the global distance field based on the single-frame distance of the voxels in the single-frame distance field; For the unfound voxels, add the voxels to the global distance field, and use the single-frame distance of the voxels in the single-frame distance field as their global distance in the global distance field.

5. The real-time reconstruction method for dynamic scenes according to claim 1, characterized in that Removing the dynamically occupied space in the free space of the spatial marker based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene specifically includes: Comparing the voxels in the single-frame distance field with the voxels in the global distance field before update to select the dynamically occupied space; Separating the selected dynamically occupied space from the free space in the spatial marker to obtain the free space of the static scene corresponding to the dynamic scene.

6. A navigation method, characterized in that, Using the dynamic scene real-time reconstruction method according to any one of claims 1-5, the navigation method specifically includes: Collecting point cloud data through a lidar carried by an unmanned device and reconstructing a scene model through the dynamic scene real-time reconstruction method; Extracting the free space in the scene model and determining path points of the unmanned device based on the free space to form a navigation path of the unmanned device.

7. A real-time reconstruction device for dynamic scenes, characterized in that, The dynamic scene real-time reconstruction device specifically includes: A collection module for collecting point cloud data of a dynamic scene through a lidar; A single-frame reconstruction module for performing grid reconstruction on the point cloud data to obtain single-frame grid data corresponding to the point cloud data; A multi-frame reconstruction module for determining a single-frame distance field based on the lidar line of sight based on the single-frame grid data, updating the global distance field based on the single-frame distance field, determining a spatial marker of the dynamic scene based on the updated global distance field, removing the dynamically occupied space in the free space of the spatial marker based on the global distance field before update and the single-frame distance field to obtain the free space of the static scene corresponding to the dynamic scene, and constructing a scene model of the dynamic scene without dynamic objects based on the free space of the static scene, wherein the global distance field is initialized based on the single-frame distance field corresponding to the first-frame point cloud data; The determining the single-frame distance field based on the lidar line of sight based on the single-frame grid data specifically includes: Calculating a bounding box of the point cloud data and constructing an octree voxel set based on the bounding box; Obtaining rays from the viewpoint of the point cloud data to each point in the point cloud data, obtaining all voxels passing through the lidar line of sight in the octree voxel set, and obtaining intersections of the lidar line of sight with single-frame grids in the single-frame grid data, wherein the voxels are initialized based on the single-frame grid data, are used to represent a local surface on a boundary grid, and record the centroid, normal vector, and confidence weight of a triangle; Calculating the single-frame distance from each voxel to the single-frame grid intersection and determining that the perceptual space marker corresponding to the point cloud data is free space, edge space, and unknown space based on the single-frame distance of the voxel to obtain a single-frame distance field based on the lidar line of sight; Wherein, the expression of the single-frame distance is: ； Among them, represents the single-frame distance field of the voxel centered on The single-frame distance field of the voxel centered on represents the viewing point, represents the grid intersection point of the lidar line of sight and the single-frame reconstruction network in the single-frame grid data, represents the size of the voxel, represents the maximum value operation.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the dynamic scene real-time reconstruction method according to any one of claims 1-5.

9. A terminal device, characterized in that, Including: A processor and a memory; A computer-readable program executable by the processor is stored on the memory; When the processor executes the computer-readable program, the steps in the dynamic scene real-time reconstruction method according to any one of claims 1-5 are implemented.

Images