Three-dimensional surface modeling method based on sparse point cloud frames

Abstract

The application discloses a three-dimensional surface modeling method based on a sparse point cloud frame. A laser radar system is used to scan an object in a real world to obtain a point cloud frame sequence. A variational encoder is used to encode features of each frame of point cloud in the point cloud frame sequence to obtain a global feature distribution and a point-by-point local feature distribution corresponding to each frame of point cloud. The point-by-point local feature distribution is used as a query matrix, and the global feature distribution is used as a key matrix, so that a density-aware mutual attention is used for point-by-point feature refinement to obtain a refined point-by-point local feature corresponding to each frame of point cloud. A random sample consensus algorithm is used to estimate a rigid change between adjacent frames of point cloud according to the refined point-by-point local feature, and the two-by-two registration is performed through the rigid change between the adjacent frames of point cloud to obtain a point cloud registration result. Geometric-aware point cloud direct regression is used according to the point cloud registration result to obtain a final three-dimensional model, so that the modeling effect is improved.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a 3D surface modeling method based on sparse point cloud frames. Background Technology

[0002] In related technologies, 3D surface modeling refers to the process of acquiring, preprocessing, registering, fusing, and denoising depth data to depict a real scene into a mathematical model that conforms to computer logic. This model can play an auxiliary role in research such as cultural relic protection, game development, and architectural design. In the past few decades, the most widely used 3D modeling sensor has been LiDAR. However, most LiDARs have centimeter-level distance errors and visible point spacing due to near-light conditions. How to utilize such sparse point clouds for 3D surface modeling has become a key issue in various fields such as autonomous driving, robot navigation, and industrial manufacturing. Existing point cloud modeling datasets, such as the KITTI dataset and the OXFORD dataset, mainly evaluate the modeling effect through the accuracy of pose, but do not provide a unified and complete evaluation method for the modeling effect itself. However, since the point cloud of sparse point cloud frames often has uneven surface points, pose-based methods will inevitably produce errors when estimating modeling accuracy.

[0003] Most existing methods refer to the environmental modeling problem as the Simultaneous Localization and Mapping (SLT) problem, where modeling is usually a post-processing operation, such as moving least squares or bundle adjustment. However, these methods rely on prior matching results and features from inter-frame registration for modeling correction, which makes them unsuitable for sparse point clouds. In addition, these traditional geometry-based methods can produce significant ambiguity when extracting features from complex objects. Compared to traditional methods, feature extraction based on deep learning has shown promise in the point cloud field. However, due to the complexity of the entire modeling process and the end-to-end requirements of deep learning, there are few frameworks that fully implement the 3D modeling process using deep learning. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one objective of this application is to propose a 3D surface modeling method based on sparse point cloud frames, which uses deep learning methods to achieve 3D modeling of complex objects in a sparse point cloud environment, thereby improving the modeling effect.

[0005] To achieve the above objectives, the first aspect of this application proposes a 3D surface modeling method based on sparse point cloud frames, comprising the following steps: constructing a lidar system and scanning real-world objects using the lidar system to obtain a corresponding point cloud dataset, wherein the point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at its acquisition time node; using a variational encoder to perform feature encoding on each point cloud frame in the point cloud frame sequence to obtain a global feature distribution and a point-by-point local feature distribution corresponding to each point cloud frame; using the point-by-point local feature distribution as a query matrix and the global feature distribution as a key matrix to refine the point-by-point features using density-aware mutual attention to obtain refined point-by-point local features corresponding to each point cloud frame; using a random sampling consensus algorithm to estimate the rigidity changes between adjacent point cloud frames based on the refined point-by-point local features, and performing pairwise registration based on the rigidity changes between adjacent point cloud frames to obtain point cloud registration results; and using geometrically aware point cloud direct regression based on the point cloud registration results to obtain the coordinates of downsampled points to obtain the final 3D model.

[0006] According to the 3D surface modeling method based on sparse point cloud frames in this application, firstly, a lidar system is built, and real-world objects are scanned using the lidar system to obtain the corresponding point cloud dataset. The point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at its acquisition time point. Then, a variational encoder is used to encode the features of each point cloud frame in the sequence to obtain the global feature distribution and point-by-point local feature distribution corresponding to each frame. Next, the point-by-point local feature distribution is used as a query matrix, and the global feature distribution is used as a key matrix. The point cloud is processed by a density-aware mutual attention algorithm to refine point-by-point features, resulting in refined point-by-point local features for each frame. Next, a random sampling consensus algorithm is used to estimate the rigidity changes between adjacent frame point clouds based on the refined point-by-point local features. Pairwise registration is then performed based on these rigidity changes to obtain the point cloud registration results. Finally, geometrically aware point cloud direct regression is used to obtain the coordinates of downsampled points based on the point cloud registration results, resulting in the final 3D model. Thus, deep learning methods are used to achieve 3D modeling of complex objects in sparse point cloud environments, thereby improving the modeling effect.

[0007] In addition, the 3D surface modeling method based on sparse point cloud frames proposed in the above embodiments of this application may also have the following additional technical features:

[0008] Optionally, the point cloud dataset may also include point cloud frames obtained by scanning with a high-precision LiDAR scanner, in order to evaluate the modeling effect based on the chamfer loss.

[0009] Optionally, the modeling effect can be evaluated according to the following formula:

[0010]

[0011] Where P is the temporal modeling point cloud of the low-precision lidar, p i Let O be the i-th point in the point cloud P, and let O be the point cloud frame captured by the high-precision LiDAR scanner. j Let N be the j-th point in point cloud O, N be the total number of points in point cloud P, and M be the total number of points in point cloud O. The chamfer distance is calculated to determine the nearest neighbor distance between each point and another point cloud, and the total distances between the two point clouds are summed to measure the difference between the point clouds.

[0012] Optionally, obtaining the downsampled point coordinates using geometric perception point cloud direct regression based on the point cloud registration result includes: performing feature encoding using a variational encoder based on the point cloud registration result to obtain the global feature distribution and point-by-point local feature distribution corresponding to the point cloud registration result; using the point-by-point local feature distribution as the key matrix and the global feature distribution as the query matrix, so that the corresponding global features can be obtained after calculation using an attention mechanism; and building a multilayer perceptron to regress the point coordinates to obtain the sampling result.

[0013] Optionally, during point cloud registration and point cloud sampling, a weighted graph is used for iterative optimization. The weighted graph includes a weighted distribution graph and a weighted importance graph. The weighted distribution graph includes neighborhood density features and allocates task-independent information for optimization. The weighted importance graph provides task-oriented information and the geometric importance of local shapes in the global shape.

[0014] To achieve the above objectives, a second aspect of the present invention proposes a three-dimensional surface modeling device based on sparse point cloud frames, comprising: an acquisition module, wherein the acquisition module scans real-world objects using a lidar system to obtain a corresponding point cloud dataset, wherein the point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at its acquisition time node; and a density-aware point cloud registration module, wherein the density-aware point cloud registration module uses a variational encoder to perform feature encoding on each point cloud frame in the point cloud frame sequence to obtain the global feature distribution and point-by-point local feature distribution corresponding to each point cloud frame. The process involves: using the point-by-point local feature distribution as the query matrix and the global feature distribution as the key matrix to refine the point-by-point features using density-aware mutual attention, resulting in refined point-by-point local features for each frame of point cloud; employing a random sampling consensus algorithm to estimate the rigidity changes between adjacent frame point clouds based on the refined point-by-point local features, and performing pairwise registration based on these rigidity changes to obtain point cloud registration results; and using a geometrically aware point cloud sampling module to obtain downsampled point coordinates based on the point cloud registration results using geometrically aware point cloud direct regression to obtain the final 3D model.

[0015] The 3D surface modeling device based on sparse point cloud frames according to embodiments of the present invention achieves 3D modeling of complex objects in a sparse point cloud environment through deep learning methods, thereby improving the modeling effect.

[0016] In addition, the three-dimensional surface modeling device based on sparse point cloud frames proposed in the above embodiments of the present invention may also have the following additional technical features:

[0017] Optionally, the point cloud dataset may also include point cloud frames obtained by scanning with a high-precision LiDAR scanner, in order to evaluate the modeling effect based on the chamfer loss.

[0018] Optionally, the modeling effect can be evaluated according to the following formula:

[0019]

[0020] Where P is the temporal modeling point cloud of the low-precision lidar, p i Let O be the i-th point in the point cloud P, and let O be the point cloud frame captured by the high-precision LiDAR scanner. j Let N be the j-th point in point cloud O, N be the total number of points in point cloud P, and M be the total number of points in point cloud O. The chamfer distance is calculated to determine the nearest neighbor distance between each point and another point cloud, and the total distances between the two point clouds are summed to measure the difference between the point clouds.

[0021] Optionally, obtaining the downsampled point coordinates using geometric perception point cloud direct regression based on the point cloud registration result includes: performing feature encoding using a variational encoder based on the point cloud registration result to obtain the global feature distribution and point-by-point local feature distribution corresponding to the point cloud registration result; using the point-by-point local feature distribution as the key matrix and the global feature distribution as the query matrix, so that the corresponding global features can be obtained after calculation using an attention mechanism; and building a multilayer perceptron to regress the point coordinates to obtain the sampling result.

[0022] Optionally, during point cloud registration and point cloud sampling, a weighted graph is used for iterative optimization. The weighted graph includes a weighted distribution graph and a weighted importance graph. The weighted distribution graph includes neighborhood density features and allocates task-independent information for optimization. The weighted importance graph provides task-oriented information and the geometric importance of local shapes in the global shape. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a three-dimensional surface modeling method based on sparse point cloud frames according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of a lidar system according to an embodiment of this application;

[0025] Figure 3 This is a 3D modeling effect image based on a real-world dataset collected according to an embodiment of this application;

[0026] Figure 4 A diagram of the depth network structure for density-aware point cloud registration in a 3D modeling framework according to an embodiment of this application;

[0027] Figure 5 This is a diagram of the depth network structure for geometry-aware point cloud sampling in a 3D modeling framework according to an embodiment of this application.

[0028] Figure 6 This is an overview diagram of a 3D modeling workflow framework according to an embodiment of this application;

[0029] Figure 7 This is a visualization of point cloud registration according to an embodiment of this application;

[0030] Figure 8 This is a visualization of point cloud sampling according to an embodiment of this application;

[0031] Figure 9 3D modeling renderings based on the existing KITTI dataset;

[0032] Figure 10This is a block diagram of a three-dimensional surface modeling apparatus based on sparse point cloud frames according to an embodiment of this application. Detailed Implementation

[0033] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0034] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0035] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0036] Figure 1 This is a flowchart illustrating a 3D surface modeling method based on sparse point cloud frames according to an embodiment of this application, as shown below. Figure 1 As shown, the 3D surface modeling method based on sparse point cloud frames includes the following steps:

[0037] S101, Build a LiDAR system and scan real-world objects using the LiDAR system to obtain the corresponding point cloud dataset. The point cloud dataset includes a sequence of point cloud frames acquired by a low-precision LiDAR and the pose of each point cloud frame at the acquisition time node.

[0038] As an example, the point cloud dataset also includes point cloud frames scanned by a high-precision LiDAR scanner to evaluate the modeling effectiveness based on the chamfer loss.

[0039] As a specific example, such as Figure 2 As shown, the lidar system consists of an optical platform, a millimeter-precision translation rail, and a milliradian-precision rotary stage. The system is equipped with three lidars of different line counts: 1. A 16-line low-precision lidar, Velodyne-16, with a scanning accuracy of 2cm; 2. A 32-line low-precision lidar, Hesai-32, with a scanning accuracy of 0.5cm; 3. A lidar scanner, Riegl VZ-1000, with a scanning accuracy of 5mm.

[0040] It should be noted that during the entire indoor scene acquisition process, the scene was kept completely static, and multiple objects with complex structures (i.e., the acquisition objects) were placed on the same horizontal plane as the system and spaced 2m apart. The LiDAR was moved using millimeter-precision translation guides and a rotating platform, and the pose of the current position was recorded after each movement to simulate rotation and translation during multi-view modeling in real-world situations. After each movement, the high-precision rotation and translation values ​​at the current time point were recorded. At the same time, a millimeter-precision LiDAR scanner was used to capture the data in a unified world coordinate system, which was then used as the global high-precision modeling point cloud (i.e., the true value of the point cloud sampling).

[0041] In summary, the dataset provides a total of 3,600 time points, each of which records the true pose values ​​obtained from high-precision translation guides and rotary tables, as well as point clouds obtained from three types of radar scans. In other words, the point cloud dataset is a high-precision, multi-beam real-world dataset that provides pose and modeling accuracy that exceeds most existing datasets, and it focuses on modeling scenarios of complex objects.

[0042] In other words, the point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at the acquisition time node (the data acquired by two low-precision lidars can be used for two sets of 3D modeling comparisons or one can be selected for subsequent 3D modeling) and point cloud frames obtained by a high-precision lidar scanner.

[0043] As an example, the modeling effect is evaluated according to the following formula:

[0044]

[0045] Where P is the temporal modeling point cloud of the low-precision lidar, p i Let O be the i-th point in the point cloud P, and let O be the point cloud frame captured by the high-precision LiDAR scanner. j Let N be the j-th point in point cloud O, N be the total number of points in point cloud P, and M be the total number of points in point cloud O. The chamfer distance is calculated to determine the nearest neighbor distance between each point and another point cloud, and the total distances between the two point clouds are summed to measure the difference between the point clouds.

[0046] In other words, after obtaining the input sequence of the low-precision LiDAR and the precise pose at each time point in the sequence (this data is collected by one of the two low-precision LiDARs), the temporal modeling result for each time point can be obtained through a 3D modeling algorithm. With a high-precision LiDAR scanner also available, the two point clouds can be evaluated using the classic evaluation metric, Chamfer Loss; where a smaller chamfer distance indicates greater similarity between the two point clouds. The visualization results on the collected data are as follows: Figure 3 As shown.

[0047] It should be noted that, based on the collected dataset, the modeling effect can be evaluated at the object level, filling the gap in this aspect of the existing dataset.

[0048] S102, a variational encoder is used to encode the features of each frame of point cloud in the point cloud frame sequence to obtain the global feature distribution and point-by-point local feature distribution corresponding to each frame of point cloud.

[0049] It should be noted that since the points scanned by low-precision LiDAR all have precision errors, which are manifested as random errors in the point coordinates, a variational encoder can be used to eliminate these errors.

[0050] Specifically, such as Figure 4 As shown, deep learning methods are used to encode the deep features of each point in the point cloud. This mainly involves local variational encoders and global variational encoders, with the overall formula as follows:

[0051]

[0052] f global =φ(p0,p1,...,p N )

[0053] in, φ and V are two implicit functions based on variational encoders, and V is the point p. j The neighborhood set is k, where k is the number of neighborhood points. The local variational encoder embeds each point from its three-dimensional coordinates into the high-dimensional feature space, while the global variational encoder embeds all point coordinates into the global feature.

[0054] To eliminate errors, the global feature distribution and the local neighborhood feature distribution are constrained to approximate a normal distribution. Furthermore, sampled features from the feature distribution are added to the original features. The optimization method is as follows:

[0055]

[0056] Where KL is the classic KL divergence used to calculate the similarity between distributions, q is the feature distribution of the ground truth point cloud, p is the feature distribution of the input point cloud, and p(z) g ) = N(0,I) is a standard normal distribution. While restricting the feature distribution of the input point cloud to be as close as possible to the feature distribution of the true point cloud, it also tries to approach the normal distribution as much as possible, so that the sum of the sampled features and the original features can eliminate the influence of random errors to the greatest extent.

[0057] S103, the point-by-point local feature distribution is used as the query matrix and the global feature distribution is used as the key matrix, so that density-aware mutual attention is used to refine the point-by-point features to obtain the refined point-by-point local features corresponding to each frame of point cloud.

[0058] It should be noted that after obtaining the global and point-by-point local features, the density-aware point cloud registration module is used to refine the point-by-point features. For detailed structure, see [link to module]. Figure 4 Specifically, the attention mechanism uses the local feature distribution as the query matrix and the global feature distribution as the key matrix. The calculation formula for the attention mechanism is as follows:

[0059] Attention = Softmax(Q*K)·V

[0060] Q, K, V = γ Q (z local ),γ K (z global ),γ V (z local )

[0061] Here, γ represents convolution calculation.

[0062] S104. The random sampling consensus algorithm is used to estimate the rigidity changes between adjacent frame point clouds based on the refined point-by-point local features, and pairwise registration is performed based on the rigidity changes between adjacent frame point clouds to obtain the point cloud registration result.

[0063] In other words, such as Figure 4 As shown, the source point cloud and the target point cloud are adjacent frame point clouds. After pairwise registration of adjacent frame point clouds, the input point cloud is finally registered to a unified world coordinate system.

[0064] It should be noted that after obtaining point-by-point features through the attention mechanism, the Random Sample Consensus (RANSAC) algorithm is used to estimate the rigidity changes between point clouds. The calculation steps are as follows:

[0065] 1. Randomly select a sample set: Randomly select a sample set from the point cloud data.

[0066] 2. Model Parameter Estimation: Calculate the model parameters using the selected sample set. Commonly used models in point cloud registration include rotation matrices and translation vectors from rigid transformations.

[0067] 3. Interior point selection: For the remaining data points, determine which data points can be considered "interior points" by calculating their distances to the estimated model. Points with a distance less than a given threshold are considered interior points.

[0068] 4. Determine model fitness: Calculate the number of inliers to measure the fitness of the estimated model for the data. If the number of inliers is less than a predetermined threshold, the model is not suitable.

[0069] 5. Iterate through steps 1 to 4: Repeat the above steps until the predetermined number of iterations is reached or a model that meets the conditions is found.

[0070] 6. Optimal model selection: Find the model with the most interior points in all iterations, and re-estimate the parameters of the optimal model using all interior points.

[0071] After obtaining the rigid transformation matrix, the registration module is optimized using the following loss function:

[0072]

[0073] Where R and t are the rotation matrix and translation vector in the rigid transformation matrix for registering the source point cloud to the target point cloud, respectively.

[0074] S105. Based on the point cloud registration results, geometric perception point cloud direct regression is used to obtain the coordinates of downsampled points to obtain the final 3D model.

[0075] It should be noted that after obtaining the global features and point-by-point local features, the coordinates of the downsampled points are directly obtained by regression using the geometry-aware point cloud sampling module. For detailed structure, see [link to documentation]. Figure 5 Unlike the point cloud registration module, point cloud sampling focuses more on global features. Therefore, the input values ​​of the query matrix and the key matrix are interchanged, as defined below:

[0076] Q, K, V = γ Q (z global ),γ K (z local ),γ V (z global )

[0077] After obtaining the global features calculated by the attention mechanism, a simple multilayer perceptron is built to regress the point coordinates, and then the sampling module is optimized using the following loss function:

[0078]

[0079] Where X is the downsampled point cloud predicted by the module, and Y is the real downsampled point cloud corresponding to the input point cloud.

[0080] As an example, the point coordinates of the downsampled number are obtained by direct regression of the geometric perception point cloud based on the point cloud registration results. This includes: using a variational encoder to encode features based on the point cloud registration results to obtain the global feature distribution and point-by-point local feature distribution corresponding to the point cloud registration results; using the point-by-point local feature distribution as the key matrix and the global feature distribution as the query matrix, so that the corresponding global features can be obtained after calculation using an attention mechanism; and building a multilayer perceptron to regress the point coordinates to obtain the sampling results.

[0081] As an example, during the point cloud registration and sampling process, a weighted graph is used for iterative optimization. The weighted graph includes a weighted distribution graph and a weighted importance graph. The weighted distribution graph includes neighborhood density features and allocates task-independent information for optimization. The weighted importance graph provides task-oriented information and the geometric importance of local shapes in the global shape.

[0082] It should be noted that, in order to achieve end-to-end training of complete 3D modeling based on deep learning, two modules (density-aware point cloud registration module and geometry-aware point cloud sampling module) are iteratively processed and optimized to form an end-to-end deep learning framework. The specific framework can be found in [link to framework details]. Figure 6 Initially, the input point cloud is registered to a unified world coordinate system using a point cloud registration module, and then sampled using a point cloud sampling module to obtain the modeling result. In the next iteration, the sampled point cloud from the previous iteration is used as the target point cloud for registration. However, iterative modeling can lead to the accumulation of errors. To effectively suppress this, two point-by-point weighted graphs obtained through the modules are used to attenuate the accumulated error. The specific definition of the weighted graph is as follows:

[0083]

[0084]

[0085] Where R is calculated from the point cloud registration module, M is the point-by-point similarity matrix between the source and target point clouds, S is calculated from the point cloud sampling module, P is the downsampled point cloud, O is the original input point cloud, and R and S represent the point-by-point importance weighted map; D is obtained from the point cloud feature encoding module shared by the two modules, μ and σ are the mean and variance of the global feature distribution, respectively, and dl(·) represents the point p i The sampling set on the local feature distribution, M is the number of points in the sampling set, and D represents the point-by-point weighted distribution graph.

[0086] In addition, the point-by-point distribution map D, which includes neighborhood density features, allocates task-independent information for optimization, while R and S provide task-oriented information and the geometric importance of local shapes in the global shape. Through iterative iteration, the point-by-point weighted map obtained from point cloud registration assists point cloud sampling in performing density-based preliminary sampling. The weighted map obtained from point cloud sampling then assists point cloud registration in filtering geometric outliers. It is worth noting that since there are no corresponding results to be calculated in the first iteration, the weight map is initialized as a matrix to represent that all points participate in the optimization calculation.

[0087] In summary, the visualization effect of point cloud registration is as follows: Figure 7 As shown in the figure, the visualization effect of point cloud sampling is as follows: Figure 8 As shown, the 3D modeling renderings on the existing KITTI dataset are as follows: Figure 9 As shown.

[0088] To achieve the above embodiments, such as Figure 10 As shown in the embodiment of this application, a three-dimensional surface modeling device based on sparse point cloud frames is also proposed, including a data acquisition module 10, a density-aware point cloud registration module 20, and a geometry-aware point cloud sampling module 30.

[0089] The acquisition module scans real-world objects using a LiDAR system to obtain a corresponding point cloud dataset. This dataset includes a sequence of point cloud frames acquired by a low-precision LiDAR and the pose of each frame at its acquisition time point. The density-aware point cloud registration module uses a variational encoder to encode the features of each point cloud frame in the sequence, obtaining the global feature distribution and point-by-point local feature distribution for each frame. The point-by-point local feature distribution is used as a query matrix, and the global feature distribution as a key matrix, to refine the point-by-point features using density-aware mutual attention, resulting in refined point-by-point local features for each frame. A random sampling consensus algorithm is used to estimate the rigidity changes between adjacent point cloud frames based on the refined point-by-point local features, and pairwise registration is performed based on these rigidity changes to obtain the point cloud registration result. The geometry-aware point cloud sampling module uses direct regression based on the point cloud registration result to obtain the coordinates of downsampled points, resulting in the final 3D model.

[0090] It should be noted that the above description of the 3D surface modeling method based on sparse point cloud frames also applies to this 3D surface modeling device based on sparse point cloud frames, and will not be repeated here.

[0091] In summary, the 3D surface modeling device based on sparse point cloud frames according to embodiments of the present invention achieves 3D modeling of complex objects in a sparse point cloud environment through deep learning methods, thereby improving the modeling effect.

[0092] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0097] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0098] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0099] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0100] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0101] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0102] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0103] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A three-dimensional surface modeling method based on sparse point cloud frames, characterized in that, Includes the following steps: A lidar system is built, and real-world objects are scanned by the lidar system to obtain the corresponding point cloud dataset. The point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at the acquisition time node. A variational encoder is used to perform feature encoding on each frame of the point cloud in the point cloud frame sequence to obtain the global feature distribution and point-by-point local feature distribution corresponding to each frame of the point cloud; The point-by-point local feature distribution is used as the query matrix, and the global feature distribution is used as the key matrix, so that density-aware mutual attention is used to refine the point-by-point features to obtain the refined point-by-point local features corresponding to each frame of point cloud. The random sampling consensus algorithm is used to estimate the rigidity changes between adjacent frame point clouds based on the refined point-by-point local features, and pairwise registration is performed based on the rigidity changes between adjacent frame point clouds to obtain the point cloud registration results. Based on the point cloud registration results, geometric perception point cloud direct regression is used to obtain the coordinates of downsampled points to obtain the final 3D model; The point coordinates of downsampled points obtained by direct regression of the point cloud based on the point cloud registration results include: Based on the point cloud registration results, a variational encoder is used to perform feature encoding to obtain the global feature distribution and point-by-point local feature distribution corresponding to the point cloud registration results; The point-by-point local feature distribution is used as the key matrix, and the global feature distribution is used as the query matrix, so that the corresponding global features can be obtained after calculation using the attention mechanism; A multilayer perceptron is built to regress the point coordinates in order to obtain the sampling results; In the process of point cloud registration and point cloud sampling, a weighted graph is used for iterative optimization. The weighted graph includes a weighted distribution graph and a weighted importance graph. The weighted distribution graph includes neighborhood density features and allocates task-independent information for optimization. The weighted importance graph provides task-oriented information and the geometric importance of local shapes in the global shape.

2. The 3D surface modeling method based on sparse point cloud frames as described in claim 1, characterized in that, The point cloud dataset also includes point cloud frames obtained by scanning with a high-precision LiDAR scanner, in order to evaluate the modeling effect based on the chamfer loss.

3. The 3D surface modeling method based on sparse point cloud frames as described in claim 2, characterized in that, The modeling effectiveness is evaluated using the following formula: Where P is the time-series modeling point cloud of the low-precision lidar. Let be the i-th point in point cloud P, and O be the point cloud frame captured by a high-precision LiDAR scanner. Let N be the j-th point in point cloud O, N be the total number of points in point cloud P, and M be the total number of points in point cloud O. The chamfer distance is calculated to determine the nearest neighbor distance between each point and another point cloud, and the total distances between the two point clouds are summed to measure the difference between the point clouds.

4. A three-dimensional surface modeling device based on sparse point cloud frames, characterized in that, include: The acquisition module scans real-world objects using a lidar system to obtain a corresponding point cloud dataset. The point cloud dataset includes a sequence of point cloud frames acquired by a low-precision lidar and the pose of each point cloud frame at the acquisition time point. The density-aware point cloud registration module employs a variational encoder to encode the features of each frame of the point cloud in the point cloud frame sequence, obtaining the global feature distribution and point-by-point local feature distribution corresponding to each frame. The point-by-point local feature distribution is used as a query matrix, and the global feature distribution as a key matrix, to refine the point-by-point features using density-aware mutual attention, resulting in refined point-by-point local features for each frame. A random sampling consensus algorithm is used to estimate the rigidity changes between adjacent frame point clouds based on the refined point-by-point local features, and pairwise registration is performed based on these rigidity changes to obtain the point cloud registration result. A geometrically perceptive point cloud sampling module obtains the coordinates of downsampled points by direct regression of the point cloud based on the point cloud registration results, in order to obtain the final 3D model. The point coordinates of downsampled points obtained by direct regression of the point cloud based on the point cloud registration results include: Based on the point cloud registration results, a variational encoder is used to perform feature encoding to obtain the global feature distribution and point-by-point local feature distribution corresponding to the point cloud registration results; The point-by-point local feature distribution is used as the key matrix, and the global feature distribution is used as the query matrix, so that the corresponding global features can be obtained after calculation using the attention mechanism; A multilayer perceptron is built to regress the point coordinates in order to obtain the sampling results; In the process of point cloud registration and point cloud sampling, a weighted graph is used for iterative optimization. The weighted graph includes a weighted distribution graph and a weighted importance graph. The weighted distribution graph includes neighborhood density features and allocates task-independent information for optimization. The weighted importance graph provides task-oriented information and the geometric importance of local shapes in the global shape.

5. The three-dimensional surface modeling device based on sparse point cloud frames as described in claim 4, characterized in that, The point cloud dataset also includes point cloud frames obtained by scanning with a high-precision LiDAR scanner, in order to evaluate the modeling effect based on the chamfer loss.

6. The three-dimensional surface modeling device based on sparse point cloud frames as described in claim 5, characterized in that, The modeling effectiveness is evaluated using the following formula: Where P is the time-series modeling point cloud of the low-precision lidar. Let be the i-th point in point cloud P, and O be the point cloud frame captured by a high-precision LiDAR scanner. Let N be the j-th point in point cloud O, N be the total number of points in point cloud P, and M be the total number of points in point cloud O. The chamfer distance is calculated to determine the nearest neighbor distance between each point and another point cloud, and the total distances between the two point clouds are summed to measure the difference between the point clouds.

Images