Augmented reality-oriented three-dimensional scene reconstruction method and system

Abstract

The invention relates to the technical field of three-dimensional modeling, in particular to an augmented reality-oriented three-dimensional scene reconstruction method and system, and is used for solving the technical problem that an iterative nearest point algorithm cannot meet high real-time performance and high-precision reconstruction requirements required by augmented reality at the same time. The method comprises the following steps: acquiring a pose transformation relationship between multiple frames of point cloud data and adjacent frames of point cloud data; extracting a plurality of feature points in each frame of point cloud data, and determining a feature descriptor of each feature point; according to the feature descriptors of the feature points, calculating a matching weight between feature point pairs in two adjacent frames of point cloud data, and calculating a local matching error based on the matching weight and a pose transformation relationship; performing global matching optimization based on the local matching error of the plurality of feature point pairs and the distribution density of each feature point in the respective point cloud frame, and determining an optimal matching point pair set which minimizes the global matching error; and according to the optimal matching point pair set, fusing the multi-frame point cloud data, and reconstructing a three-dimensional scene model.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, specifically to 3D scene reconstruction methods and systems for augmented reality. Background Technology

[0002] 3D scene reconstruction technology for augmented reality aims to rapidly and accurately digitally recreate the real environment using digital models, serving as a crucial foundation for achieving stable integration and real-time interaction between virtual information and the physical world. This technology typically relies on sensors to collect multi-frame point cloud data of the real scene and, through a series of processes such as point cloud matching, fusion, and surface reconstruction, generates a 3D scene model that can be used in augmented reality systems.

[0003] Currently, point cloud registration typically uses the Iterative Closest Point (ILP) algorithm to iteratively optimize and find the spatial transformation relationship between two point cloud frames, thereby achieving gradual scene alignment and reconstruction. However, when dealing with scenes that are structurally complex, feature-repeating, or have significant viewpoint changes, the ILP algorithm is prone to problems such as low matching efficiency, high mismatch rate, and geometric distortion of reconstruction results, making it difficult to simultaneously meet the high real-time performance and high-precision reconstruction requirements of augmented reality. Summary of the Invention

[0004] To address the technical problem that existing iterative closest point algorithms cannot simultaneously meet the high real-time performance and high-precision reconstruction requirements of augmented reality, this application aims to provide a 3D scene reconstruction method for augmented reality. The specific technical solution adopted is as follows: Acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data; Multiple feature points are extracted from each frame of point cloud data, and a feature descriptor for each feature point is determined; the feature descriptor is used to characterize the local structural information at the location of the feature point. Based on the feature descriptors of the feature points, calculate the matching weight between feature point pairs in two adjacent frames of point cloud data, and calculate the local matching error based on the matching weight and pose transformation relationship. Based on the local matching error of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, global matching optimization is performed to determine the optimal set of matching point pairs that minimizes the global matching error. Based on the optimal set of matching point pairs, multi-frame point cloud data are fused to reconstruct a 3D scene model.

[0005] In one possible implementation, extracting multiple feature points from each frame of point cloud data includes: performing the following process for each data point in each frame of point cloud data to determine multiple feature points in each frame of point cloud data: determining a set of other data points within the spatial neighborhood of the target data point; the target data point being any data point in each frame of point cloud data; calculating the covariance matrix of the other data point sets and determining the eigenvalues ​​of the covariance matrix; determining the coefficient of variation of the target data point based on the eigenvalues ​​of the target data point and the eigenvalues ​​of the covariance matrix; the coefficient of variation being used to characterize the structural saliency of the data point; determining the probability value of the target data point as a feature point based on the curvature and the coefficient of variation of the target data point; and determining the target data point as a feature point if the probability value is greater than a preset threshold.

[0006] In one possible implementation, determining the feature descriptor for each feature point includes: when the target data point is determined to be a feature point, using a vector containing the curvature and coefficient of variation of the target data point as the feature descriptor of the target data point.

[0007] In one possible implementation, the matching weight between feature point pairs in two adjacent frames of point cloud data is calculated based on the feature descriptors of the feature points. This includes: determining target feature point pairs in two adjacent frames of point cloud data; the target feature point pair is any feature point pair in the two adjacent frames of point cloud data; obtaining the first feature descriptor of the first feature point and the second feature descriptor of the second feature point in the target feature point pair; calculating the degree of difference between the first feature descriptor and the second feature descriptor; and determining the matching weight of the target feature point pair based on the degree of difference, the probability value corresponding to the first feature point, and the probability value corresponding to the second feature point.

[0008] In one possible implementation, the local matching error is calculated based on the matching weights and the pose transformation relationship, including: performing a pose transformation on the first feature point based on the pose transformation relationship to obtain a third feature point; calculating the Euclidean distance between the third feature point and the second feature point; weighting the Euclidean distance based on the matching weights to obtain a weighted distance between a pair of feature points; determining the weighted distances of multiple pairs of feature points, and using the sum of the squares of the weighted distances of multiple pairs of feature points as the local matching error.

[0009] In one possible implementation, global matching optimization is performed based on the local matching errors of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame to determine the optimal set of matching point pairs that minimizes the global matching error. This includes: constructing multiple candidate set of matching point pairs; each candidate set of matching point pairs contains one or more feature point pairs between two adjacent frames of point cloud data; determining the density correction factor for each feature point pair based on the distribution density of each feature point in its respective point cloud frame; determining the global matching error of each candidate set of matching point pairs based on the local matching error and density correction factor of multiple sets of feature point pairs in each candidate set of matching point pairs; and determining the candidate set of matching point pairs with the minimum global matching error as the optimal set of matching point pairs.

[0010] In one possible implementation, the density correction factor of the feature point pair is determined based on the distribution density of each feature point in its respective point cloud frame, including: determining the target feature point pair in two adjacent frames of point cloud data; calculating the number of feature points of the first feature point in the target feature point pair in the local neighborhood as the first distribution density; calculating the number of feature points of the second feature point in the target feature point pair in the local neighborhood as the second distribution density; and determining the density correction factor of the target feature point pair according to the first distribution density and the second distribution density.

[0011] One possible implementation involves acquiring multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data, including: acquiring point cloud data of a real scene using a LiDAR or depth camera to obtain multi-frame point cloud data; recording the viewpoint transformation between adjacent point cloud frames by monitoring the movement of the user's viewpoint, and using the viewpoint transformation as the pose transformation relationship.

[0012] In one possible implementation, multi-frame point cloud data is fused based on the optimal set of matching point pairs to reconstruct a 3D scene model. This includes: calculating precise transformation parameters to align multi-frame point cloud data to the same coordinate system based on the optimal set of matching point pairs; transforming multi-frame point cloud data to the same coordinate system and fusing them based on the precise transformation parameters to form a dense 3D point cloud; and performing surface reconstruction on the fused 3D point cloud to generate a 3D scene model with geometric details and topological structure.

[0013] This application also provides a 3D scene reconstruction system for augmented reality, the system comprising: The acquisition unit is used to acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data. The extraction unit is used to extract multiple feature points from each frame of point cloud data and determine the feature descriptor for each feature point; the feature descriptor is used to characterize the local structural information of the feature point's location. The local matching unit is used to calculate the matching weight between feature point pairs in two adjacent frames of point cloud data based on the feature descriptors of the feature points, and to calculate the local matching error based on the matching weight and pose transformation relationship. The global optimization unit is used to perform global matching optimization based on the local matching error of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, and to determine the optimal set of matching point pairs that minimizes the global matching error. The 3D reconstruction unit is used to fuse multi-frame point cloud data based on the optimal set of matching point pairs to reconstruct a 3D scene model.

[0014] This application offers the following advantages: By constructing a complete process encompassing feature point extraction, descriptor-based weighted matching, local error calculation, global distribution density optimization, and final scene reconstruction, this application effectively improves the overall performance of point cloud registration. By introducing local structural features into the matching weight calculation and utilizing global density information for optimization and correction, high-precision matching results can still be obtained in complex scenes. This provides more accurate data support for the subsequent generation of accurate and reliable 3D scene models. When facing scenes with complex structures, repetitive features, or significant viewpoint changes, it avoids problems such as low matching efficiency, high mismatch rates, and geometric distortion of reconstruction results, thus meeting the high real-time and high-precision reconstruction requirements of augmented reality. Attached Figure Description

[0015] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a 3D scene reconstruction method for augmented reality provided in one embodiment of this application. Figure 1 ; Figure 2 A flowchart illustrating a 3D scene reconstruction method for augmented reality provided in one embodiment of this application. Figure 2 ; Figure 3 A flowchart illustrating a 3D scene reconstruction method for augmented reality provided in one embodiment of this application. Figure 3 ; Figure 4 A flowchart illustrating a 3D scene reconstruction method for augmented reality provided in one embodiment of this application. Figure 4 ; Figure 5This is a schematic diagram of the system architecture of a 3D scene reconstruction system for augmented reality provided in one embodiment of this application. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by this application to achieve the intended inventive purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a three-dimensional scene reconstruction method and system for augmented reality proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0019] Unless otherwise specified, the normalization function Norm() mentioned in this application uses maximum and minimum value normalization. The maximum and minimum values ​​are preset empirical extreme values ​​derived from a large amount of historical experimental data. If the calculation result exceeds the [0,1] interval, a truncation function is used to limit it to the [0,1] range (i.e., if the result is less than 0, it is taken as 0; if it is greater than 1, it is taken as 1) to eliminate the influence of outliers on the evaluation index.

[0020] The following description, in conjunction with the accompanying drawings, details a specific scheme for a three-dimensional scene reconstruction method for augmented reality provided in this application.

[0021] Please see Figure 1 It illustrates a flowchart of a three-dimensional scene reconstruction method for augmented reality provided in one embodiment of this application, as follows: Figure 1 As shown, the method includes the following steps: Step 101: Obtain multi-frame point cloud data of the real scene and the pose transformation relationship between adjacent frame point cloud data.

[0022] As one possible implementation, this step can be implemented as follows: acquiring point cloud data of the real scene through LiDAR or depth camera to obtain multi-frame point cloud data; recording the viewpoint transformation between adjacent frames of point cloud data by monitoring the movement of the user's viewpoint, and using the viewpoint transformation as the pose transformation relationship.

[0023] Specifically, 3D point cloud data is collected using methods such as LiDAR, and point clouds from different viewpoints are obtained by recording user interactions (such as monitoring viewpoint movement), thus generating a point cloud frame sequence. where i is the frame number. This represents the point cloud data corresponding to the i-th frame; these point clouds are used to reflect the 3D information of the real scene and serve as the basis for 3D scene reconstruction. To achieve point cloud matching, point cloud analysis needs to be performed on adjacent frames, and the viewpoint transformation relationship (i.e., pose transformation relationship) between adjacent frames needs to be recorded. This pose transformation relationship is denoted by T and satisfies... Optionally, the pose transformation relationship T is a rigid body transformation matrix characterizing rotation and translation in three-dimensional space, mathematically expressed as a 4×4 homogeneous coordinate transformation matrix containing rotation and translation vectors. This pose transformation relationship can be calculated based on the relative motion state of the sensors during the acquisition of point cloud data from adjacent frames, using visual odometry, inertial measurement units, or pose estimation algorithms combining camera intrinsic and extrinsic parameters. For example... , where R is a 3×3 rotation matrix and t is a 3×1 translation vector. This matrix is ​​used to describe the rotation and translation transformation relationship of the point cloud from frame i to frame i+1.

[0024] Step 102: Extract multiple feature points from each frame of point cloud data and determine the feature descriptor for each feature point.

[0025] Among them, the feature descriptor is used to characterize the local structural information of the location of the feature point.

[0026] As one possible implementation, this step quantifies the likelihood of a data point being a feature point by evaluating its curvature and structural saliency within its neighborhood, and then selects feature points based on this likelihood. For each identified feature point, its curvature is combined with a coefficient of variation representing local structural saliency to construct a feature descriptor. This descriptor effectively expresses the uniqueness of the feature point's location, thus supporting subsequent accurate feature matching. Based on this, key points that can significantly characterize the local geometry of the scene can be selected from the original point cloud, and a discriminative feature description vector is generated for each feature point, laying the foundation for subsequent point cloud matching.

[0027] Step 103: Calculate the matching weight between feature point pairs in two adjacent frames of point cloud data based on the feature descriptors of the feature points, and calculate the local matching error based on the matching weight and pose transformation relationship.

[0028] Optionally, in this step, the feature point pairs to be matched in two adjacent frames of point cloud data are first determined, and the feature descriptors corresponding to each feature point are obtained; the feature difference between the point pairs is calculated based on the feature descriptors, and the matching weight is determined in combination with the saliency index of the feature points themselves; after that, the feature points are spatially transformed based on the pose transformation relationship, and the transformed position deviation is weighted and fused according to the matching weight to obtain the local matching error.

[0029] This step introduces a comprehensive evaluation mechanism that combines feature descriptor differences and feature point saliency, assigning appropriate weights to feature point pairs with different levels of reliability. This allows subsequent error calculations to focus more on high-confidence matching point pairs. By using weighted local matching errors as the target for registration optimization, robustness to noise and outliers is effectively improved, laying an accurate local alignment foundation for subsequent global optimization.

[0030] Step 104: Based on the local matching errors of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, perform global matching optimization to determine the optimal set of matching point pairs that minimizes the global matching error.

[0031] Optionally, in this step, based on the established initial feature point correspondence, multiple candidate sets of matching point pairs are generated, each set representing a possible global matching scheme; for each candidate set, the global matching error corresponding to the candidate set is calculated by combining the local matching error of each feature point pair and its distribution density in their respective point clouds; finally, based on the optimization selection strategy, the set of matching point pairs that minimizes the global matching error is determined from all candidate sets as the optimal matching result.

[0032] This step introduces feature point distribution density as a constraint on spatial distribution, simultaneously considering local matching accuracy and global spatial distribution consistency during the optimization process. This effectively avoids mismatch problems caused by local structural similarity, significantly improves the accuracy and robustness of point cloud registration in complex scenes, and provides a key guarantee for the final high-quality 3D reconstruction.

[0033] Step 105: Based on the optimal set of matching point pairs, fuse the point cloud data from multiple frames to reconstruct the 3D scene model.

[0034] As one possible implementation, this step can be implemented as follows: Based on the optimal set of matching point pairs, calculate the precise transformation parameters for aligning multi-frame point cloud data to the same coordinate system; based on the precise transformation parameters, transform the multi-frame point cloud data to the same coordinate system and fuse them to form a dense 3D point cloud; perform surface reconstruction on the fused 3D point cloud to generate a 3D scene model with geometric details and topological structure. The surface reconstruction process described above can use the Poisson reconstruction algorithm or other reconstruction algorithms, and this application does not limit the specific algorithms used.

[0035] Based on the above technical solution, this application effectively improves the overall performance of point cloud registration by constructing a complete process that includes feature point extraction, descriptor-based weighted matching, local error calculation, global distribution density optimization, and final scene reconstruction. By introducing local structural features into the matching weight calculation and using global density information for optimization and correction, high-precision matching results can still be obtained in complex scenes. This provides more accurate data support for the subsequent generation of accurate and reliable 3D scene models. When facing scenes with complex structures, repetitive features, or significant viewpoint changes, it avoids problems such as low matching efficiency, high mismatch rate, and geometric distortion of reconstruction results, thus meeting the high real-time and high-precision reconstruction requirements of augmented reality.

[0036] It should be noted that in the process of 3D scene reconstruction, the Iterative ClosestPoint (ICP) matching algorithm mainly relies on the relative positional relationship between point clouds to achieve matching between different frames. However, this method does not fully consider the point cloud distortion caused by changes in viewpoint. Although point clouds at different viewpoints may correspond to the same structural features, due to the influence of distortion, point clouds at the same spatial location will show significant differences under different viewpoints, thus affecting the matching accuracy of adjacent point cloud frames. Therefore, relying solely on the relative position of point clouds for feature point selection and matching has limitations and needs to be further analyzed in conjunction with point cloud structural features. Point cloud structural features can be represented by feature points and descriptors. The process of obtaining feature points and descriptors is explained in detail below: like Figure 2 As shown, in one possible implementation, the process of extracting multiple feature points from each frame of point cloud data in step 102 specifically includes: performing the following process for each data point in each frame of point cloud data to determine multiple feature points in each frame of point cloud data: Step 201: Determine the set of other data points within the spatial neighborhood of the target data point.

[0037] The target data point is any data point in each frame of point cloud data.

[0038] Optionally, this application determines the spatial neighborhood of the target data point using the K-nearest neighbor method or the spherical neighborhood method. Taking the K-nearest neighbor method as an example, with the target data point as the center, the K points in three-dimensional space that are closest to the target data point in Euclidean distance are selected to form the spatial neighborhood of the target data point. K is a preset integer, for example, K=30, and its value is usually determined experimentally based on the point cloud density to ensure that the neighborhood can effectively reflect the local geometric features.

[0039] Taking the spherical neighborhood method as an example, a spherical region is defined with the target data point as the center and a preset length R as the radius. All points within this region constitute the neighborhood. The value of the radius R is related to the point cloud density, for example, it is set to 5-10 times the average spacing of the point cloud to ensure that the neighborhood contains a sufficient number of points for feature calculation.

[0040] Step 202: Calculate the covariance matrix of other data point sets and determine the eigenvalues ​​of the covariance matrix.

[0041] Specifically, for each point in the point cloud, a neighborhood set of points is constructed. Within this set, the covariance matrix of the point cloud data is calculated, and then the eigenvalues ​​and eigenvectors of the covariance are obtained. Eigenvalues ​​and eigenvectors reflect the local structural distribution characteristics of the current point cloud. For example, if the eigenvalue is small along a certain eigenvector direction, it indicates that the point cloud is relatively concentrated in that direction. By analyzing the consistency between eigenvalues, the stability of the local point cloud structure can be determined: higher consistency indicates a more uniform point cloud structure in that region, making it less prone to significant changes, and thus lowering the likelihood of that point being a feature point.

[0042] Step 203: Determine the coefficient of variation of the target data points based on the eigenvalues ​​of the target data points and the eigenvalues ​​of the covariance matrix.

[0043] The coefficient of variation is used to characterize the structural significance of data points.

[0044] Optionally, this step quantifies the local structural salience of the target point relative to its surrounding environment by calculating the degree of difference between the local structural feature vectors of the target point and its neighbors, and obtaining the coefficient of variation of these differences. The larger the coefficient of variation, the more obvious the structural abrupt change characteristics at the location of the point.

[0045] As an example, the coefficient of variation is: Where λ represents the feature vector of the target data point (a three-dimensional vector in three-dimensional space). This represents the feature vector of the k-th point cloud within the neighborhood of the target data point; (⋅) represents the anomaly coefficient value, calculated as the ratio of the standard deviation to the mean of the input value, used to measure the degree of anomaly of the current point cloud feature relative to the neighboring point cloud. This indicates the calculation of Euclidean distance.

[0046] Step 204: Determine the probability value of the target data point as a feature point based on the curvature and coefficient of variation of the target data point.

[0047] Optionally, in real-world scenarios, such as areas with unevenness or undulation on a wall, although a local area may contain multiple structural features, the sheer number of major structures may lead to a high degree of similarity in the point cloud structure within the neighborhood. In such cases, it is necessary to combine curvature information to distinguish different structures, for example, by using the difference in curvature between flat and uneven areas of the wall for feature discrimination.

[0048] The probability of a target data point being a feature point is determined by the curvature and coefficient of variation of the target data point. Specifically, curvature... The calculation method can be referred to relevant technologies, and this application does not limit it.

[0049] For example, the probability value of a target data point as a feature point. Satisfy the following formula: in, The normalized value of the curvature of the target data point. The coefficient of variation is the largest possible value of the point cloud. The larger the coefficient of variation, the more unique the point cloud is compared to the surrounding point clouds, and thus the more likely it is to be a feature point. At the same time, if it has a large curvature, it reflects that it is in a region of obvious geometric change (such as edges, corners, etc.).

[0050] Step 205: If the probability value is greater than the preset threshold, determine the target data point as the feature point.

[0051] Optionally, the above-mentioned preset threshold can be set according to the statistical distribution of the probability values ​​in the point cloud, for example, set to the 70% to 85% quantile of the probability value distribution of all data points, so as to obtain a sufficient number of feature points for matching while ensuring the significance of features.

[0052] Based on this, this application calculates the eigenvalues ​​of the covariance matrix of the local neighborhood of the point cloud and combines curvature and coefficient of variation to quantify the probability of each point being a feature point, thus achieving precise selection of feature points. This method can effectively distinguish between ordinary planar points and structural points with significant geometric features, ensuring that the extracted feature points can stably represent the key structures of the scene, providing a high-quality data foundation for subsequent high-precision registration, and avoiding matching failures or errors caused by the lack of representativeness of feature points.

[0053] like Figure 2 As shown, in one possible implementation, the process of determining the feature descriptor for each feature point in step 102 above specifically includes: Step 206: If the target data point is determined to be a feature point, the vector containing the normalized value of the curvature of the target data point and the coefficient of variation is used as the feature descriptor of the target data point.

[0054] Optionally, this application normalizes the curvature of the target data points. and coefficient of variation The combination forms a feature descriptor s for the target data points, satisfying: .

[0055] Based on this, this application combines curvature and coefficient of variation into a feature descriptor vector, giving each feature point a compact and discriminative feature representation. This descriptor simultaneously includes the local curvature of the point and its structural saliency relative to its neighborhood, comprehensively characterizing the unique identity of the feature point. This enables more accurate and reliable matching of truly corresponding feature point pairs in subsequent matching processes, effectively reducing the occurrence of false matches.

[0056] like Figure 3 As shown, in one possible implementation, the process of calculating the matching weight between feature point pairs in two adjacent frames of point cloud data based on the feature descriptors of the feature points in step 103 specifically includes: Step 301: Determine the target feature point pairs in two adjacent frames of point cloud data.

[0057] The target feature point pair is any pair of feature points in two adjacent frames of point cloud data. The two feature points in this pair are a pair of feature points that are associated through the initial pose transformation, located in adjacent frames and have similar local structural features.

[0058] Optionally, the target feature point is denoted as the first feature point for the feature point located in the previous frame. The feature point located in the next frame is denoted as the second feature point. .

[0059] Step 302: Obtain the first feature descriptor of the first feature point and the second feature descriptor of the second feature point in the target feature point pair.

[0060] Specifically, in the point cloud matching process, it is necessary to evaluate the similarity of feature points between adjacent frames. Matched points correspond to similar feature descriptors. To this end, firstly, based on the transformation relationship T from the device motion state (e.g., obtained through camera calibration), the point cloud of the previous frame f1 is spatially transformed to obtain the transformed point cloud position. Subsequently, based on the transformed positional relationship, the feature descriptor of the transformed point cloud can be recalculated using the same feature extraction method, denoted as T·s (where s is the original feature descriptor). Since the viewpoint transformation does not change the local structural features of the point cloud itself, the transformed feature descriptor can directly use the feature descriptor before the transformation, i.e., T·s = s.

[0061] As an example, the first feature descriptor of the first feature point is denoted as... The second feature descriptor of the second feature point is denoted as .

[0062] Step 303: Calculate the degree of difference between the first feature descriptor and the second feature descriptor.

[0063] Optionally, this step can be implemented as follows: Obtain the feature descriptors of the first feature point and the second feature point. Then, calculate the Euclidean distance between these two feature descriptor vectors; this distance value represents the degree of difference. The Euclidean distance is calculated by summing the squares of the differences between corresponding components in the two vectors and then taking the square root. The smaller the degree of difference, the more similar the two feature points are in their local structure; the larger the degree of difference, the more significant the difference in their local structure. This degree of difference is used to subsequently calculate the matching weights for feature point pairs to evaluate the reliability of the match.

[0064] As an example, the degree of difference between the first feature descriptor and the second feature descriptor is expressed as follows: , This indicates the calculation of Euclidean distance.

[0065] Step 304: Determine the matching weight of the target feature point pair based on the degree of difference, the probability value corresponding to the first feature point, and the probability value corresponding to the second feature point.

[0066] Specifically, by comparing the transformed point cloud features with the actual point cloud features acquired in the current frame, if a point cloud not only has a position close to the corresponding point in the current frame after transformation, but also maintains a high degree of similarity in its local structural features, then the point cloud has a high matching reliability. However, different feature points have different importance in matching: for example, feature points that belong to flat wall areas in adjacent frames (i.e., low feature saliency) should be allowed a larger tolerance in matching, even if there is a certain positional deviation, to avoid mismatches; conversely, for points with salient features (such as corners, concave and convex areas), whose local structures differ significantly from their surroundings, more precise matching is required to avoid ignoring key structures in the 3D scene and affecting the accuracy of reconstruction.

[0067] Therefore, in the improved ICP matching process, higher weights should be assigned to point pairs with significant features. As an example, the point clouds of a pair of adjacent point cloud frames in adjacent frames are defined. The matching weight between them is It satisfies the following formula: in, For the first feature point mentioned above, For the second feature point mentioned above, The probability value corresponding to the first feature point. This represents the probability value corresponding to the second feature point. The first feature descriptor represents the first feature point. This represents the second feature descriptor of the second feature point, where norm represents the normalization operation. These are parameter tuning coefficients, and their values ​​should be very small integers (e.g., 0.01) to avoid denominators of 0.

[0068] like Figure 3 As shown, in one possible implementation, the process of calculating the local matching error based on the matching weights and pose transformation relationship in step 103 specifically includes: Step 305: Perform pose transformation on the first feature point based on the pose transformation relationship to obtain the third feature point.

[0069] As an example, for the i-th Feature points in the previous frame (That is, the first feature point) is transformed in pose to obtain the third feature point, denoted as . .

[0070] Step 306: Calculate the Euclidean distance between the third feature point and the second feature point.

[0071] Step 307: Weight the Euclidean distance based on the matching weight to obtain the weighted distance between a pair of feature points.

[0072] Step 308: Determine the weighted distance of multiple pairs of feature points, and use the sum of the squares of the weighted distances of multiple pairs of feature points as the local matching error.

[0073] As an example, local matching error Satisfy the following formula: in, This represents the local matching error, where i represents... , This represents the set of indices for all candidate matching feature point pairs. Indicates the i-th Feature points in the previous frame, Indicates the i-th representation Feature points in the next frame, This involves transforming the pose of the first feature point to obtain the third feature point. This process minimizes the objective function of the ICP algorithm to obtain the optimal matching point cloud and its local matching error. This method improves matching accuracy by introducing point cloud structural features as matching weights, overcoming the problem that traditional ICP algorithms rely solely on point-location relationships while ignoring structural consistency.

[0074] Based on the above technical solution, this application introduces matching weights based on feature descriptor differences and feature point saliency to optimize the distance error in the traditional ICP algorithm. This ensures that the registration process prioritizes high-precision alignment of salient feature points, thereby effectively improving matching accuracy, robustness, and the quality of the final 3D reconstruction model, and meeting the real-time and accuracy requirements of augmented reality applications.

[0075] like Figure 4 As shown, in one possible implementation, the process of performing global matching optimization based on the local matching errors of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, and determining the optimal set of matching point pairs that minimizes the global matching error in step 104, specifically includes: Step 401: Construct a set of multiple candidate matching point pairs.

[0076] Each candidate matching point pair set contains one or more feature point pairs between two adjacent frames of point cloud data.

[0077] It should be noted that in the process of 3D scene reconstruction, the traditional ICP algorithm incurs high computational overhead due to the need to traverse the entire point cloud, affecting the response speed of the augmented reality system. To improve efficiency, the algorithm first relies on the feature probability values ​​of the point cloud. Threshold filtering is performed, and the filtered point cloud is used as feature points, and the point pairs participating in the matching are specified. ( ) must be feature points. When analyzing the point cloud structure, the maximum matching error threshold for its feature points is set based on the previous frame of the point cloud. The point cloud that meets the matching criteria in the next frame is taken as the candidate target point cloud set M. If set M is empty, the feature point is considered to have failed to match (e.g., due to occlusion caused by a change in viewpoint), and only the successfully matched feature points are processed subsequently. Optionally, a maximum matching error threshold can be set. It can be set according to the point cloud resolution and expected matching accuracy, for example, set to 3 to 5 times the average point distance of the point cloud.

[0078] Specifically, to evaluate the global consistency of matching, multiple candidate matching schemes need to be constructed and optimized. The specific process is as follows: Obtain all successfully matched feature points in the current frame (i.e., the set M of candidate matching points in the next frame is not empty), and record their positions to construct the source feature point position matrix. Using pose transformation T, calculate the theoretical position of all points in P transformed to the target frame coordinate system. .

[0079] Construct a candidate matching point pair set MP, which represents a global matching scheme, specifically including: for each feature point in the source point cloud frame. Select a point from its candidate matching point set M. (For example, the i-th point), forming a matching pair ( , Combining all such matching pairs forms a set of candidate matching pairs. The above For feature points The elements in the corresponding target point cloud set M.

[0080] Calculate each pair of feature points in the MP set ( , Local matching error This error reflects the similarity between two points within a local area.

[0081] Because it can be for each By selecting different points in M, multiple distinct sets of MP can be constructed. The optimal solution is selected by comparing the global consistency of these sets (e.g., using a global matching error function), thus ensuring that all matched feature points maintain the highest overall consistency under the viewpoint transformation T.

[0082] Step 402: Determine the density correction factor for each feature point pair based on the distribution density of each feature point in its respective point cloud frame.

[0083] As one possible implementation, this step can be implemented as follows: determine the target feature point pair in two adjacent frames of point cloud data; calculate the number of feature points of the first feature point in the target feature point pair in the local neighborhood as the first distribution density; calculate the number of feature points of the second feature point in the target feature point pair in the local neighborhood as the second distribution density; and determine the density correction factor of the target feature point pair based on the first distribution density and the second distribution density.

[0084] Specifically, for any feature point pair in the candidate matching point pair set MP First, we need to calculate separately. Distribution density in the first point cloud frame ,as well as Distribution density in the second point cloud frame .

[0085] The method for calculating the distribution density of feature points is as follows: using target feature points (e.g., Centered on the target feature point, define a spherical neighborhood with a radius r equal to a second preset threshold (e.g., 0.5% to 1% of the diagonal length of the point cloud scene bounding box). Count the number of all feature points within this spherical neighborhood, excluding the target feature point itself, and use this count as the distribution density of the target feature point. .right Using the same neighborhood radius r and statistical method, its distribution density is obtained. .

[0086] Then, the density correction factor for the feature point pair is calculated based on the two distribution densities. Density correction factor The calculation formula is: in, Represents the first distribution density With the second distribution density The absolute difference This represents the sum of the first and second distribution densities. Density correction factor. This measures whether the distribution of local feature points in their respective point cloud frames is consistent for the matched point pairs. When the two densities differ significantly... A value greater than 1 indicates that the matching pair may have inconsistencies in its global spatial distribution, and will be given a larger penalty weight in subsequent global error calculations; when the two densities are similar... A value close to 1 indicates that the matching pair has good consistency in distribution. These are parameter tuning coefficients, and their values ​​should be very small integers (e.g., 0.01) to avoid denominators of 0.

[0087] Step 403: Based on the local matching error and density correction factor of multiple feature point pairs in each candidate matching point pair set, determine the global matching error of each candidate matching point pair set.

[0088] Specifically, for any set of candidate matching point pairs MP, its global matching error Defined as the local matching error of all feature point pairs in the set. After their respective density correction factors The weighted sum. Its calculation formula is: Where N is the number of feature point pairs in the candidate matching point pair set MP. It is based on the local matching error value. The candidate matching set was comprehensively evaluated for its local matching accuracy (by... (reflection) and consistency of global spatial distribution (by) (Regulation) Two aspects of performance. The smaller the value, the higher the overall matching quality of the candidate matching set, indicating that it is accurate in local matching and consistent in global feature distribution.

[0089] Step 404: Determine the set of candidate matching point pairs with the minimum global matching error, which is the optimal set of matching point pairs.

[0090] Specifically, in order to find the match from all possible candidate matching pairs that minimizes the global matching error The smallest set That is, to solve: Optionally, since constructing all possible combinations of candidate matching points would lead to a combinatorial explosion problem, this embodiment employs a greedy algorithm to efficiently find an approximate optimal solution.

[0091] Based on the above technical solution, this embodiment of the invention solves the ambiguity problem that may exist in local matching by constructing multiple candidate matching sets and calculating their global error. Specifically, by introducing a correction factor based on the feature point distribution density, the matching evaluation is expanded from a single local geometric similarity to a consideration of the global distribution consistency of matching point pairs in their respective scenes. This mechanism can adaptively suppress mismatch weights caused by local similarity in feature-redundant regions (such as repeating structures), while enhancing the matching constraint in feature-sparse regions, thereby effectively improving the overall accuracy and robustness of point cloud matching in complex and non-uniform scenes, laying a reliable foundation for subsequent high-precision 3D reconstruction.

[0092] like Figure 5 As shown in the figure, this application provides a 3D scene reconstruction system for augmented reality, the system comprising: The acquisition unit 501 is used to acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data.

[0093] Extraction unit 502 is used to extract multiple feature points from each frame of point cloud data and determine the feature descriptor for each feature point. The feature descriptor is used to characterize the local structural information of the feature point's location.

[0094] The local matching unit 503 is used to calculate the matching weight between feature point pairs in two adjacent frames of point cloud data based on the feature descriptors of the feature points, and to calculate the local matching error based on the matching weight and pose transformation relationship.

[0095] The global optimization unit 504 is used to perform global matching optimization based on the local matching error of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, and to determine the optimal set of matching point pairs that minimizes the global matching error.

[0096] The 3D reconstruction unit 505 is used to fuse multi-frame point cloud data based on the optimal set of matching point pairs to reconstruct a 3D scene model.

[0097] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0098] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for reconstructing a 3D scene for augmented reality, characterized in that, The method includes: Acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data; Multiple feature points are extracted from each frame of point cloud data, and a feature descriptor for each feature point is determined; the feature descriptor is used to characterize the local structural information of the feature point's location. Based on the feature descriptors of the feature points, calculate the matching weight between feature point pairs in two adjacent frames of point cloud data, and calculate the local matching error based on the matching weights and the pose transformation relationship; Based on the local matching error of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, global matching optimization is performed to determine the optimal set of matching point pairs that minimizes the global matching error. Based on the optimal set of matching point pairs, the multi-frame point cloud data are fused to reconstruct a 3D scene model.

2. The three-dimensional scene reconstruction method for augmented reality according to claim 1, characterized in that, Extracting multiple feature points from each frame of point cloud data includes: performing the following process for each data point in each frame of point cloud data to determine multiple feature points in each frame of point cloud data: Determine the set of other data points within the spatial neighborhood of the target data point; the target data point is any data point in each frame of point cloud data. Calculate the covariance matrix of the other data point set, and determine the eigenvalues ​​of the covariance matrix; The coefficient of variation of the target data point is determined based on the eigenvalues ​​of the target data point and the eigenvalues ​​of the covariance matrix; the coefficient of variation is used to characterize the structural significance of the data point. Based on the curvature and coefficient of variation of the target data point, determine the probability value of the target data point as a feature point; If the probability value is greater than a preset threshold, the target data point is determined to be the feature point.

3. The three-dimensional scene reconstruction method for augmented reality according to claim 2, characterized in that, Determine the feature descriptor for each feature point, including: If the target data point is determined to be the feature point, the vector containing the curvature and coefficient of variation of the target data point is used as the feature descriptor of the target data point.

4. The three-dimensional scene reconstruction method for augmented reality according to claim 2, characterized in that, Based on the feature descriptors of the feature points, calculate the matching weights between feature point pairs in two adjacent frames of point cloud data, including: Determine a pair of target feature points in two adjacent frames of point cloud data; the pair of target feature points is any pair of feature points in the two adjacent frames of point cloud data. Obtain the first feature descriptor of the first feature point and the second feature descriptor of the second feature point in the target feature point pair; Calculate the degree of difference between the first feature descriptor and the second feature descriptor; Based on the degree of difference, the probability value corresponding to the first feature point, and the probability value corresponding to the second feature point, the matching weight of the target feature point pair is determined.

5. The three-dimensional scene reconstruction method for augmented reality according to claim 4, characterized in that, Based on the matching weights and the pose transformation relationship, the local matching error is calculated, including: Based on the pose transformation relationship, the first feature point is transformed to obtain the third feature point; Calculate the Euclidean distance between the third feature point and the second feature point; The Euclidean distance is weighted based on the matching weight to obtain the weighted distance of the pair of feature points; Determine the weighted distances of multiple pairs of feature points, and use the sum of the squares of the weighted distances of the multiple pairs of feature points as the local matching error.

6. The three-dimensional scene reconstruction method for augmented reality according to claim 1, characterized in that, Based on the local matching errors of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, global matching optimization is performed to determine the optimal set of matching point pairs that minimizes the global matching error, including: Construct multiple candidate matching point pair sets; each candidate matching point pair set contains one or more feature point pairs between two adjacent frames of point cloud data; Based on the distribution density of each feature point in its respective point cloud frame, the density correction factor for each feature point pair is determined. The global matching error of each candidate matching point pair set is determined based on the local matching error and density correction factor of multiple feature point pairs in each candidate matching point pair set. The set of candidate matching point pairs with the minimum global matching error is determined as the optimal matching point pair set.

7. The three-dimensional scene reconstruction method for augmented reality according to claim 6, characterized in that, Based on the distribution density of each feature point in its respective point cloud frame, the density correction factor for the feature point pair is determined, including: Identify target feature point pairs in two adjacent frames of point cloud data; Calculate the number of feature points in the local neighborhood of the first feature point in the target feature point pair, and use it as the first distribution density; Calculate the number of feature points in the local neighborhood of the second feature point in the target feature point pair, and use it as the second distribution density; The density correction factor of the target feature point pair is determined based on the first distribution density and the second distribution density.

8. The three-dimensional scene reconstruction method for augmented reality according to claim 1, characterized in that, Acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data, including: The point cloud data of the real scene is collected by LiDAR or depth camera to obtain the multi-frame point cloud data. By monitoring the movement of the user's viewpoint, the viewpoint transformation between adjacent point cloud frames is recorded, and the viewpoint transformation is used as the pose transformation relationship.

9. The three-dimensional scene reconstruction method for augmented reality according to claim 1, characterized in that, Based on the optimal set of matching point pairs, the multi-frame point cloud data is fused to reconstruct a 3D scene model, including: Based on the optimal set of matching point pairs, calculate the precise transformation parameters that align the multi-frame point cloud data to the same coordinate system. Based on the precise transformation parameters, the multi-frame point cloud data are transformed to the same coordinate system and fused to form a dense three-dimensional point cloud. The surface of the fused 3D point cloud is reconstructed to generate a 3D scene model with geometric details and topological structure.

10. A three-dimensional scene reconstruction system for augmented reality, characterized in that, The system includes: The acquisition unit is used to acquire multi-frame point cloud data of a real scene and the pose transformation relationship between adjacent frame point cloud data. An extraction unit is used to extract multiple feature points from each frame of point cloud data and determine the feature descriptor for each feature point; the feature descriptor is used to characterize the local structural information of the location of the feature point. The local matching unit is used to calculate the matching weight between feature point pairs in two adjacent frames of point cloud data based on the feature descriptors of the feature points, and to calculate the local matching error based on the matching weights and the pose transformation relationship. The global optimization unit is used to perform global matching optimization based on the local matching error of multiple feature point pairs and the distribution density of each feature point in its respective point cloud frame, and to determine the optimal set of matching point pairs that minimizes the global matching error. The 3D reconstruction unit is used to fuse the multi-frame point cloud data according to the optimal matching point pair set to reconstruct a 3D scene model.

Images