Real-time 3D reconstruction method for large scene based on 3R

Abstract

The method for large scene real-time three-dimensional reconstruction based on 3R belongs to the field of computer vision and three-dimensional reconstruction technology, and comprises the following steps: acquiring images of an RGB monocular camera; constructing a multi-level reconstruction architecture of "ordinary frame-key frame-scene frame", and dynamically screening frame categories based on inter-frame point cloud overlap and reconstruction confidence; performing multi-frame three-dimensional reconstruction based on 3R on ordinary frames and related scene frames to generate point clouds in the same coordinate system; after a new key frame is confirmed, taking the scene frame as a medium, calculating a preliminary transformation between point clouds based on singular value decomposition, and then performing fine registration by using a GICP algorithm; and performing point cloud fusion by using a TSDF technology, and selectively locking high-quality areas to improve efficiency. The method breaks through the bottleneck of existing methods in terms of reconstruction accuracy and the like by innovatively integrating 3R multi-frame reconstruction, GICP registration and TSDF technology, and realizes large-scale scene three-dimensional reconstruction with high robustness, high accuracy and real-time performance.

Description

Technical Field

[0001] This invention relates to a real-time 3D reconstruction method for large scenes based on 3R, belonging to the field of computer vision and 3D reconstruction technology. Background Technology

[0002] Real-time 3D reconstruction of large scenes has significant application value in fields such as smart city construction, digital preservation of cultural relics, autonomous driving, robot navigation, and virtual and augmented reality. RGB camera-based 3D reconstruction methods, due to their low hardware cost and lightweight equipment, can be widely applied in scenarios such as drone aerial photography, handheld devices, and mobile robots. These methods typically utilize multi-view geometric techniques, such as Structure from Motion (SfM) and Multi-View Stereo (MVS), to estimate camera pose and reconstruct the 3D structure of the scene from images taken from multiple angles.

[0003] However, traditional RGB camera-based methods have the following limitations: they rely on dense viewpoint acquisition and precise camera calibration, increasing the complexity of data acquisition; reconstruction accuracy significantly decreases in complex environments with missing textures, changing lighting, or dynamic objects; and processing efficiency is low, making it difficult to meet the needs of real-time applications. In recent years, although 3D reconstruction methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have made significant progress in reconstruction quality and novel viewpoint synthesis, these methods typically require lengthy optimization processes, have high computational complexity, and are difficult to meet real-time requirements.

[0004] To achieve real-time 3D reconstruction of large scenes, researchers have proposed solutions based on Simultaneous Localization and Mapping (SLAM). These methods first utilize SLAM technology for real-time camera localization, then combine an efficient MVS network or a stereo matching network to estimate pixel depth, ultimately recovering the point cloud's position in the global coordinate system. To improve processing efficiency, techniques such as voxels and truncated signed distance fields (TSDF) have been introduced for real-time point cloud fusion and updating. However, these methods still have significant limitations in terms of reconstruction accuracy, point cloud quality, and system complexity. They are not only constrained by the performance bottleneck of the depth estimation network but also prone to problems such as point cloud misalignment and multi-layer reconstruction. Furthermore, they require multiple modules to work together, increasing system complexity.

[0005] To simplify the processing flow, researchers have explored end-to-end real-time 3D reconstruction methods. Techniques such as DUST3R, Monst3R, SLAM3R, and VGGT, trained on large-scale datasets, integrate tasks such as point cloud reconstruction and feature matching into a unified deep learning framework. The core of these methods is the use of a feedforward neural network architecture, employing structures such as Multi-Layer Perceptrons (MLPs) or Transformers to directly learn the mapping relationship of 3D geometric representations from the input image sequence. The network typically includes feature extraction layers, cross-view attention mechanism layers, and geometric decoding layers, enabling multiple tasks such as feature extraction, matching, and depth estimation to be completed simultaneously in a single forward propagation, avoiding the complex multi-stage processing flow of traditional methods. While this end-to-end feedforward network design has a defined computational path and latency during inference, its massive network size (typically containing hundreds of millions of parameters) leads to high computational resource consumption, demanding hardware requirements, and limitations, making it difficult to meet the real-time processing needs of large scenes while ensuring reconstruction quality.

[0006] In summary, existing technologies face multiple challenges in terms of accuracy, efficiency, and scalability in large-scale real-time 3D reconstruction tasks. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time 3D reconstruction method for large scenes based on 3R, aiming to solve the problem that it is difficult to balance accuracy and efficiency in the existing technology of real-time 3D reconstruction of large scenes.

[0008] A real-time 3D reconstruction method for large scenes based on 3R includes the following steps: (1) For a large target scene, image data from a single RGB camera is acquired in real time, and the image data is preprocessed by distortion correction and uniform cropping. (2) Construct a multi-level reconstruction architecture of "normal frame - key frame - scene frame": each frame image is initially a normal frame, the first frame is automatically set as the first key frame and the first scene frame, and the coordinate system of the first frame is used as the world coordinate system; (3) Starting from the second frame, determine whether there are scene frames with a similarity higher than the preset similarity threshold with the current ordinary frame: (3.1) If it exists, the scene frame with the highest similarity is selected and the current ordinary frame is used for 3D reconstruction. When the scene frame with the highest similarity is not the same frame as the nearest key frame, the scene frame with the highest similarity, the nearest key frame and the current ordinary frame are selected for 3D reconstruction. The point cloud of the current ordinary frame, the scene frame with the highest similarity and the nearest key frame in the coordinate system of the scene frame is generated. At the same time, the coordinate system of the scene frame is defined as the local coordinate system of the current reconstruction. Then determine whether the point cloud overlap between the current ordinary frame and the nearest key frame is lower than the preset threshold. If not, discard the current ordinary frame and do not perform any subsequent operations. If yes, promote the current ordinary frame to a key frame and perform the point cloud alignment operations in steps (4)-(6). (3.2) If it does not exist, the most recent keyframe is first promoted to a scene frame, and then the 3D reconstruction and subsequent operations described in step (3.1) are performed; (4) When the current ordinary frame is promoted to a key frame, since the scene frame coordinate system used for reconstruction is already in the world coordinate system, based on the singular value decomposition (SVD) method, using the scene frame as the medium, the preliminary transformation matrix from the local coordinate system of the current reconstruction to the local coordinate system of the scene frame in the world coordinate system is calculated. (5) The generalized iterative nearest neighbor algorithm (GICP) is used to optimize the preliminary transformation matrix to achieve fine alignment of the point cloud; after aligning the coordinate system, the point cloud of the nearest key frame is stored in the point cloud set of the world coordinate system; (6) Output the point cloud in the world coordinate system in real time after it is updated; (7) Repeat steps (3)-(6) until the large scene real-time 3D reconstruction is completed.

[0009] The aforementioned 3R-based real-time 3D reconstruction method for large scenes also includes fusing the finely aligned point cloud based on the Truncated Signed Distance Field (TSDF) technique, then locking the regions with alignment and fit higher than a preset threshold, and keeping the regions with fit lower than the preset threshold in an open update state.

[0010] The determination of whether a scene frame meets the similarity threshold includes: based on the point cloud overlap O between the current ordinary frame and the nearest keyframe, using a dynamic similarity threshold calculation function to obtain the dynamic similarity threshold of the current ordinary frame. h (O), the calculation function is as follows: Then, the 3R technique is used to calculate the image similarity score between the current normal frame and all scene frames. When a similarity score greater than 1 is found... h When the scene frame with the highest score is (O), the scene frame with the highest score is selected for subsequent 3D reconstruction; when all image similarity scores are less than (O), the scene frame with the highest score is selected for subsequent 3D reconstruction. h When (O), it means that there is no scene frame that meets the similarity threshold with the current ordinary frame, and it is impossible to perform multi-frame 3D reconstruction based on 3R technology with the existing scene frame. Therefore, the key frame that is closest in time is selected as the new scene frame to participate in the subsequent 3D reconstruction.

[0011] The 3R technology is a deep learning-based 3D reconstruction framework that can extract effective information from monocular RGB image sequences and reconstruct the point clouds corresponding to each frame in a unified coordinate system. Starting from the second frame, the following frames are selected for 3D reconstruction: the current ordinary frame, the two frames before the current ordinary frame, the keyframe closest to the current ordinary frame, the scene frame determined in the previous step, the three ordinary frames before the scene frame, and the three ordinary frames after the scene frame. The minimum number of frames required to perform 3D reconstruction is the current ordinary frame and its scene frame. After 3D reconstruction, the selected frames yield point clouds located in the same coordinate system.

[0012] The calculation of preliminary transformations between point clouds based on the Singular Value Decomposition (SVD) method includes: Based on a shared scene frame reference, a point-to-point correspondence can be established between the stored point clouds in the current local coordinate system and the world coordinate system through the pixel-to-point cloud mapping relationship, resulting in two sets of point clouds that correspond one-to-one. ; Calculate the centroid of the two point clouds. And construct a decentroided point cloud. ; Calculate the covariance matrix And perform singular value decomposition: ; Calculate the rotation matrix Translation vector .

[0013] The aforementioned use of the Generalized Iterative Nearest Neighbor (GICP) algorithm to achieve fine-grained alignment of point clouds includes: For source cloud and target point cloud Define the objective function: ,in, It is the distance vector between point pairs. It is the local covariance matrix of the points in the source point cloud. The corresponding point in the target point cloud The local covariance matrix, The covariance matrix of the target point cloud after transformation is: ; Calculate the local covariance matrix for each point, where the covariance matrix is ​​calculated based on the set of the point's k nearest neighbors, with a given point as an example. For example: ,in It is a point The set of k nearest neighbors, It is the j-th point in the nearest neighbor set. It is the mean of the nearest neighbors; The optimal rotation matrix and translation vector are obtained iteratively through nonlinear optimization: .

[0014] The process of fusing the finely aligned point cloud based on truncated symbolic distance field technology, followed by locking regions with alignment scores higher than a preset threshold and keeping regions with alignment scores lower than a preset threshold open for updating (point cloud fusion, locking, and updating) includes: The truncated symbolic distance field technique maintains a symbolic distance value D(x) and a cumulative confidence weight W(x) for each voxel x in space. D(x) represents the directed distance from the voxel center to the nearest object surface. Positive values ​​indicate that the object is in front of or outside the surface, negative values ​​indicate that the object is behind or inside the surface, and zero values ​​are located on the surface. W(x) characterizes the reliability of the distance value. Initial definition: When voxel x is observed for the first time (i.e., n=1), its TSDF value and cumulative weight are initialized as follows: = , ; Here, d1(x) is the symbolic distance value obtained from the observation point cloud of the current frame. Specifically, by finding the nearest neighbor of the voxel x-center in the 3D point cloud of the current frame, or by fitting a local tangent plane using its neighboring points, the absolute value of d1(x) is the vertical distance from the voxel center to the nearest neighbor or the local tangent plane. Its sign is determined by the orientation of the voxel relative to the nearest neighbor or the local tangent plane (positive on the sensor side, negative otherwise). This value records the precise geometric relationship of the voxel relative to the observation surface at the time of the first observation. w1(x) is the confidence weight of the first observation, which is related to the noise model and the observation angle.

[0015] For voxels in space In position The update formulas for TSDF values ​​and weights are as follows: , , in, This is the merged TSDF value. It is a cumulative weight. This is the newly observed symbol distance value. This represents the weight of the new observation, and n+1 is the number of times the voxel has been observed. Define point cloud alignment and fit metrics ,in, It is the variance of n observations at that voxel. It is a normalized parameter. It is the cumulative weight. It is the upper limit of weight. Set the locking state according to the fit: , For locked voxels, the TSDF value remains unchanged; for unlocked voxels, the TSDF update continues. The final TSDF update rule is as follows: .

[0016] The image data preprocessing includes: Use the cv2.getOptimalNewCameraMatrix() function to obtain the camera intrinsic parameter matrix; The cv2.undistort() function from the OpenCV library is used to perform radial and tangential distortion correction based on the camera intrinsic matrix and distortion coefficients. Uniform region clipping can be achieved using the cv2.remap() function or array slicing operations.

[0017] The method further includes: It provides a real-time visualization module to dynamically display the current camera pose, the reconstructed 3D map, and the fusion results; Output the 3D reconstruction results in point cloud format.

[0018] Based on the above method, the present invention also provides a large-scene real-time 3D reconstruction device based on 3R, the device comprising: The data acquisition module is used to acquire image data from the RGB monocular camera and perform preprocessing. The architecture building module is used to construct a multi-layered reconstruction architecture of "normal frames - key frames - scene frames"; The 3D reconstruction module is used to generate point clouds based on the 3R multi-frame 3D reconstruction method. The point cloud registration module is used to perform point cloud alignment based on singular value decomposition and GICP. The point cloud fusion module is used for point cloud fusion and selective locking based on TSDF technology. The results output module is used to output real-time 3D reconstruction results for large scenes.

[0019] Beneficial effects This invention innovatively integrates 3R-based multi-frame 3D reconstruction, singular value decomposition to solve point cloud rigid body transformation, generalized iterative nearest point (GICP) point cloud registration, and truncated symbolic distance field (TSDF) technology to construct a multi-level progressive reconstruction architecture of "ordinary frame-key frame-scene frame". This breakthrough overcomes the technical bottlenecks of existing methods in terms of computational efficiency, reconstruction accuracy, and scene scale, and achieves large-scale scene 3D reconstruction with high robustness, high accuracy, and real-time performance. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the real-time 3D reconstruction method for large scenes based on 3R of the present invention. Figure 2 This is a schematic diagram of the multi-frame 3D reconstruction results based on 3R according to the present invention; Figure 3 This is a schematic diagram of the multi-level reconstruction architecture of "ordinary frame-key frame-scene frame" of the present invention; Figure 4 This is a schematic diagram of the 3D point cloud alignment result based on GICP of the present invention; Figure 5 This is a schematic diagram of the real-time 3D reconstruction result of a large scene according to the present invention. Detailed Implementation

[0021] This invention provides a method for real-time 3D reconstruction of large scenes based on 3R. Figure 1 The flowchart of the large-scene real-time 3D reconstruction method based on 3R provided by the present invention is shown below. Figure 1 As shown, the method includes steps S110 to S170.

[0022] S110. Acquire image data from an RGB monocular camera and perform distortion correction and uniform cropping preprocessing on the image data.

[0023] In this embodiment, a stable connection between the RGB monocular camera and the processing device is first established. The camera can be an industrial or consumer-grade camera from brands such as Basler, PointGrey, Intel RealSense, or ZED, and can be connected via interfaces such as USB 3.0, Gigabit Ethernet, Camera Link, or MIPI CSI. Among these, the USB 3.0 interface is widely used due to its plug-and-play feature and 5Gbps transmission bandwidth. Camera parameter settings include: a frame rate of 30fps is preferred, which can be reduced to 15fps when computing resources are limited; a resolution of 1920×1080 is preferred, while 1280×720 can be selected for embedded devices; exposure control uses automatic exposure mode in most scenarios, aiming to maintain the average image brightness at a medium gray level (approximately 128 / 255), while a fixed exposure mode can be used in industrial inspection scenarios with stable lighting, with the exposure time set between 1 / 30 second and 1 / 1000 second; white balance uses automatic mode in scenarios with varying light sources, while a corresponding preset mode can be selected in specific light source environments (daylight 5500K, fluorescent lamp 4000K, incandescent lamp 3200K); gain control uses automatic gain control (AGC), with the gain limit set below 20dB, and low gain used when lighting is sufficient.

[0024] Camera calibration employs the Zhang Zhengyou calibration method to obtain internal camera parameters and distortion coefficients. The calibration process includes: preparing a 9×6 or 11×8 checkerboard calibration board with a grid size of 20-30 mm, printed on a flat, rigid material such as acrylic or aluminum composite panel; taking 20-30 images of the calibration board from different angles and distances, including images facing the board directly, at different tilt angles (±30 degrees, ±45 degrees), at different distances (0.5 meters to 2 meters), and at different positions (center, corners, edges); and using Harris or Shi-Tom... The ASI corner detector finds the checkerboard corners and improves the detection accuracy to the 0.1 pixel level through a sub-pixel optimization algorithm, with a corner detection success rate of over 95%. The camera intrinsic parameter matrix K (including focal lengths fx and fy and principal point coordinates cx and cy) is optimized by minimizing the reprojection error. Radial distortion coefficients k1, k2, and k3 (k1 is usually between -0.5 and 0.5) and tangential distortion coefficients p1 and p2 (usually between -0.01 and 0.01) are calculated.

[0025] Distortion correction adjusts distorted images to resemble those from an ideal pinhole camera model. The specific steps are as follows: First, calculate the optimal intrinsic parameter matrix of the new camera based on the calibration parameters, maximizing the preservation of the effective field of view while removing distortion. Second, establish a pixel mapping relationship between the distorted and corrected images, pre-calculating and storing this mapping relationship as a lookup table to accelerate subsequent processing. Third, for each pixel location, find the corresponding original image coordinates using the lookup table, calculate the pixel value at that location using bilinear or bicubic interpolation, complete image resampling, and generate the final corrected image.

[0026] Uniform cropping ensures that all processed images have the same effective field of view. Specific cropping methods are as follows: When using center cropping, the rectangular area at the center of the image is retained, while edges with significant distortion are removed; when using effective region cropping, the effective pixel area after distortion correction is first identified, and then the largest inscribed rectangle within that area is cropped; when using fixed-ratio cropping, the image is cropped according to standard display ratios such as 16:9 or 4:3 to ensure the output image conforms to standard display formats. The three cropping methods can be selected based on specific application requirements.

[0027] S120. Construct a multi-layered reconstruction architecture of "normal frames - key frames - scene frames".

[0028] This invention innovatively proposes a three-layer frame management architecture, such as... Figure 3As shown, this layered design fully considers the various needs in real-time 3D reconstruction of large scenes: real-time requirements necessitate reducing the number of frames processed, accuracy requirements demand sufficient viewpoint coverage, and robustness requirements require handling rapid motion and sudden viewpoint changes. By dividing image frames into three layers, each undertaking a different function, an optimal balance between computational efficiency and reconstruction quality is achieved.

[0029] Ordinary frames are the basic input to the system; each frame of image captured by the camera is first entered into the system as an ordinary frame. The main function of ordinary frames is to provide continuous visual observation for real-time camera tracking and pose estimation. Processing ordinary frames is relatively lightweight; it involves selecting scene frames that meet similarity criteria, such as... Figure 2 As shown, 3R-based 3D reconstruction is performed. This design enables the system to perform high frame rate tracking, ensuring the system's real-time response capability.

[0030] Keyframes are representative frames selected from ordinary frames for building and maintaining 3D maps. Keyframe selection requires balancing two conflicting needs: too many selections increase computational burden and memory consumption, while too few selections lead to map sparsity and tracking failures. This invention selects keyframes based on the overlap of reconstructed point clouds between frames.

[0031] The overlap criterion calculates the point cloud overlap between the current frame and the previous keyframe. When the overlap is less than 40%, it indicates that a large amount of new scene content has been observed in the current frame, and it should be selected as a keyframe.

[0032] Scene frames are a subset of keyframes and represent the core perspective of the scene. The design goal of scene frames is to express the main structure of the entire scene with the fewest possible frames, used for global optimization and loop closure detection. The selection of scene frames is more stringent, considering not only the quality of individual frames but also the complementarity between frames and global coverage.

[0033] The selection of scene frames employs a strategy combining a greedy algorithm and posterior optimization. In the initial stage, the first keyframe automatically becomes the first scene frame, serving as the seed for scene representation. Subsequent scene frame selection is based on the following principles: the coverage principle requires that new scene frames can observe areas not fully covered by the current scene frame set; the uniqueness principle requires that scene frames have significant visual features to facilitate relocalization and loop closure detection; and the distribution principle requires that scene frames be spatially uniformly distributed to avoid local clustering.

[0034] Similarity calculation between scene frames and ordinary frames is a key technology for scene frame selection. This invention utilizes deep features extracted using the 3R framework for dynamic similarity calculation. 3R features are learned global descriptors capable of capturing the semantic and geometric information of an image.

[0035] The dynamic threshold adjustment mechanism is an innovation of this invention. The similarity threshold is not fixed, but dynamically adjusted according to the characteristics of the current scene. Assuming that the current frame has the most similar viewpoint to the nearest keyframe, the threshold for finding scene frames in the current frame is calculated by fitting a function based on the similarity between the current frame and the nearest keyframe.

[0036] Frame lifecycle management ensures system memory efficiency. Once tracking is complete, the image data of ordinary frames is released promptly unless they are promoted to keyframes. Keyframes and scene frames retain complete image and feature information and are permanently stored in the system.

[0037] S130. Three-dimensional reconstruction of multi-frame images based on the 3R multi-frame three-dimensional reconstruction method.

[0038] 3R (Reconstruction, Registration, and Refinement) is an end-to-end multi-view approach proposed in recent years. Figure Three 3D reconstruction method. Unlike traditional feature matching-based methods, 3R learns 3D geometry directly from images through deep learning, enabling it to handle scenarios with weak textures and repetitive textures that are difficult for traditional methods to process.

[0039] The architecture of the 3R network fully considers the characteristics of multi-view geometry. The network input consists of multiple RGB images, and the output is the depth map and camera pose for each image.

[0040] The multi-frame selection strategy employs an 11-frame input design. The specific selection method is as follows: the current ordinary frame is selected as the latest observation; the three frames preceding the ordinary frame are selected to provide short-term temporal information; based on the 3R technique, the scene frame most similar to the current frame is selected to provide long-term global constraints; and the three frames before and after the scene frame are selected to enhance the contextual information of the scene frame. This selection method ensures that up to 11 frames of input contain both local fine-grained information and global consistency constraints. The fixed 10-frame design strikes a balance between GPU memory consumption and reconstruction quality, making network inference time predictable and meeting the requirements of real-time systems.

[0041] The conversion process from depth map to point cloud is as follows: First, the camera intrinsic parameter inverse matrix is ​​used to convert each pixel coordinate (u,v) into normalized camera coordinates, and then multiplied by the corresponding depth value to obtain a 3D point in the camera coordinate system. Point cloud post-processing includes: removing invalid points with depth values ​​of 0 or exceeding a reasonable range; applying statistical filtering to remove outliers that deviate excessively from the neighborhood average based on neighborhood statistical characteristics; using voxel filtering for downsampling, retaining a representative point within each voxel; and estimating the local normal vector for each point. During multi-frame point cloud fusion, the confidence score output by the 3R network is used for weighted fusion. The confidence score comprehensively considers network prediction uncertainty and multi-view... Figure OneConsistency, depth continuity, and image texture richness are considered; points with high confidence receive greater weight during fusion.

[0042] S140. Calculate the preliminary transformation between point clouds based on singular value decomposition.

[0043] Once a new ordinary frame is identified as a keyframe, the system needs to align the local point cloud of that keyframe with the global map. This alignment process consists of two steps: first, a preliminary transformation is calculated using Singular Value Decomposition (SVD), and then fine-tuning is performed using Global Image Processing (GICP). SVD provides an analytical, globally optimal solution, while GICP further optimizes local accuracy.

[0044] Scene frames play a crucial bridging role in the alignment process. Since a scene frame participates in both multi-frame reconstructions, it has a corresponding point cloud in both reconstruction results. More importantly, there is a definite correspondence between each pixel in the scene frame and its corresponding 3D point cloud, and this correspondence remains consistent in both reconstructions. Utilizing this characteristic, a precise correspondence can be established between the two sets of point clouds.

[0045] Establishing corresponding points requires handling several technical details. First, it's necessary to ensure that scene frames generate valid depth values ​​in both reconstructions. Due to occlusion, viewpoint limitations, or reconstruction failures, some pixels may only have valid depth in one reconstruction. Second, the resolution difference of the depth maps needs to be considered; if different resolutions are used in the two reconstructions, appropriate interpolation or downsampling is required. Finally, the reasonableness of the correspondence needs to be verified by checking whether the distance between corresponding points is within a reasonable range.

[0046] Singular value decomposition (SVD) is a classic problem for solving rigid body transformations. Given two corresponding sets of 3D points... The goal is to find the optimal rotation matrix. Translation vector This makes the transformed point set and The distance is minimized. This problem has a closed-form solution and can be solved efficiently using SVD.

[0047] The mathematical principle behind the SVD solution process is based on least squares optimization. First, the centroids of the two point sets are calculated; these are the geometric centers of the point sets. Then, all points are translated into a coordinate system with the centroids at the origin. This process is called centroid removal. The purpose of centroid removal is to decouple rotation and translation, allowing the rotations to be solved first, followed by the translations.

[0048] Constructing the covariance matrix is ​​a core step in SVD. Covariance matrix It is the sum of the outer products of all corresponding point pairs, encoding the correlation between the two sets of points. This is obtained by performing singular value decomposition on H. , among which and It is an orthogonal matrix. It is a diagonal matrix.

[0049] Calculating the translation vector is relatively simple. After obtaining the rotation matrix, the optimal translation vector is... ,in and These are the centroids of the source and target point sets, respectively. The physical meaning of this formula is: first rotate the source point set, then translate it so that the centroid of the rotated set coincides with the centroid of the target point set.

[0050] S150, Refined point cloud alignment based on GICP algorithm.

[0051] After obtaining the initial transformation between the two point clouds, point cloud alignment based on GICP is performed, and a nonlinear optimization algorithm is used to refine the point cloud alignment. GICP (Generalized Iterative Closest Point) improves registration accuracy by considering the local geometry of the point cloud and is a generalization of the ICP algorithm.

[0052] The core idea of ​​GICP is to assign a covariance matrix to each point, describing its local uncertainty. Traditional ICP algorithms assume all points have equal importance, while GICP encodes the local geometric features of points through the covariance matrix. This results in points in a planar region having greater uncertainty in the direction perpendicular to the plane and less uncertainty in the direction parallel to the plane. This representation better reflects actual geometric constraints.

[0053] The calculation of the local covariance matrix is ​​based on the k nearest neighbors of a point. For each point, its k nearest neighbors are first found (usually k=20), and then the covariance matrix of these neighborhood points is calculated. The eigenvalues ​​and eigenvectors of the covariance matrix reflect the local geometry: the eigenvector corresponding to the smallest eigenvalue points in the direction of the normal vector, and the larger eigenvalues ​​correspond to the direction of the tangent plane.

[0054] The objective function of GICP is to minimize the Mahalanobis distance between pairs of points. For the source point cloud... and target point cloud The objective function is defined as the sum of weighted distances between all corresponding pairs of points, where the weight matrix is ​​the inverse of the sum of the covariance matrices of the two points. This form naturally handles uncertainties in different directions: small deviations in the normal vector direction incur a large cost, while deviations in the tangent plane direction have a smaller cost.

[0055] GICP employs an iterative optimization strategy. Each iteration consists of two main steps: correspondence search and transformation update. Correspondence search uses KD-trees or other spatial index structures to accelerate nearest neighbor queries. Transformation update obtains the transformation increment by linearizing the objective function, constructing a normal equation, and solving it.

[0056] To improve convergence speed and avoid local optima, this invention employs a multi-resolution strategy. First, registration is performed at a coarse resolution (e.g., voxel size of 0.2 meters) to obtain better initial values. Then, the resolution is gradually refined to a fine resolution (e.g., 0.05 meters). This coarse-to-fine strategy not only improves convergence speed but also enhances the algorithm's robustness.

[0057] The use of robust kernel functions further enhances the algorithm's resistance to outliers. Commonly used robust kernel functions include the Huber kernel, Cauchy kernel, and Tukey kernel. These kernel functions maintain quadratic growth for small errors, but grow more slowly for larger errors, thus reducing the impact of outlier points on the optimization results.

[0058] Convergence is determined based on several criteria. The primary convergence criterion is that the norm of the transformation increment is less than a threshold (e.g., translation less than 1 mm, rotation less than 0.1 degrees). Auxiliary criteria include that the relative change of the objective function is less than a threshold, or that the maximum number of iterations has been reached. In practice, GICP typically converges within 10-30 iterations.

[0059] like Figure 4 As shown, point cloud alignment based on GICP can achieve accurate point cloud registration, with registration errors typically at the centimeter level, laying the foundation for subsequent point cloud fusion.

[0060] S160, point cloud fusion and selective locking based on TSDF technology.

[0061] After performing refined point cloud alignment, the point cloud is fused and updated using the Truncated Signed Distance Function (TSDF) technique. TSDF is an implicit surface representation method that represents geometry by storing signed distances to the nearest surface in 3D space.

[0062] The basic principle of TSDF is to divide 3D space into a regular voxel mesh. Each voxel stores two values: a signed distance value to the nearest surface and a cumulative weight. The sign of the distance value indicates whether the point is inside the object (negative value) or outside (positive value), with zero indicating that it is exactly on the surface. Truncation means that only precise distance values ​​are stored within a certain range near the surface, and values ​​outside this range are truncated to fixed values.

[0063] In this invention, the voxel size is dynamically determined based on the reconstruction depth: for near-range reconstruction, a smaller voxel size is used to ensure fine reconstruction results; for mid-range reconstruction, a medium voxel size is used to balance accuracy and efficiency; and for long-range reconstruction, a larger voxel size is used to save memory and computing resources. The cutoff distance is uniformly set to three times the voxel size. This strategy of adaptively adjusting the voxel size based on the reconstruction depth ensures appropriate reconstruction quality across different distance ranges while optimizing system resource utilization.

[0064] The core of TSDF fusion is weighted average updates. When a new depth map arrives, it is first converted into a TSDF representation, and then weighted and averaged with an existing TSDF. The weights are designed considering several factors: observation angle (vertical observations have higher weights), depth value (near-range observations have higher weights), and confidence level (the confidence level of the 3R network output).

[0065] Selective locking is the innovative mechanism of this invention. Unlike traditional TSDF fusion, which treats all voxels uniformly, this method introduces a voxel locking mechanism: for regions with high reconstruction quality and consistent observations over multiple periods, the corresponding voxels are locked, and subsequent updates are stopped, thereby protecting the completed high-quality reconstruction regions from interference by noisy observations.

[0066] Locking is determined based on a point cloud alignment fit index. This index considers two factors: observation consistency and cumulative weight. Observation consistency is assessed by calculating the variance of multiple observations at a given voxel; a smaller variance indicates more consistent observations. The cumulative weight reflects the number of times the voxel has been observed; a larger weight indicates more thorough observations. When the fit exceeds a set threshold, it indicates that a stable and reliable reconstruction result has been obtained for that voxel, and it is locked, no longer accepting new observation updates.

[0067] Define point cloud alignment and fit metrics ,in, It is the variance of n observations at that voxel. It is a normalized parameter. It is the cumulative weight. It is the upper limit of the weight.

[0068] Set the locking state according to the fit: .

[0069] Implementing the locking mechanism requires maintaining an additional Boolean array to mark the locking status of each voxel. During TSDF updates, the locking flag is checked first, and updates are skipped for voxels that are already locked. This selective updating not only improves reconstruction quality but also reduces computational cost.

[0070] Memory management employs a sparse storage and block-based management strategy. Specifically, a spatial hash table is used to store TSDF data, allocating memory only to voxels containing object surfaces; empty voxels do not occupy storage space. The entire voxel mesh is divided into fixed-size blocks (e.g., 16×16×16 voxels), with each block managed as an independent memory unit. A dynamic memory allocation mechanism is implemented: when a new surface region is observed, a corresponding block is allocated; when a region has not been accessed for a long time, it is unloaded to external storage and reloaded when needed. This method significantly reduces memory consumption, making large-scene reconstruction possible.

[0071] S170: Outputs real-time 3D reconstruction results for large scenes.

[0072] like Figure 4 As shown, the system's final output is a high-quality 3D reconstruction result, which can be provided to downstream applications in multiple formats. The main output formats include: point cloud format (PLY, PCD), mesh format (OBJ, STL), and voxel format (for further processing).

[0073] The 3D model export offers two formats: point cloud and mesh. Point cloud generation employs a ray casting algorithm: rays are emitted from the camera center at a specified viewpoint towards each pixel, TSDF values ​​are sampled step-by-step along the rays, and precise intersections of zero isosurfaces are located using linear interpolation. These intersections are then output as 3D points on the visible surface. Mesh generation utilizes the Marching Cubes algorithm: it iterates through all voxels, and based on the signed distance values ​​of the eight vertices of each voxel, a predefined lookup table is used to determine the triangular facet configuration within that voxel, generating a continuous triangular mesh. The generated mesh can be further post-processed with Laplacian smoothing, quadratic error metric simplification, and other operations to optimize mesh quality before being exported in a standard format for rendering or 3D printing.

[0074] Real-time visualization is a crucial function of the system. This invention implements an OpenGL-based real-time rendering module capable of displaying the current reconstruction results at a frame rate of 15fps. Visualization includes multiple modes: point cloud mode displays the original 3D points; mesh mode displays the triangular mesh; texture mode maps RGB information onto geometry; and normal vector mode uses color-coded surface normals for easy quality checks.

[0075] like Figure 5 As shown, the method of this invention can achieve real-time 3D reconstruction of large scenes, achieving high reconstruction accuracy while ensuring real-time performance (above 15fps). The reconstruction results have good integrity and consistency, and are suitable for various applications such as navigation, measurement, and visualization.

[0076] In one embodiment, the system also supports incremental map updates. When the camera revisits a mapped area, new observations are merged with the existing map, further improving the reconstruction quality. This incremental update mechanism allows the system to run for extended periods, continuously improving map quality.

[0077] The map's persistent storage employs a layered structure. The top layer stores global information, such as map boundaries, resolution, and other metadata. The middle layer stores block indexes, recording the location and status of each block. The bottom layer stores the actual TSDF data, which can be compressed to save space. This structure supports efficient partial loading and streaming.

[0078] The technical advantages of this invention are reflected in several aspects. In terms of real-time performance, through a multi-layered architecture and optimized algorithms, the system can achieve the frame rate requirements for real-time processing, meeting the needs of online applications. Regarding accuracy, high-precision 3D reconstruction results are achieved through GICP fine registration and TSDF fusion. In terms of scalability, continuous reconstruction of large-scale scenes is supported through scene frame mechanisms and memory optimization. Regarding robustness, the 3R deep learning method can effectively handle challenging scenes such as weak textures, lighting variations, and dynamic objects, ensuring stable performance of the system in complex environments.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for real-time 3D reconstruction of large scenes based on 3R, characterized in that, Includes the following steps: (1) For a large target scene, image data from a single RGB camera is acquired in real time, and the image data is preprocessed by distortion correction and uniform cropping. (2) Construct a multi-level reconstruction architecture of "normal frame - key frame - scene frame": each frame image is initially a normal frame, the first frame is automatically set as the first key frame and the first scene frame, and the coordinate system of the first frame is used as the world coordinate system; (3) Starting from the second frame, determine whether there are scene frames with a similarity higher than the preset similarity threshold with the current ordinary frame: (3.1) If it exists, the scene frame with the highest similarity is selected and the current ordinary frame is used for 3D reconstruction. When the scene frame with the highest similarity is not the same frame as the nearest key frame, the scene frame with the highest similarity, the nearest key frame and the current ordinary frame are selected for 3D reconstruction. The point cloud of the current ordinary frame, the scene frame with the highest similarity and the nearest key frame in the coordinate system of the scene frame is generated. At the same time, the coordinate system of the scene frame is defined as the local coordinate system of the current reconstruction. Then determine whether the point cloud overlap between the current ordinary frame and the nearest key frame is lower than the preset threshold. If not, discard the current ordinary frame and do not perform any subsequent operations. If yes, promote the current ordinary frame to a key frame and perform the point cloud alignment operations in steps (4)-(6). (3.2) If it does not exist, the most recent keyframe is first promoted to a scene frame, and then the 3D reconstruction and subsequent operations described in step (3.1) are performed; (4) When the current ordinary frame is promoted to a key frame, since the scene frame coordinate system used for reconstruction is already in the world coordinate system, based on the singular value decomposition method, using the scene frame as the medium, the preliminary transformation matrix from the local coordinate system of the current reconstruction to the local coordinate system of the scene frame in the world coordinate system is calculated. (5) The generalized iterative nearest neighbor algorithm is used to optimize the preliminary transformation matrix to achieve fine alignment of the point cloud; After aligning the coordinate systems, store the point cloud of the most recent keyframe into the point cloud set in the world coordinate system; (6) Output the point cloud in the world coordinate system in real time after it is updated; (7) Repeat steps (3)-(6) until the large scene real-time 3D reconstruction is completed.

2. The method as described in claim 1, characterized in that: It also includes fusing the finely aligned point cloud based on truncated symbolic distance field technology, and then locking the regions with alignment and fit with the existing point cloud higher than a preset threshold, while keeping the regions with fit and fit lower than the preset threshold open for updating.

3. The method as described in claim 1, characterized in that: Step (3) of determining whether there is a scene frame that meets the similarity threshold includes: based on the point cloud overlap O between the current ordinary frame and the nearest key frame, using the dynamic similarity threshold calculation function to obtain the dynamic similarity threshold of the current ordinary frame. h (O), the calculation function is as follows: Then, the 3R technique is used to calculate the image similarity score between the current normal frame and all scene frames. When a similarity score greater than 1 is found... h When the scene frame with the highest score is (O), the scene frame with the highest score is selected for subsequent 3D reconstruction; when all image similarity scores are less than (O), the scene frame with the highest score is selected for subsequent 3D reconstruction. h When (O), it means that there is no scene frame that meets the similarity threshold with the current ordinary frame, and it is impossible to perform multi-frame 3D reconstruction based on 3R technology with the existing scene frame. Therefore, the key frame that is closest in time is selected as the new scene frame to participate in the subsequent 3D reconstruction.

4. The method as described in claim 1, characterized in that: The calculation of preliminary transformations between point clouds based on the singular value decomposition method includes: Based on a shared scene frame reference, a point-to-point correspondence can be established between the stored point clouds in the current local coordinate system and the world coordinate system through the pixel-to-point cloud mapping relationship, resulting in two sets of point clouds that correspond one-to-one. ; Calculate the centroid of the two point clouds. And construct a decentroided point cloud. ; Calculate the covariance matrix And perform singular value decomposition: ; Calculate the rotation matrix Translation vector .

5. The method as described in claim 1, characterized in that: The method of using the generalized iterative nearest neighbor algorithm to achieve fine alignment of point clouds includes: For source cloud and target point cloud Define the objective function: ,in, It is the distance vector between point pairs. It is the source point cloud midpoint The local covariance matrix, The corresponding point in the target point cloud The local covariance matrix, The covariance matrix of the target point cloud after transformation is: ; Calculate the local covariance matrix for each point, where the covariance matrix is ​​calculated based on the set of the point's k nearest neighbors, with a given point as an example. For example: ,in It is a point The set of k nearest neighbors, It is the j-th point in the nearest neighbor set. It is the mean of the nearest neighbors; The optimal rotation matrix and translation vector are obtained iteratively through nonlinear optimization: .

6. The method as described in claim 2, characterized in that: The finely aligned point cloud is fused using truncated symbolic distance field technology. Regions with alignment scores higher than a preset threshold are then locked, while regions with alignment scores lower than the preset threshold remain open for updates. This includes: The truncated symbolic distance field technique maintains a symbolic distance value D(x) and a cumulative confidence weight W(x) for each voxel x in space. D(x) represents the directed distance from the voxel center to the nearest object surface. Positive values ​​indicate that the object is in front of or outside the surface, negative values ​​indicate that the object is behind or inside the surface, and zero values ​​are located on the surface. W(x) characterizes the reliability of the distance value. Initial definition: When voxel x is observed for the first time, i.e., n=1, its TSDF value and cumulative weight are initialized as follows: = , ; Wherein, d1(x) is the symbolic distance value obtained from the observation point cloud of the current frame. Specifically, by finding the nearest neighbor of the voxel x center in the 3D point cloud of the current frame, or by fitting a local tangent plane using its neighboring points, the absolute value of d1(x) is taken as the vertical distance from the voxel center to the nearest neighbor or the local tangent plane. Its sign is determined by the orientation of the voxel relative to the nearest neighbor or the local tangent plane. This value records the precise geometric relationship of the voxel relative to the observation surface at the time of the first observation. w1(x) is the confidence weight of the first observation, which is related to the noise model and the observation angle. For voxels in space In position The update formulas for TSDF values ​​and weights are as follows: in, This is the merged TSDF value. It is a cumulative weight. This is the newly observed symbol distance value. It is the weight of the new observation, and n+1 is the number of times the voxel has been observed; Define point cloud alignment and fit metrics ,in, It is the variance of n observations at that voxel. It is a normalized parameter. It is the cumulative weight. It is the upper limit of weight; Set the locking state according to the fit: ; For locked voxels, the TSDF value remains unchanged; for unlocked voxels, the TSDF update continues. The final TSDF update rule is as follows: 。 7. The method as described in claim 1, characterized in that: The image data preprocessing includes: Use the cv2.getOptimalNewCameraMatrix() function to obtain the camera intrinsic parameter matrix; The cv2.undistort() function from the OpenCV library is used to perform radial and tangential distortion correction based on the camera intrinsic matrix and distortion coefficients. Uniform region clipping can be achieved using the cv2.remap() function or array slicing operations.

Images