Indoor scene three-dimensional reconstruction method based on deep fusion and confidence modeling

Abstract

The invention discloses an indoor scene three-dimensional reconstruction method based on deep fusion and confidence modeling, and belongs to the technical field of computer vision. According to the method, composite data frames such as a color image, a depth map, an IMU (Inertial Measurement Unit) and a camera attitude are comprehensively utilized to carry out regional three-dimensional reconstruction on an indoor scene: firstly, the scene is divided into a smooth region (such as a wall, a ground, a ceiling, a glass plane, a mirror surface, a blackboard and other planes) and a complex curved surface region; aiming at the smooth area, adopting geometric prior guide plane fitting provided by a visual large model, and combining sensor attitude information to quickly reconstruct a regular plane model; for a complex curved surface area, a multi-frame point cloud fusion strategy is adopted to accumulate different view angle information, and a deep residual error refining network is utilized to recover curved surface details, so that a high-precision curved surface model is obtained. According to the method, the three-dimensional structure of the indoor scene can be efficiently reconstructed, the global framework of the smooth area is reserved, and the details of the surface of a complex object are depicted in detail.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a method for 3D reconstruction of indoor scenes based on depth fusion and confidence modeling. This method fuses composite data frames such as color images, depth maps, inertial measurement units (IMUs), and attitude data to achieve high-precision and high-efficiency 3D modeling of indoor environments. Background Technology

[0002] In recent years, 3D reconstruction of indoor scenes has become increasingly important in applications such as virtual / augmented reality (VR / AR), robot navigation, and intelligent interior design. Traditional methods typically employ multi-view geometric reconstruction or rely on data from a single sensor: for example, using only RGB cameras for structure from motion (SfM) and multi-view stereo (MVS) techniques, or directly generating point cloud models using RGB-D cameras (color + depth). However, these methods have some limitations in practical applications: Data noise and model over-density are problems. Directly accumulating depth data from each frame often generates highly dense point cloud models containing a large number of redundant points and noise. Without further refinement or simplification, such models are difficult to use directly for real-time rendering or analysis. Furthermore, camera pose errors and depth sensor noise cause the accumulated model to gradually deviate from the real structure, resulting in blurring or distortion. Moreover, current RGB-D cameras, due to limitations in their imaging principles, cannot accurately detect depth under certain conditions. For example, LiDAR-based RGB-D cameras can hardly accurately detect the depth of glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity), even though these materials are common in indoor scenes. Large visual models, such as VGGT, can predict these smooth surfaces well without difficulty. Therefore, it is necessary to combine depth / point cloud predictions from large models during the reconstruction process.

[0003] To address the limitations of planar modeling and reduce model complexity and noise, a typical strategy is to leverage the structured characteristics of indoor scenes (the Manhattan world hypothesis) to extract planar areas such as walls, ceilings, floors, glass surfaces, mirrors, and blackboards for simplified reconstruction. However, many existing planar-based reconstruction methods tend to focus only on large planar areas, ignoring or directly removing other objects with freeform shapes from the scene. The result is a simplified but overly idealized reconstruction model that lacks fidelity in capturing complex details such as furniture, and particularly struggles to reconstruct the 3D information of narrow, thin objects, making it unsuitable for applications requiring high geometric detail, such as indoor walkthroughs and games. Furthermore, due to the noise in depth data, extracting geometric priors such as curved surfaces while preserving their original shapes is not easy, and the process is complex and time-consuming.

[0004] Recent advances in deep learning-based reconstruction have led to the emergence of new methods that directly utilize large-scale visual models in 3D reconstruction. For example, point map regression methods such as DUSt3R, Fast3R, and VGGT use pre-trained large models like the Transformer to directly predict high-density 3D point clouds from single-frame images. These methods learn global geometric priors, avoiding traditional multi-view geometric computations and generating smoother, more continuous geometry. In particular, they are unaffected by depth map discontinuities, providing clear reconstructions of object edges, and are unaffected by specific object materials, accurately predicting depth / point clouds for glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity). However, pure deep learning methods often lack utilization of real-world sensor data, and directly integrating it into existing reconstruction pipelines presents challenges, such as addressing the misalignment between the predicted point cloud and the actual camera pose.

[0005] To address the need for multi-sensor fusion and improve reconstruction robustness, some works have begun to integrate information from other sensors, such as IMUs. For example, existing methods acquire keyframes and their poses in real time using visual-inertial odometry, then reconstruct dense point clouds offline and extract interior layout planes. These methods utilize IMUs to improve camera trajectory accuracy and extract building structures (roof, floor, walls) for interior layout modeling. However, such solutions often separate planar structure extraction from non-planar objects, or focus only on the room's frame layout, neglecting complex curved objects such as interior furniture. Furthermore, they lack mechanisms for assessing uncertainties during multi-source data fusion, making it impossible to dynamically adjust reconstruction strategies based on data reliability.

[0006] In summary, the aforementioned technologies cannot simultaneously meet the requirements of rapid reconstruction and faithful detail. They either simplify the model but lose details, or produce rich details but with excessive noise, or completely fail to reconstruct glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity), resulting in significant losses in reconstruction integrity and accuracy. Furthermore, the lack of modeling for the reliability of sensor data during the fusion process may lead to the inclusion of incorrect geometry in the final model. To address these issues, a novel method for 3D reconstruction of indoor scenes is urgently needed. This method should comprehensively utilize multi-source data such as color, depth, and IMU data, employ differentiated strategies for different types of regions, and ensure the reliability of the fusion process through confidence modeling. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for 3D reconstruction of indoor scenes based on deep fusion and confidence modeling, which can simultaneously meet the requirements of rapid reconstruction and detail fidelity, and reduce sensor data errors during the fusion process.

[0008] To achieve the above objectives, the present invention provides the following technical solution: A method for 3D reconstruction of indoor scenes based on deep fusion and confidence modeling includes the following steps: S1 Composite Data Frame Acquisition: Acquire a sequence of composite data frames for an indoor scene. Each frame includes a color image, depth map, inertial measurement unit (IMU) data, and camera attitude data, which are used to provide multimodal synchronization information. S2 Region Division: The composite data frame is processed, and the scene is divided into smooth regions and complex curved surface regions according to geometric features and image features. The smooth regions include the approximate planar structures of walls, floors, ceilings, glass planes, mirrors, and blackboards, while the complex curved surface regions include the non-planar structures of furniture and curved objects. S3 Smooth Region Geometric Modeling: For the smooth region, a geometric modeling method guided by large model visual features is used for 3D reconstruction; global geometric priors are extracted from color images using a pre-trained visual model, and combined with depth maps and IMU pose information to perform plane fitting and modeling on the smooth region, quickly reconstructing 3D models of planar structures such as walls, floors, glass planes, mirrors, and blackboards. S4 Complex Surface Region Fusion and Refinement: For the complex surface region, a point cloud fusion strategy combined with a deep residual refinement network is used for reconstruction; multiple frames of deep point clouds are registered and fused in the global coordinate system to generate an initial complex region point cloud model, and the model is refined by a deep learning network to compensate for surface details, resulting in a high-precision 3D model of the complex region. S5 Confidence Modeling and Optimization: A confidence modeling module is introduced during the reconstruction process to calculate a multi-dimensional confidence index by integrating historical observation records, physical simulation, and time continuity factors. The reconstruction process is dynamically adjusted based on the confidence index, including weighted fusion of high-confidence data, removal of low-confidence data, and optimization of regional classification and model parameters, outputting a complete 3D model of the indoor scene optimized by confidence.

[0009] The principle and effects of this scheme are as follows: In the S1 composite data frame acquisition, a sequence of composite data frames of the indoor scene is acquired. Each frame includes a synchronized color image, depth map, IMU data, and camera pose information. The color image provides texture and semantic information, the depth map provides geometric depth, and the IMU and pose are used to improve the accuracy of camera trajectory estimation, forming the basic data for reconstruction.

[0010] In S2 scene region segmentation, each frame of data is divided into two categories: smooth regions and complex curved surface regions. Smooth regions refer to roughly planar areas with low surface curvature, such as walls, floors, and ceilings. Complex curved surface regions refer to objects with obvious free-form surface features or non-rigid shapes, such as furniture surfaces, curved cylinders, and decorative items. Region segmentation can be achieved by combining depth map normal vector change analysis and image visual features.

[0011] In the rapid modeling of smooth regions in S3, a large-model visual feature-guided geometric modeling method is used for rapid 3D reconstruction, particularly for areas that are undetectable by sensors, such as glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity). Specifically, a pre-trained large-scale visual model is used to extract geometric priors for smooth regions in color images (e.g., predicting dense depth maps / point clouds or planar parameters of the region from the image using a VGGT model). This is combined with actual measurement data of the depth map (supplemented by predicted depth / point clouds if detection is missing) and attitude / orientation information (such as gravity direction) provided by the IMU for planar fitting and geometric modeling. This allows for the rapid acquisition of model equations and extents for planes such as walls and floors, achieving efficient modeling of smooth regions.

[0012] In the deep fusion and refinement of complex surface regions in S4, an advanced point cloud fusion strategy combined with a deep residual refinement network is used to reconstruct details for complex surface regions. First, the point clouds of complex regions from multiple depth maps are registered and fused in a global coordinate system. The fusion strategy can draw on the method described in the applicant's previous patent CN118887100A, using manifold embedding to perform hash index accumulation on the voxelized point clouds of each frame, obtaining a preliminary fused global point cloud model. Then, a deep residual refinement neural network is introduced to refine the preliminary fused model: the network takes the coarsely fused point cloud and corresponding color texture as input, predicts the geometric residual (the residual refers to the difference between the model estimate and the actual data) for compensation, refines surface details, and corrects noise errors. By combining the global consistency of point cloud fusion with the local detail learning of the deep network, a high-fidelity model of complex surface regions can be generated.

[0013] In S5 confidence modeling and optimization, a confidence modeling module is introduced to evaluate data and reconstruction results throughout the entire modeling process. This module integrates multi-dimensional information, including historical observation records, physical simulation factors (such as illumination consistency and material reflection characteristics), and temporal continuity factors, outputting multi-dimensional confidence indices for each reconstructed element (point cloud points, planar segments, or object regions). Multi-dimensional confidence includes, for example, geometric confidence (based on the degree of agreement between the reconstruction model and observational data), photometric confidence (based on the consistency between image brightness / color and the illumination model), and temporal confidence (based on the sustained stability across frame observations). These confidence values ​​are used to guide the dynamic optimization of surface fitting and region classification: increasing the data weight of high-confidence regions and removing or remodeling low-confidence regions, thereby further improving the accuracy and reliability of the reconstruction model.

[0014] The method proposed in this invention employs differentiated processing based on the characteristics of different regions, ensuring both reconstruction efficiency and detail. Partition processing efficiently balances efficiency and detail, using different modeling strategies according to regional differences. For planar regions, rapid fitting simplifies the model and saves computation, while complex regions undergo fine reconstruction to preserve details and avoid oversimplification, achieving a balance between structure and detail. Confidence assessment enhances model reliability; the confidence modeling module evaluates data credibility in real time, avoiding the inclusion of transient noise or outliers in the final model. Fusion and optimization guided by multi-dimensional confidence results in a final reconstruction that significantly outperforms traditional methods lacking confidence feedback in terms of detail integrity and noise suppression.

[0015] The method of this invention can be applied to real-time or offline indoor scanning systems, such as mobile devices (phones, tablets, or robots) equipped with RGB-D cameras and IMUs. Rapid smooth surface modeling and incremental point cloud fusion enable real-time reconstruction; while deep network refinement and confidence optimization ensure that the result quality meets application requirements.

[0016] In summary, this invention achieves the simultaneous fulfillment of the requirements for rapid reconstruction and detail fidelity, while reducing sensor data errors during the fusion process.

[0017] Furthermore, the composite data frame consists of synchronously acquired multimodal sensor data, wherein the color image and depth map are aligned pixel by pixel, the IMU data provides the camera's acceleration and angular velocity, and the attitude data is the camera's position and orientation in the world coordinate system; preferably, the attitude data is obtained by fusing color image features and IMU measurements through visual inertial odometry to improve the accuracy and robustness of camera trajectory estimation.

[0018] Furthermore, the point cloud normal vector and curvature are calculated using the depth map, and regions with consistent normal vectors and low curvature are marked as candidates for smooth regions. Combining the gravity direction provided by the IMU, candidate regions with normal vectors that are close to vertical or horizontal are identified as planes such as walls or the ground. At the same time, the location of the plane is identified using color image segmentation or detection algorithms, and the smooth region boundary is confirmed by combining it with depth information. Point clouds that are not classified into smooth regions are automatically classified into complex curved surface regions. The region division ensures that smooth regions and complex regions complement each other in space without omission or repetition.

[0019] Furthermore, the smooth region geometric modeling step includes: using a robust fitting algorithm to perform planar fitting on the smooth region point cloud to obtain planar parameters; introducing geometric priors provided by a pre-trained large-scale visual model, performing monocular 3D reconstruction prediction on the color image to obtain a dense depth map / point cloud or planar guess, especially for glass planes (high transmittance), mirror surfaces (high reflectance), and pure black planes (low reflectance) that cannot be accurately detected by RGB-D cameras; aligning the prior with the actual depth map at scale, and then performing alignment and fusion through point cloud-to-point cloud registration; missing regions are supplemented by predicted depth / point cloud to expand and smooth the fitting results, significantly improving completeness and accuracy; during the fusion process, different weights are assigned to the data from the sensor and the data from the prior model, especially in the RGB-D depth detection missing regions, the prediction is used as a complete supplement (with the largest weight), to ensure that the fitting results are consistent with actual measurements and have global consistency. The smooth region model obtained through the above steps is represented in the form of planar segments, with accurate normal vectors, boundary ranges, and high geometric accuracy.

[0020] Furthermore, the complex surface region fusion and refinement step includes: progressive fusion using multi-frame depth map point clouds, accumulating the complex region point clouds of each frame into a global 3D grid using strategies such as voxel hashing or voxel filtering to generate a preliminary fused point cloud model; after obtaining the initial fused model, performing surface reconstruction to generate a coarse triangular mesh, and performing local geometric correction on the coarse mesh through a depth residual refinement network to improve the surface detail accuracy. The depth residual refinement network takes the difference between the measured depth and the current mesh projection depth as input and outputs the residual used to update the mesh vertex position, thereby obtaining a refined complex surface region model.

[0021] Furthermore, the depth residual refinement network is a Vision Transformer neural network, whose structure includes an encoder and a decoder. The encoder extracts multi-scale features that differ between the current frame observation and the coarse model, while the decoder outputs a depth residual map with the same size as the image. The network is trained offline on multiple indoor datasets, and the training objective is to minimize the loss function that minimizes the difference between the prediction residual and the true model, including pixel depth error loss and normal vector smoothing loss. During the inference phase, the network processes the depth data of each frame or several frames, progressively refining and updating the mesh model of complex regions to reconstruct high-frequency details.

[0022] Furthermore, the confidence modeling module calculates multi-dimensional confidence for each data point, local region, or global model, including: geometric confidence, photometric confidence, and temporal confidence. Geometric confidence is based on the geometric error assessment between the reconstructed model and the observation data; the smaller the error, the higher the confidence. Photometric confidence is based on the degree of conformity between image observation and physical lighting model; if the brightness change of the same surface under multiple views conforms to the Lambertian model, the photometric confidence is high; conversely, if there is strong specular reflection or light variation, the confidence decreases. Temporal confidence is based on the consistency and continuity of cross-frame observations; static structures that are continuously observed in multiple frames have high confidence, while points that appear occasionally or whose positions change have low confidence. The confidence modeling module integrates the above factors to form a confidence vector for each reconstructed element, which is used to guide model optimization.

[0023] Furthermore, the confidence index is used to provide feedback for optimizing the 3D reconstruction process, including: averaging multi-frame data using confidence as a weight during point cloud fusion, assigning greater weight to high-confidence observations and accumulating low-confidence observations with smaller weights, thereby reducing the impact of outlier data on the model; removing outliers or abnormal regions based on confidence during plane fitting and surface refinement, such as discarding noise points with low geometric confidence and excluding dynamic object point clouds with low temporal confidence, to ensure that the output model only contains stable and reliable structures; and re-evaluating previous classification results using confidence during region classification, triggering re-classification or changing the modeling strategy for regions where the overall confidence deviates from expectations, thereby adaptively improving model accuracy and completeness.

[0024] Furthermore, the method is applicable to real-time 3D reconstruction systems, wherein composite data frames are acquired in real time by an RGB-D camera and IMU on a mobile device. The system includes a data acquisition module, a region segmentation module, a smooth region modeling module, a complex region modeling module, a confidence assessment module, and a model fusion module. The system is configured to execute the above steps in parallel or pipeline mode, so that the 3D model can be updated frame by frame and the reconstruction results can be displayed in real time during the process of the sensor moving and scanning the indoor environment. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the indoor scene 3D reconstruction method based on deep fusion and confidence modeling of the present invention. Figure 2 This is a schematic diagram of the functional architecture of the main modules of the system in the method of this invention; Figure 3 This is a flowchart of the algorithm pseudocode of the method of the present invention; Figure 4 This is a display of the results after 3D reconstruction of the actual scene 1; Figure 5 This is a display of the results after 2D reconstruction of the actual scene; Figure 6 This is a display of the results after 3D reconstruction of the actual scene; Figure 7 This is a display of the results after 4D reconstruction of the actual scene; Figure 8 This is a display of the results after 3D reconstruction in actual scene 5 using only VGGT; Figure 9 This is a display of the results after 3D reconstruction in actual scene 6 using only VGGT. Detailed Implementation

[0026] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the features of the following embodiments can be combined with each other to form new implementation schemes.

[0027] A method for 3D reconstruction of indoor scenes based on deep fusion and confidence modeling (e.g.) Figure 1 (As shown), including the following steps: S1 Composite Data Frame Acquisition: Acquire a sequence of composite data frames for an indoor scene. Each frame includes a color image, depth map, inertial measurement unit (IMU) data, and camera attitude data, which are used to provide multimodal synchronization information. S2 Region Division: The composite data frame is processed, and the scene is divided into smooth regions and complex curved surface regions according to geometric features and image features. The smooth regions include roughly planar structures such as walls, floors, and ceilings, while the complex curved surface regions include non-planar structures such as furniture and curved objects. S3 Smooth Region Geometric Modeling: For the smooth region, a geometric modeling method guided by large model visual features is used for 3D reconstruction; global geometric priors are extracted from color images using a pre-trained visual model, and combined with depth maps and IMU pose information to perform plane fitting and modeling on the smooth region, quickly reconstructing 3D models of planar structures such as walls and floors. S4 Complex Surface Region Fusion and Refinement: For the complex surface region, a point cloud fusion strategy combined with a deep residual refinement network is used for reconstruction; multiple frames of deep point clouds are registered and fused in the global coordinate system to generate an initial complex region point cloud model, and the model is refined by a deep learning network to compensate for surface details, resulting in a high-precision 3D model of the complex region. S5 Confidence Modeling and Optimization: A confidence modeling module is introduced during the reconstruction process to calculate multi-dimensional confidence indices by integrating historical observation records, physical simulations, and time continuity. The reconstruction process is dynamically adjusted based on the confidence indices, including weighted fusion of high-confidence data, removal of low-confidence data, and optimization of regional classification and model parameters, outputting a complete 3D model of the indoor scene optimized by confidence.

[0028] For specific usage, please refer to the overall process. Figure 1 The method of this invention begins with data acquisition from sensors, then proceeds through region segmentation into two main branches: smooth region reconstruction and complex region reconstruction. These are processed in parallel, and the results are then fused to finally output a complete 3D model. Throughout the entire process, a confidence modeling module is used to evaluate and provide feedback on the results at each stage for optimization.

[0029] In S1 composite data frame acquisition, a device equipped with an RGB-D camera and IMU scans the indoor environment to acquire composite data frame sequences in real time. Composite data frames Includes: color images Depth map aligned with it The current frame's IMU readings (linear acceleration, angular velocity, etc.) and camera pose. (For example, pose estimated by visual inertial odometry or provided by a motion capture system). In this embodiment, the camera acquires approximately 30 frames of composite data per second, ensuring sufficient parallax between adjacent frames for 3D reconstruction. The color image and depth map are calibrated using intrinsic and extrinsic parameters to ensure a one-to-one correspondence between depth values ​​and image pixels, and the camera pose is then used for... The depth map is converted to a global coordinate system to form a point cloud. .

[0030] "Composite data frame" refers to an input frame that simultaneously contains multimodal synchronous data; "keypoint" can refer to feature points extracted from a color image or salient points in a spatial point cloud, used for registration and feature matching; "fit residual" refers to the deviation between the model's predicted values ​​and the actual observed values ​​during the model fitting process, such as the distance error from a point to the fitting plane. To ensure consistency of terminology, these terms will be interpreted as described above throughout the text.

[0031] When converting each frame's depth map into a point cloud, the camera intrinsic parameter matrix is ​​used. And depth values, pixel coordinates With depth Transformed into 3D points in camera coordinate system Then through the pose matrix Transform to the lower point of the global coordinate system For example, the following relationship can be used: , in For pixels depth value The above matrix relationship represents the pixel... and its depth value Through the inverse matrix of intrinsic parameters (included in) In the above formula ( Simplified representation as depth value To represent camera coordinates (normalized coordinates based on focal length and principal point offset multiplied by depth), the coordinates are transformed to the world coordinate system to obtain global point cloud coordinates. Through this process, a corresponding point cloud with pose is generated for each frame. This lays the foundation for future integration.

[0032] In the S2 scene region division, the acquired point cloud and color images The analysis categorizes the scene into two main types: smooth regions and complex curved surface regions. This categorization employs a combination of geometric analysis and visual feature analysis. First, the normal vector and curvature of each point are calculated on the point cloud geometry. For dense depth data, the direction variation can be calculated using principal component analysis in the sampling neighborhood of each point. Regions with low curvature and locally approximately consistent normal vectors are initially marked as smoothing candidate regions.

[0033] By combining the gravity direction provided by the IMU, and since planes such as walls and floors are usually vertical or horizontal, the gravity vector estimated by the IMU can be used to quickly identify vertical planes (orthogonal to the gravity direction, such as walls) and horizontal planes (parallel to the gravity direction, such as floors and ceilings). Planar segments that are close to these two types of directions are promoted as candidates for smooth regions.

[0034] Simultaneously, visual features of color images are utilized for auxiliary segmentation. For example, large deep learning-based models (such as indoor scene segmentation networks) are applied to detect the location of planar regions in the image, obtaining possible masks for wall and floor areas. Straight lines and vanishing points can also be detected to infer the main planar orientation, and combined with point cloud normal vector information to accurately pinpoint the planar location.

[0035] After the above steps, regions that consistently satisfy planar characteristics are marked as smooth regions (e.g., a continuous point cloud of a wall with similar normal vectors in an image). The remaining unclassified point cloud portions are treated as complex curved surface regions, including furniture, clutter, and any non-planar surfaces. For small protrusions embedded in the plane (such as decorations or stickers on the wall), their inclusion in the smooth region model (if the size is very small, their impact on the plane fitting can be ignored) or they can be segmented separately as complex regions based on a size threshold.

[0036] In this embodiment, after the initial scene division, typical smooth areas include: the four walls of the room, the floor, the ceiling, and the surfaces of large flat furniture (such as cabinet side panels, tabletops, and other flat areas). Complex curved areas include: the curved / curved parts of furniture (such as the curved armrests of a sofa, the curved back of a chair), cluttered objects (irregular shapes), and interior decorations (sculptures, plants, etc.).

[0037] In the rapid modeling of smooth regions, this invention employs a visually prior-guided geometric modeling method for the defined smooth regions to achieve rapid and accurate 3D reconstruction. Smooth region modeling includes the following steps: S3.1 Initial Plane Fitting: Based on the segmentation results, extract the point set of the smooth region from the point cloud of each frame. For example, for the identified wall region, collect all point data of that region in different frames. RANSAC (Random Sample Consensus) or least squares plane fitting methods are applied to calculate the plane model parameters. The plane equation satisfies The model should fit these points as closely as possible. Robust estimation is used to reduce the interference from outliers with large residuals, resulting in an initial planar model of the smooth region.

[0038] For planes observed only in a single frame (e.g., where camera motion hasn't yet covered the entire wall), the depth data for that region in the current frame can be used for fitting, but the reliability may be slightly lower. Subsequently, as more frames of data accumulate, the plane parameters can be iteratively optimized.

[0039] S3.2 Large Model Prior Fusion: For large and regular smooth regions, geometric priors provided by a pre-trained large visual model are introduced to improve the accuracy of planar modeling. In specific implementations, a model similar to VGGT (Visual GeometryGrounded Transformer) can be used, which can directly predict the dense depth map / point cloud of a scene from one or more color images. The color images... Input the VGGT model to obtain the predicted point cloud for the entire scene. The predicted point cloud includes all surfaces, including smooth regions, but due to its learning-based nature, its detail may not be as accurate as the depth map. Large-scale model depth / point cloud prediction, exemplified by VGGT, is characterized by smooth prediction combined with high-curvature transition edges. This feature makes the predicted depth / point cloud primarily based on smoothness assumptions, resulting in a smooth depth map / point cloud combined with high-curvature transition edges. This is a drawback for complex curved surfaces, as it loses a significant amount of fine surface detail. However, for the reconstruction of smooth regions, this feature can compensate for sensor noise interference. Simultaneously, the large model can effectively predict areas that RGB-D cameras cannot accurately detect, such as glass planes (high transmittance), mirror surfaces (high reflectivity), and pure black planes (low reflectivity). It provides globally consistent geometry and complete edge information, serving as a reference and supplement to the completeness of smooth region reconstruction.

[0040] First, the predicted depth and the camera's depth measurement are scale-aligned, meaning the model's predicted depth map is scale-aligned with the RGB-D camera's depth map. Alignment is relatively simple; multiple RGB predicted depth maps and actual depth maps are compared point-to-point (outliers or holes are removed) to calculate the average alignment coefficient. Then, the predicted point cloud is multiplied by the alignment coefficient, and the smooth regions are segmented and registered with the initially fitted plane. The VGGT-SLAM algorithm can be used for rigid alignment of the predicted and planar point clouds. After alignment, the predicted point cloud provides a wider range of information about the planar region or supplements the unmeasured portions. For example, if the actual depth map only captures a portion of the plane, but the prediction model "sees" a more complete planar outline based on the context, especially areas that the RGB-D camera cannot accurately detect—such as glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity)—the planar range can be expanded accordingly. Then, the plane parameters are updated by combining the distribution of the predicted point cloud, for example, by weighted fitting: points from the sensor depth are given higher weights, and points from the predicted prior are given the second highest weights, and the new plane equations are solved jointly.

[0041] S3.3 Planar Model Optimization and Output: After the above fusion, the resulting smooth region planar model is more accurate and complete in terms of parameters (normal vector, position) and range. At this point, the fitting residual for each participating point can be calculated; the residual is defined as the distance from the point to the fitting plane. For the plane equation... and any point The formula for calculating the residual is: , The quality of a planar model can be assessed based on the residual distribution. If most of the residuals are much smaller than the depth sensor noise level, the planar fit is considered reliable. If the residuals are large and exhibit systematic bias, it may indicate that the region is not strictly planar or that the depth data is distorted. In this case, the model needs to be re-divided or adjusted.

[0042] Finally, the smooth region modeling result is represented as several planar segments (polygons with normal vectors and boundary vertices, which can be obtained by finding the intersection of the plane with the room boundary or intersecting planes). These planar segments will be output as part of the final 3D model and combined with the complex region model to form a complete scene.

[0043] In the deep fusion and refinement of complex surface regions in S4, for point cloud regions marked as complex surfaces, this invention uses a multi-frame point cloud deep fusion combined with deep learning refinement to reconstruct the complex details. For example... Figure 1 As shown, its processing flow includes steps such as point cloud fusion, coarse model generation, and depth residual refinement: S4.1 Multi-Frame Point Cloud Fusion: Since complex regions often require observation from multiple perspectives to fully capture their shape, this step utilizes camera pose to accumulate point clouds of complex regions at different times into a global model, thereby improving point cloud density and coverage. In implementation, the manifold embedding-based point cloud fusion method proposed in the applicant's patent CN118887100A can be referenced. In short, each frame of the complex region point cloud undergoes voxelization preprocessing, and then the point clouds from each frame are inserted into the global voxel raster for accumulation using a hash index. For example: A global 3D raster data structure is established as an accumulation model, with the raster size selected based on depth noise and desired resolution (e.g., 5 mm). For each incoming frame of point cloud, its points are quantized and mapped to the nearest raster cell. If the raster cell has no data, a new node is created; if data already exists, the new point is merged with the existing points (e.g., recording the average or high-confidence value).

[0044] During the fusion process, the cumulative number of points, average coordinates, color, and other attributes of each grid can be recorded simultaneously. This is similar to voxel hashing, which uses hash tables to efficiently locate 3D spatial grids, enabling large-scale real-time fusion of point clouds from multiple frames. The fusion algorithm can be executed in parallel to adapt to high frame rate input.

[0045] After the above processing, a preliminary global point cloud model can be obtained. It integrates information from multiple perspectives. Compared to single-frame data, this fused point cloud reduces sensing blind spots (because different frames are complementary), lowers random noise (through averaging from multiple observations), and fills in density. For example, when scanning a complex piece of furniture, the point clouds collected from different angles are aggregated, and parts that were not visible from a single perspective, such as the back and internal contours, are gradually filled into the model.

[0046] S4.2 Deep Residual Refinement Network: For the initial coarse point cloud model This invention introduces a depth residual refinement network to further refine the depiction of complex surfaces. This network employs an encoder-decoder structure, using the depth differences between the coarse model projected onto each view as a supervisory signal to learn and compensate for reconstruction errors.

[0047] Specifically, design a Vision Transformer (ViT) model. The input is the depth map of the current frame. With the rough model in camera pose Composite depth map rendered below The difference between them (or the concatenation of the two as a multi-channel input). The synthesis depth is to... The depth, projected onto the current camera's viewpoint, should match the true depth if the model is perfect; discrepancies represent details or errors that the model has not yet captured. (Network) Learn from this difference image to output a residual depth map. Used for correction The geometry. That is, to obtain the optimized depth. .

[0048] Network prediction Robustness can be improved by fusing information from multiple frames: the differences between several consecutive frames can be stacked as input, or the previously fused global model can be used as a condition. For example, the network not only reacts to single-frame differences, but also refers to the model errors of historical frames to ensure that the output correction is continuous and smooth in time, without introducing instantaneous error deformations.

[0049] During the training phase, offline training can be performed using publicly available datasets (such as ScanNet, TUM RGB-D, etc.). During training, scenes with real 3D scans are used as the ground truth model. Partial point cloud perturbations or cropping are used to generate a "coarse model," allowing the network to learn to reconstruct the ground truth surface. The loss function can include depth residuals. The loss function includes a consistency loss for surface normals to encourage smooth and detailed reconstructed surfaces. The total loss function is as follows: , in It is the weighting coefficient of each loss component, depth residual loss. The goal is to make the predicted depth residual Approximate the true residual as closely as possible.

[0050] Normal vector consistency loss To ensure the correctness of the geometry in the surface normal direction, the surface normal vector is first estimated from the depth map. , loss function The cosine distance between the angles of the normal vectors:

[0051] in For the corresponding point, For point The normal vector at that location. In scenarios with multi-frame input and fusion updates, to avoid instability in cross-frame prediction, the temporal consistency loss is... , After processing by a deep residual network, the model is updated with details in each frame. These updates can be accumulated into the global model: for example, the residuals... The corresponding points are added to the fused point cloud. To refine the global point cloud, or directly to the mesh The vertices are offset-corrected based on the predicted residuals. This yields a fine-grained mesh model. Its surface details (such as the curves of object edges and the texture undulations of rough surfaces) are more realistic than the original rough model.

[0052] S4.4 Model Merging and Integration: After both the smooth and complex region models have been generated, they are merged and stitched together to form a complete scene. Since the boundaries of the two types of regions were reasonably defined during region partitioning, the merging process mainly handles the transition at the intersection. For example, if the mesh of a complex region intersects with a planar model at its edge, a truncation or stitching operation is used: the portion exceeding the plane is trimmed off, and the edge vertices are aligned with and welded to the planar boundary to eliminate gaps. On the other hand, for some details that fit on the plane (such as picture frames on a wall, whose outlines are modeled as complex regions, but the wall behind them is a smooth plane), the retention of the complex model can be determined based on the confidence level (if the picture frame point data is sufficient and the confidence level is high, the geometric protrusions should be retained; if the data is scarce, they may be ignored as noise, and the wall plane should remain flat).

[0053] After completing the above process, a 3D reconstruction model of the entire indoor scene is obtained: planes are represented by regular plane segments, complex objects are represented by high-detail meshes, and the overall coordinate system is aligned with the real scale.

[0054] In S5 confidence modeling and dynamic optimization, the key feature of this invention is the introduction of a confidence modeling module into the reconstruction process. This module evaluates sensor data and reconstruction results throughout the entire process, assigning multi-dimensional confidence indices to each point, each patch, and even the entire model. The modeling process is then optimized using these confidence levels as feedback. The sources and functions of confidence levels are explained in detail below.

[0055] S5.1 Geometric Confidence This reflects the degree of fit between the geometric data and the model. For smooth regions, it can be defined based on the plane fitting residuals: the smaller the residuals, the closer the points fit the plane, and the higher the geometric confidence. One calculation method is to map the residuals to scores from 0 to 1, for example, using a Gaussian function.

[0056] in The distance residual from a point to the model (such as a plane or mesh). The scale parameter is determined based on sensor noise. This formula makes the residual zero. As the residual increases, the confidence level decreases exponentially. For point clouds in complex regions, geometric confidence can also be defined by comparing the distance error between the point and the current fused mesh, or by comparing the difference between the current frame depth and the synthesized depth. In summary, A high value indicates that the data point is highly consistent with the model and may be part of a stable structure; a low value means that the point may be noise or an anomaly that the model cannot explain.

[0057] S5.2 Photometric Confidence This reflects the degree of agreement between the data and the physical lighting model. In indoor scenes, the color intensity collected by the sensor is affected by lighting and material. For diffuse surfaces, under stable lighting conditions, the color of the same point from different viewing angles should be basically consistent; however, highly reflective or transparent materials may lead to unreliable depth measurements or color variations with viewing angle. Photometric confidence modeling uses a physical simulation method: assuming each surface has fixed material properties, corresponding to a certain light reflection distribution. For static light sources and materials, the brightness difference of the same surface point at different times should mainly be due to the imaging angle changing according to Lambert's cosine law. If the observed brightness change cannot be explained by a simple lighting model, then special material or lighting variations are suspected. In practical implementation, photometric error can be calculated for the corresponding tracked pixels: such as a point in the current frame's color image. color Corresponding point in the previous frame Color difference (Given a known pose, a point can be projected onto another frame to find its corresponding position). Then consider the change in illumination between the two frames' imaging angles; if... If the value is significantly outside the normal range, the photometric confidence level should be lowered. For example, define:

[0058] in It is based on the brightness change predicted by the surface normal vector change and the light source model.

[0059] These are empirical parameters. Surface albedo Light source intensity, Direction of the light source With normal vector The angle between them. In this way, highly reflective or transparent areas (such as mirrors and glass) will achieve a lower [angle]. This suggests that the depth of these areas may be unreliable; areas with drastic changes in illumination (such as flashlight beams) will also be temporarily marked as low confidence. Conversely, areas with consistent illumination and that match model predictions will receive high confidence. .

[0060] S5.3 Time Confidence This reflects the continuity and stability of the scene over time. Static indoor scenes should appear consistently across multiple frames; if certain points or areas appear only in one frame or their positions change significantly across different frames, they may be dynamic objects or noise. This is for modeling purposes. This invention integrates historical record analysis and inter-frame consistency: on the one hand, it tracks the frequency and duration of each reconstructed point in the frame sequence, for example, assigning an "observation count" to each point during association; a higher number of observations distributed over a longer period indicates higher temporal confidence. On the other hand, it evaluates the consistency between the current frame observation and the pose prediction results of the previous frame. If a structure is predicted to be seen in the current frame based on the previous frame model but is not actually observed, the prediction for that structure is reduced. (May be occluded or moved). Conversely, if a new point suddenly appears in this frame that was previously invisible, it is temporarily given a low priority. This process waits for confirmation from subsequent frames. In this way, dynamic elements such as moving people or open doors can be identified as low-confidence, thus avoiding their incorrect inclusion in the static scene model.

[0061] S5.4 Confidence Fusion and Multidimensional Output: The confidence module comprehensively evaluates the information from the above dimensions and outputs a set of multidimensional confidence values. Specifically, it can calculate for different levels: for a single point, it can output... For a region of surface / object, the confidence score distribution of all its points is summarized to obtain the overall confidence score of the region (e.g., the mean of the geometric confidence scores is used as the planar confidence score). Confidence scores can be expressed numerically or in rank, such as high, medium, and low. Ultimately, the multidimensional confidence score can use a single dimension (focusing on a specific factor in a particular scenario) or be weighted to form a comprehensive confidence score. For example, defining a comprehensive confidence score... ( (as weights), or use the minimum value criterion (if the confidence level of any dimension is low, the whole is considered unreliable).

[0062] S5.5 Dynamic Optimization Based on Confidence: After obtaining the confidence assessment, the system will take measures for low-confidence data or model parts to optimize the reconstruction quality. Outlier removal: Points with extremely low geometric confidence (such as isolated points that drift away from the main structure or point clouds with large deviations from the plane) can be identified as noise points and removed from the fusion model in a timely manner to avoid affecting plane fitting or surface generation.

[0063] Model reclassification may require reclassification if a region has low overall geometric confidence but does not belong to any known smooth plane. For example, a flat tabletop initially classified as a complex region due to debris covering it, resulting in poor reconstruction, might be misclassified as a smooth surface if the confidence module detects high residuals and good photometric consistency. The system can then re-label the region as a smooth area and apply planar modeling to improve reconstruction quality. Conversely, if a smooth region has low confidence in fitting a plane (e.g., significant curvature is detected), it can be relabeled as a complex region and processed using mesh refinement methods.

[0064] Weighted fusion incorporates confidence levels as weights during point cloud fusion and plane fitting. High-confidence observations have a greater impact on model updates, while low-confidence observations have a smaller impact. For example, in plane fitting, each point is assigned a weight. This reduces the contribution of points with luminous anomalies to the planar solution. This prevents, for example, depth distortion points caused by strong reflections from significantly shifting the planar position.

[0065] Refinement iterations can be implemented by locally increasing sampling or guiding the deep network to focus more on areas with the lowest confidence levels. For example, if some details on a curved surface consistently have low confidence levels, it may indicate that the sensor is having difficulty acquiring them or that the model is incomplete. A re-sampling strategy can be adopted (such as prompting the user to scan around the area several times), or the weights for low-confidence areas can be increased in the loss function of the deep refinement network to encourage the network to work harder to restore the details in those areas.

[0066] Dynamic object filtering allows the system to exclude or separately label objects with low temporal confidence (suspected to be moving) from the final model. This ensures that the final output model primarily reflects the static scene structure. For example, if someone is walking in a room and captured by a depth camera, the temporal confidence will rate the human point cloud as low, thus preventing its accumulation during fusion and ensuring that the model does not retain a "ghosting" effect after the person disappears.

[0067] Through the continuous action of the confidence module, the reconstruction process becomes adaptive and highly reliable: the system constantly adjusts itself based on the confidence level, making full use of reliable information and promptly correcting or discarding unreliable information. This feedback-driven closed-loop mechanism effectively avoids model collapse or distortion caused by the accumulation of erroneous data in traditional passive reconstruction.

[0068] Tested on a set of indoor scenes in the ScanNet dataset: ScanNet contains a rich collection of indoor room scan sequences, including various structures such as walls, furniture, and clutter.

[0069] The method of this invention was used to reconstruct an office scene. First, the system utilizes IMU-assisted camera trajectory to ensure precise alignment of multi-frame point clouds, constructing a dense point cloud of the room during real-time scanning. Subsequently, region segmentation successfully identified walls, floors, and ceilings as smooth areas, and sofas, tables, and chairs as complex curved surfaces. For walls, the planar modeling module quickly provides the planar equation of the entire wall and, with the help of a large model prior, extends to corners not directly observed, ensuring a complete and intact reconstructed wall surface. For curved furniture such as sofas, point cloud fusion densely merges surface points acquired from different angles, clearly presenting the curved contours of the sofa; the depth refinement network further compensates for minute details such as cushion wrinkles, making the sofa surface more realistic. The confidence module plays a crucial role in this process: due to reflections, some depth points on the sofa leather surface have large deviations and are assigned low geometric confidence. The network pays special attention to these areas during refinement, adjusting the surface shape to match the surrounding high-confidence areas. Similarly, a large glass window in the scene, due to its transparency making depth measurement impossible, is assigned an extremely low confidence level by the confidence module based on light anomalies. The system avoids outputting incorrect geometry in this area, and at the same time, by modeling the smooth region, it approximates the window position with only one plane, thereby ensuring model consistency without introducing noise points.

[0070] Tests were conducted on several sequences from the TUM RGB-D dataset, which provides sequences acquired by a handheld RGB-D camera moving indoors and offers precise trajectories as an evaluation benchmark. Taking the "fr2 / desk" sequence as an example (a desk scene with a desk, chair, and walls), the method of this invention reconstructs the desktop scene using the provided trajectory (equivalent to incorporating IMU information). Results show that the desktop and whiteboards parallel to the walls are accurately classified as smooth areas and reconstructed as regular planes; computer monitors, keyboards, chairs, etc., as complex areas, are reconstructed with high precision (e.g., ...). Figure 4As shown), its curved edges are clear and free of unnecessary noise. In particular, the support frame of a certain chair has a curved shape, which traditional planar reconstruction methods might overlook. However, this method successfully reconstructs the curved details of the chair through point cloud fusion and network refinement, without losing information due to its small area (e.g., Figure 6 As shown). Compared to the reconstruction results without confidence modeling (which show some scattered points and blurred edges), introducing confidence eliminates the erroneous point cloud caused by the swaying door, excluding it from the final model, thus keeping the background wall surface flat and free of stray artifacts (as shown). Figure 7 As shown in the figure, in the comparison between the reconstructed point cloud and the real scan, the average point distance error is reduced, the flatness error of the planar region is reduced, and the model size (the number of mesh faces generated by the final point cloud through the Boson reconstruction) is reduced (because the plane is simplified to a small number of polygons), which proves that the present invention can control the model complexity while improving accuracy.

[0071] In contrast, if only VGGT is used in the TUM dataset, the result is as follows: Figure 8 , Figure 9 As shown, the modeling defects are clearly visible.

[0072] In practical applications, the method of this invention has been applied in a real-time scanning system developed in the laboratory.

[0073] Indoor scene 3D reconstruction system based on deep fusion and confidence modeling (e.g.) Figure 2 As shown, it includes a composite data frame acquisition module, a region division module, a smooth region modeling module, a complex region fusion and refinement module, and a confidence modeling module.

[0074] The composite data frame acquisition module is used to acquire a sequence of composite data frames for indoor scenes. Each frame includes a color image, a depth map, inertial measurement unit (IMU) data, and camera attitude data, which are used to provide multimodal synchronization information. The region segmentation module is used to process the composite data frame and divide the scene into smooth regions and complex curved surface regions based on geometric features and image features. The smooth regions include roughly planar structures such as walls, floors, and ceilings, while the complex curved surface regions include non-planar structures such as furniture and curved objects. The smooth region modeling module is used to perform 3D reconstruction of the smooth region using a geometric modeling method guided by the visual features of a large model. It uses a pre-trained visual model to extract global geometric priors from color images, combines them with depth maps and IMU pose information, performs plane fitting and modeling on the smooth region, and quickly reconstructs 3D models of planar structures such as walls and floors. The complex region fusion and refinement module is used to reconstruct the complex curved surface region by combining a point cloud fusion strategy with a deep residual refinement network. Multiple frames of deep point clouds are registered and fused in the global coordinate system to generate an initial complex region point cloud model. The model is then refined using a deep learning network to compensate for surface details and obtain a high-precision 3D model of the complex region. The confidence modeling module is used to introduce confidence modeling into the reconstruction process, calculate multi-dimensional confidence indices by integrating factors such as historical observation records, physical simulations and time continuity; and dynamically adjust the reconstruction process based on the confidence indices, including weighted fusion of high-confidence data, removal of low-confidence data, and optimization of regional classification and model parameters, outputting a complete 3D model of the indoor scene optimized by confidence.

[0075] Users can walk around an indoor space for about 1-2 minutes with an RGB-D camera, and a 3D model of the current indoor environment will be generated in real time on a tablet. During the reconstruction process, smooth building structures will be presented as a simple flat surface. For items of particular interest to the user (such as a sculpture or piece of furniture), the model will be gradually refined through multiple scans from multiple angles. The confidence level interface will indicate which parts of the model have insufficient data or poor quality, prompting the user to rescan. The final output 3D model can be used for indoor navigation, interior design, etc., and its accuracy and completeness are far superior to traditional point cloud stacking methods, demonstrating good practicality.

[0076] The above embodiments fully verify the effectiveness of the present invention. Of course, the present invention is not limited to the specific examples described above. This method is applicable to different indoor environments and different sensor configurations. For example, if a binocular camera is used instead of an RGB-D camera, depth can be calculated visually first, and then composite frames can be formed to apply the various modules of this method. Furthermore, for outdoor or dynamic scenes, the confidence module of this method can also assist in filtering out dynamic objects and improving the quality of reconstructed static parts. All equivalent substitutions or improvements made within the spirit and principle of this invention should be included within the scope of protection of this invention.

[0077] For ease of implementation, the pseudocode flow of this invention is as follows, to further illustrate the logic of each step (corresponding to...). Figure 3 (as shown) Input: Composite data frame sequence Each frame contains

[0078] Output: 3D model M of the indoor scene (a comprehensive model including smooth areas and complex curved areas) initialization: Global point cloud model

[0079] Global confidence model

[0080] Process each frame of data in a loop: For : 1) Data Acquisition and Preprocessing: - from Extract color images Depth map ,attitude .

[0081] - Correct the depth map and convert it to a point cloud in camera coordinate system. .

[0082] - Utilizing posture Will Transform to the global coordinate system to obtain the point cloud. .

[0083] - If a keyframe strategy is used, the decision to process this frame is made based on the motion amplitude and reconstruction requirements.

[0084] 2) Scene area division: -calculate Normal vectors and curvature at each point in the middle, combined with Image segmentation, dividing point clouds into smooth regions and complex curved regions: .

[0085] .

[0086] - Use the IMU gravity direction to classify and smooth the horizontal / vertical planes in the candidates.

[0087] - Output smooth region point cloud collection and complex region point cloud collection .

[0088] 3) Geometric modeling of smooth regions: For each connected region in : - Add the point cloud of this region to the point set of the corresponding global planar model (append if it already exists, otherwise create a new planar model).

[0089] - Perform plane fitting on the accumulated set of plane points to solve for and update the plane parameters. .

[0090] - Prior point cloud prediction using large visual models (such as VGGT) Align scale and pose and fuse to enhance the current planar model.

[0091] - Calculate the plane fitting residuals and update the geometric confidence distribution of the plane model.

[0092] EndFor 4) Point cloud fusion and refinement in complex regions: - Will Integrate into global point cloud (Voxel hash accumulation)

[0093] - Obtain a rough model .

[0094] - Utilize current frame depth and Differences in synthesis depth, after passing through a residual network Predicted depth residual .

[0095] - according to Correction Local vertex or update The corresponding points are used to refine the model.

[0096] 5) Confidence assessment and feedback: - Geometric confidence level: calculated based on residual or point cloud consistency. Assign a value to each point / region.

[0097] - Photometric confidence level: comparison Differences from historical images, calculation distributed.

[0098] - Time confidence: Observation count at update point, calculated wait.

[0099] - Combining to obtain multidimensional confidence scores .

[0100] - based on Adjust the processing result of the current frame: * Remove Outliers that are too low.

[0101] * If a certain smooth model as a whole If the value is low, move to a complex region and remodel; conversely, if the value is low, proceed with the modeling.

[0102] * During fusion and fitting As a contribution to the weight adjustment point.

[0103] * Low-confidence areas were recorded and marked as requiring further scanning / optimization.

[0104] EndFor Output phase: - Summarizes all global planar models and point clouds of complex curved surface regions.

[0105] - Perform model stitching: trim conflicts, stitch boundaries, and ensure model coherence.

[0106] - Reconstruct the final cypress tree from the final point cloud to output the final 3D model. (Including a hybrid representation of planes and grids), with confidence ratings for each part.

[0107] The essential explanation for choosing the VGGT model is that RGB-D cameras cannot accurately detect the depth / point cloud of flat glass surfaces (high transmittance), mirror surfaces (high reflectivity), and pure black surfaces (low reflectivity). These areas are commonly found in indoor environments (glass, blackboards, glass doors and windows, mirrors). Furthermore, large models like VGGT are not limited by imaging principles and can compensate for these deficiencies, significantly improving the completeness and accuracy of indoor reconstructions. Before introducing a large model for planar guidance, registration and alignment are required, consisting of two parts. First, scale alignment, i.e., aligning the model's predicted depth with the depth detected by the RGB-D camera, is relatively simple. This can be achieved by calculating the average alignment ratio using multiple RGB predicted depth maps and actual detected depth maps, point-to-point (removing outliers or holes). The second part involves using VGGT-SLAM for pose alignment. Additionally, the model's predicted output is depth and point cloud, which can be converted using camera parameters.

[0108] To further illustrate the advantages of this method, other methods were also introduced for reconstruction of Replica data in this embodiment, and the results are shown in the table below: Accuracy (cm) Completeness (cm) Completeness rate (%) iMap 6.95 5.33 66.6% DI-Fusion 19.40 10.19 72.96% NICE-SLAM 2.85 3.00 89.33% Method of the present invention 2.00 2.6 89.62% Table 1 Replica Data Reconstruction Results The replica is a virtual dataset that includes reconstructed real values, which can quantitatively evaluate the superiority of the method of this invention.

[0109] The pseudocode above describes the main flow of the method of this invention. In actual implementation, real-time frame-by-frame processing or batch offline processing can be selected according to hardware performance. In addition, in order to improve reconstruction efficiency, the modules can be executed in parallel. For example, while running deep residual network inference on the GPU, the CPU performs point cloud fusion and confidence calculation.

Claims

1. A method for 3D reconstruction of indoor scenes based on deep fusion and confidence modeling, characterized in that, Includes the following steps: S1 Composite Data Frame Acquisition: Acquire a sequence of composite data frames for an indoor scene. Each frame includes a color image, depth map, inertial measurement unit (IMU) data, and camera attitude data, which are used to provide multimodal synchronization information. S2 Region Division: The composite data frame is processed, and the scene is divided into smooth regions and complex curved surface regions according to geometric features and image features. The smooth regions include planar structures such as walls, floors, ceilings, glass planes, mirrors, and blackboards, while the complex curved surface regions include non-planar structures such as furniture and curved objects. S3 Smooth Region Geometric Modeling: For the smooth region, a geometric modeling method guided by large model visual features is used for 3D reconstruction; global geometric priors are extracted from color images using a pre-trained visual model, and combined with depth maps and IMU pose information to perform plane fitting and modeling on the smooth region, quickly reconstructing the 3D model of the planar structure of the wall and ground. S4 Complex Surface Region Fusion and Refinement: For the complex surface region, a point cloud fusion strategy combined with a deep residual refinement network is used for reconstruction; multiple frames of deep point clouds are registered and fused in the global coordinate system to generate an initial complex region point cloud model, and the model is refined by a deep learning network to compensate for surface details, resulting in a high-precision 3D model of the complex region. S5 Confidence Modeling and Optimization: A confidence modeling module is introduced during the reconstruction process to calculate a multi-dimensional confidence index by integrating historical observation records, physical simulation, and time continuity factors. The reconstruction process is dynamically adjusted based on the confidence index, including weighted fusion of high-confidence data, removal of low-confidence data, and optimization of regional classification and model parameters, outputting a complete 3D model of the indoor scene optimized by confidence.

2. The method according to claim 1, characterized in that, The composite data frame consists of synchronously acquired multimodal sensor data, wherein the color image and depth map are aligned pixel by pixel, the IMU data provides the camera's acceleration and angular velocity, and the attitude data is the camera's position and orientation in the world coordinate system; preferably, the attitude data is obtained by fusing color image features and IMU measurements through visual inertial odometry to improve the accuracy and robustness of camera trajectory estimation.

3. The method according to claim 2, characterized in that, The region segmentation steps include: calculating the point cloud normal vector and curvature using the depth map, marking regions with consistent normal vectors and low curvature as smooth region candidates; combining the gravity direction provided by the IMU, identifying candidate regions with normal vectors that are close to vertical or horizontal as planes of walls or the ground; simultaneously using color image segmentation or detection algorithms to identify the location of the plane, and combining it with depth information to confirm the boundary of the smooth region; automatically classifying point clouds not classified as smooth regions as complex curved surface regions; the region segmentation ensures that smooth regions and complex regions complement each other spatially without omission or repetition.

4. The method according to claim 3, characterized in that, The smooth region geometric modeling steps include: using a robust fitting algorithm to perform planar fitting on the point cloud of the smooth region to obtain planar parameters; introducing geometric priors provided by a pre-trained large-scale vision model, performing monocular 3D reconstruction prediction on the color image to obtain a dense depth map / point cloud or planar guess, aligning the prior with the actual depth map at scale, and then performing alignment and fusion through point cloud-to-point cloud registration to expand and smooth the fitting results; during the fusion process, different weights are assigned to the data from the sensor and the data from the prior model to ensure that the fitting results conform to both actual measurements and global consistency. The smooth region model obtained through the above steps is represented in the form of planar segments, with accurate normal vectors, boundary ranges, and high geometric accuracy.

5. The method according to claim 4, characterized in that, The complex surface region fusion and refinement steps include: progressive fusion using multi-frame depth map point clouds; accumulating the complex region point clouds from each frame into a global 3D raster using voxel hashing or voxel filtering strategies to generate a preliminary fused point cloud model; after obtaining the initial fused model, performing surface reconstruction to generate a coarse triangular mesh, and performing local geometric correction on the coarse mesh through a depth residual refinement network to improve the surface detail accuracy. The depth residual refinement network takes the difference between the measured depth and the current mesh projection depth as input and outputs the residual used to update the mesh vertex positions, thereby obtaining a refined complex surface region model.

6. The method according to claim 5, characterized in that, The deep residual refinement network is a VisionTransformer neural network, which consists of an encoder and a decoder. The encoder extracts multi-scale features that differ between the current frame observation and the coarse model, and the decoder outputs a deep residual map with the same size as the image. The network is obtained through offline training on multiple indoor datasets. The training objective is to minimize the loss function that minimizes the difference between the prediction residual and the real model, including pixel depth error loss and normal vector smoothing loss. During the inference phase, the network processes the depth data of each frame or several frames, gradually refining and updating the mesh model of complex regions to achieve the reconstruction of high-frequency details.

7. The method according to claim 6, characterized in that, The confidence modeling module calculates multidimensional confidence for each data point, local region, or global model, including: geometric confidence, photometric confidence, and temporal confidence. Geometric confidence is based on the geometric error assessment between the reconstructed model and the observation data; the smaller the error, the higher the confidence. Photometric confidence is based on the degree of conformity between image observation and physical lighting model; if the brightness change of the same surface under multiple views conforms to the Lambertian model, the photometric confidence is high; conversely, if there is strong specular reflection or light variation, the confidence decreases. Temporal confidence is based on the consistency and continuity of cross-frame observations; static structures that are continuously observed in multiple frames have high confidence, while points that appear occasionally or whose positions change have low confidence. The confidence modeling module integrates the above factors to form a confidence vector for each reconstructed element, which is used to guide model optimization.

8. The method according to claim 7, characterized in that, The confidence index is used to provide feedback and optimize the 3D reconstruction process, including: averaging multi-frame data using confidence as a weight during point cloud fusion, assigning greater weight to high-confidence observations and accumulating low-confidence observations with smaller weights, thereby reducing the impact of outlier data on the model; removing outliers or abnormal regions based on confidence during plane fitting and surface refinement, such as discarding noise points with low geometric confidence and excluding dynamic object point clouds with low temporal confidence, to ensure that the output model only contains stable and reliable structures; and re-evaluating previous classification results using confidence during region classification, triggering re-classification or changing the modeling strategy for regions where the overall confidence deviates from expectations, thereby adaptively improving model accuracy and completeness.

9. The method according to any one of claims 1-8, characterized in that, The method is applicable to real-time 3D reconstruction systems, where composite data frames are acquired in real time by an RGB-D camera and IMU on a mobile device. The system includes a data acquisition module, a region segmentation module, a smooth region modeling module, a complex region modeling module, a confidence assessment module, and a model fusion module. The system is configured to execute the above steps in parallel or pipeline mode, so that the 3D model can be updated frame by frame and the reconstruction results can be displayed in real time during the process of the sensor moving and scanning the indoor environment.

Images