A structured three-dimensional reconstruction and generative view repair method for a library scene

Abstract

This invention discloses a structured 3D reconstruction and generative viewpoint restoration method for library scenarios, comprising: Step 1, processing the acquired monocular video stream to obtain a dense point map of the target to be reconstructed; Step 2, constructing a 3D Gaussian sputtering model based on the dense point cloud through an anchor-driven sliding window joint optimization mechanism; Step 3, using an octree data structure to spatially index the 3D Gaussian primitives, combining view frustum visibility analysis to perform occlusion culling and redundant point pruning, and constructing a multi-detail level attribute aggregation model; Step 4, actively rendering low-confidence regions by applying pose perturbation to expose defects, using the rendered depth map as a strong geometric constraint to guide the diffusion model to generate restoration textures, and feeding them back as pseudo-ground values ​​to the 3D Gaussian field for incremental training.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D graphics and generative artificial intelligence, and to a structured 3D reconstruction and generative perspective restoration method for library scenarios. Background Technology

[0002] With the rapid development of digital twin and metaverse technologies, high-fidelity 3D reconstruction of large and complex indoor scenes has demonstrated tremendous application value in fields such as spatial digitization, virtual tours, cultural heritage protection, and mixed reality. However, these complex large indoor scenes typically possess highly challenging unstructured characteristics: firstly, the scene contains a large number of geometrically repetitive structures (such as densely and regularly arranged shelves or bookshelves); secondly, it is filled with large areas of weak texture (such as solid-color ceilings and large areas of white walls); and finally, due to the compact spatial layout, severe visual obstruction is inevitable during physical data acquisition.

[0003] In existing 3D reconstruction technology pipelines, the Structure-from-Motion (SfM) algorithm is typically heavily relied upon for front-end camera pose estimation and sparse point cloud reconstruction. However, when dealing with the complex indoor scenes described above, traditional SfM algorithms face significant bottlenecks. On the one hand, recurring textures easily lead to "perceptual aliasing" during the feature matching stage; on the other hand, large areas of weak texture cannot yield sufficient and effective feature points. These dual shortcomings make traditional methods prone to global scale drift of camera trajectories in the reconstruction of long video sequences, and may even lead to direct breaks in the reconstructed trajectory.

[0004] In recent years, explicit radiative field techniques, represented by 3D Gaussian Splatting (3DGS), have become a research hotspot in the field of 3D vision due to their superior rendering quality and real-time rendering speed. However, when directly applying existing 3DGS technology to complex large-scale indoor scenes, the following technical challenges still urgently need to be addressed:

[0005] First, there is the issue of geometric structure collapse and artifacts. 3DGS is highly dependent on the accuracy of the initial point cloud and camera pose. When the front-end SfM provides poor initial values, the Gaussian optimization process, which lacks explicit geometric smoothing constraints, will produce ill-conditioned updates in weak texture regions, resulting in severe geometric deformation and "floating artifacts" in the reconstruction results.

[0006] Second, there's the bottleneck of memory bloat and resource scheduling. Existing 3DGS uses unordered discrete primitives to represent scenes, lacking scheduling and spatial indexing mechanisms for Level of Detail (LOD). When faced with large-scale scenes involving tens or even hundreds of millions of GoS primitives, not only is the storage volume enormous, but it also easily leads to memory explosion on terminal devices, making it difficult to achieve smooth real-time rendering on resource-constrained conventional hardware.

[0007] Third, there is the challenge of restoring 3D consistency in occluded areas. Existing viewpoint restoration techniques for geometric holes caused by physical occlusion mostly remain at the level of 2D image generation. Because 2D generation models lack perception and geometric constraints of 3D physical perspective, directly using them for 3D spatial perspective synthesis leads to severe "multi-view inconsistency" (i.e., the Janus problem) and perspective collapse, failing to meet the fidelity requirements for omnidirectional, blind-spot-free roaming.

[0008] In summary, overcoming the pose drift problem of traditional visual algorithms in extremely complex environments, breaking through the memory bottleneck and artifact defects of explicit radiation fields, and achieving intelligent repair of occluded areas with three-dimensional geometric consistency have become key technical challenges that urgently need to be solved in the field of large-scale indoor digital twins. Summary of the Invention

[0009] Purpose of the invention: The technical problem to be solved by the present invention is to provide a structured 3D reconstruction and generative perspective repair method for complex large indoor scenes, which addresses the shortcomings of the existing technology.

[0010] To address the aforementioned technical problems, this invention discloses a structured 3D reconstruction and generative perspective restoration method for complex large-scale indoor scenes, characterized by the following detailed steps:

[0011] Step 1: Adaptive temporal segmentation of the acquired monocular video stream is performed using a large visual geometric model to regress the local dense point map and relative pose, and the coordinate system of each sub-map is optimized and aligned through the global pose map.

[0012] Specifically, step 1 includes the following sub-steps:

[0013] Step 1-1, Adaptive Temporal Segmentation and Feature Extraction of Video Stream: Obtaining long monocular video sequences of complex large indoor scenes (such as libraries). Where T is the total number of frames. Since long-sequence global optimization incurs huge memory overhead and is prone to cumulative drift, this invention employs a sliding window mechanism to divide the video into a series of overlapping local sub-windows. In each child window Internally, images are input into a pre-trained Visual Geometry Large Model (VGGT). This large model contains a feature extraction backbone network based on a multi-head self-attention mechanism, capable of extracting dense feature maps with geometric translation and rotation invariance within the global receptive field. Through feature matching and a depth regression head, it directly outputs the dense correspondence (dense point map) of adjacent image pairs within a sub-window and the initial inter-frame relative pose.

[0014] Steps 1-2, Trajectory estimation using the local similarity transformation group Sim(3): In the traditional rigid body transformation group SE(3) (which only includes rotation and translation), the inherent scale ambiguity of a monocular camera easily leads to trajectory scale drift in large scenes. Therefore, this invention introduces a relative scale factor s in the local coordinate system, extending the optimization space to the similarity transformation group Sim(3). For the relative pose between any two frames... Its matrix form is expressed as:

[0015]

[0016] in, for orthogonal rotation matrix, It is a three-dimensional translation vector. is the relative scale factor. The optimal local pose set within the sub-window is obtained by minimizing the reprojection error of the dense point map through bundle adjustment.

[0017] Steps 1-3, Global Pose Graph Optimization Based on Lie Algebra Cut Space: For each sub-window To address the issue of inconsistent coordinate systems, constraint edges are established at the overlapping shared frames of adjacent sub-windows to construct a global pose graph. , and These correspond to the sets of similarity transformation matrices between the local sub-window set and the adjacent sub-window set, respectively. Define the objective function for global pose graph optimization. To solve for the optimal similarity transformation matrix Align all local sub-windows to a unified world coordinate system:

[0018]

[0019] In the formula, and These are the global similarity transformation matrices from the k-th and (k+1)-th sub-windows to the world coordinate system, respectively; and For overlapping shared frames, the local poses are located within their respective local sub-windows. The logarithmic mapping operator from the Sim(3) group to its corresponding Lie algebra sim(3) is used to map the transformation matrix on the manifold space to the Euclidean tangent space in order to facilitate the calculation of the gradient and error vector; The square of the Mahalanobis distance. The covariance matrix (the inverse of the information matrix) is determined by the local reconstruction confidence output by the large visual geometry model. It is used to reduce the weight of weak texture regions and prevent mismatches from causing global alignment failure.

[0020] Step 2: Construct an anchor-driven sliding window joint optimization mechanism, incorporate camera extrinsic parameters into the differential rendering pipeline, and alternately optimize 3D Gaussian properties and camera trajectory by combining monocular depth prior.

[0021] Specifically, step 2 includes the following sub-steps:

[0022] Step 2-1, Explicit Parametric Representation of the 3D Gaussian Scene: Using the dense point cloud obtained after alignment in Step 1 as the initial value, a 3D Gaussian Splatting model is constructed. The scene is represented as a set of millions of discrete 3D Gaussian elements. For each Gaussian element, its spatial distribution function is expressed as:

[0023]

[0024] in, The mean (center coordinate) of the Gaussian elements. This is a three-dimensional covariance matrix. To ensure that the covariance matrix remains positive semidefinite throughout the gradient descent optimization process, the three-dimensional covariance matrix is ​​decomposed into scaling matrices. With rotation matrix The product of: In addition, each Gaussian meta-element is also bound to an opacity property. And the coefficients c of the spherical harmonics (SH) function used to represent view-dependent colors.

[0025] Step 2-2, Differentiable rasterization and anchor-driven trajectory smoothing: The Gaussian primitives described above are projected onto a two-dimensional image plane, and rendering is performed using a differentiable rasterizer. For pixel coordinates (u,v), the rendered color C(u,v) and the rendered depth... Both calculations are based on the Alpha mixing principle and involve approximate volume integrals along the ray.

[0026]

[0027]

[0028] In the formula, N is the set of intersecting Gaussian elements sorted along the viewpoint ray. Let the depth of the k-th Gaussian element along the Z-axis in the camera coordinate system be denoted as . The opacity is calculated by weighting the two-dimensional Gaussian probability density.

[0029] To correct the residual small pose error from step 1, the camera extrinsic parameters are treated as learnable parameters and incorporated into the differentiable rendering pipeline. High-confidence keyframes are selected as "anchor frames" and their poses are frozen, allowing the remaining "non-anchor frames" to be fine-tuned in the Lie algebra se(3) space. To prevent trajectory distortion caused by unconstrained fine-tuning, a trajectory smoothing loss based on large model priors is applied. :

[0030]

[0031] In the formula, and The poses of adjacent frames are optimized for the current network iteration. For a priori relative transformation, The vee operator for Lie algebras is used to extract a 6-dimensional error vector (3-dimensional rotation error and 3-dimensional translation error) from an antisymmetric matrix.

[0032] Steps 2-3, Scale-Transfer Invariant Depth Regularization: In indoor scenes, large, weakly textured areas such as ceilings and white walls are prone to generating floating noise under pure photometric loss (e.g., L1 and D-SSIM). Therefore, a dense depth map is extracted from the output of the monocular depth estimation network. Due to the unknown global scale factor and depth offset in monocular depth measurement, this system dynamically calculates the optimal affine parameters. :

[0033]

[0034] The least squares problem has a closed-form solution in each iteration. Subsequently, a scale-displacement invariant depth regularization loss is constructed. Apply direct surface constraints to the geometric position of the three-dimensional Gaussian:

[0035] Step 3: Spatial indexing of 3D Gaussian primitives is performed using an octree data structure. Occlusion culling and redundant point pruning are combined with view frustum visibility analysis, and a multi-level detail attribute aggregation model is constructed.

[0036] Specifically, step 3 includes the following sub-steps:

[0037] Step 3-1, Octree Space Partitioning Based on Morton Code: Obtain the 3D coordinate extrema of all Gaussian elements in the scene, and construct a global axial bounding box (AABB) enclosing the entire indoor scene as the root node of the octree. The mean center point of the Gaussian elements... Discretization and interleaving operations are performed to generate unique Morton codes. Morton codes reduce three-dimensional spatial coordinates to one dimension, enabling contiguous storage of spatially adjacent Gaussian primitives in memory. All primitives are then allocated to the leaf nodes of the octree according to their Morton codes.

[0038] Step 3-2, Hierarchical Aggregation and Moment Matching of Gaussian Attributes: The "Super Gaussian" attribute (a single Gaussian primitive that can physically and visually replace the set of all smaller child primitives within it) of the non-leaf nodes (intermediate nodes) of the octree is calculated from bottom to top. For spatial merging, an opacity-weighted moment matching algorithm is used.

[0039] For a given parent node, whose child node indices are all k, the spatial center position of the aggregated parent node (i.e., the Gaussian mean) is... ) and opacity The calculation is as follows:

[0040]

[0041] To maintain the original spatial geometric volume distribution, the covariance matrix of the parent node... Not only is it necessary to aggregate the covariance of child nodes, but also to compensate for the geometric offset covariance from the center of each child node to the center of the parent node. The precise calculation formula is as follows:

[0042]

[0043] in, , , These represent the opacity, covariance matrix, and mean of the k-th Gaussian element in the child node, respectively. After obtaining these "super Gaussian" properties, they play a decisive role in the subsequent differentiable rasterization rendering stage: the mean... This determines the absolute center position of the "super Gaussian" in three-dimensional space, and its two-dimensional center coordinates on the screen are determined by projection through camera parameters during rendering; the covariance matrix ( The opacity determines the three-dimensional shape and physical volume of the "super Gaussian," and during the rendering stage, it controls the size and stretching direction of the pixels it covers on the two-dimensional image surface; This determines the visual occlusion and solidity of the macroscopic set, and in the alpha blending calculation of the rendering, it determines the transmittance of the background information.

[0044] Step 3-3, Hierarchical Frustum Analysis and Visibility Physics Pruning: During the rendering inference phase, the octree constructed in Step 3-1 is used for scene traversal. Based on the spatial distance between the current camera viewpoint and nodes, the "Super Gaussian" or leaf node Gaussian primitives calculated in Step 3-2 are dynamically scheduled for rendering, thereby achieving seamless switching of Levels of Detail (LOD). During this traversal, the octree nodes are projected onto screen space through the camera intrinsic matrix, and frustum culling is performed. Simultaneously, a block-based depth buffer is maintained. By comparing the camera coordinate system Z-value (depth) of the current Gaussian node with the nearest surface depth value recorded in the depth buffer, nodes completely behind the current rendered pixel depth value, or whose accumulated opacity on the ray has reached 1.0 (i.e., light cannot penetrate), are skipped early.

[0045] To permanently reduce memory usage, during the later stages of offline training, the system uniformly samples M virtual views across the entire space and calculates the highest transparency contribution of each Gaussian unit across all views. When the contribution value is lower than the preset minimum threshold... When the primitive is determined to be a "dead Gaussian" enclosed inside an object (such as a point inside the solid wood structure of a bookshelf), it is physically deleted from the video memory.

[0046] Step 4: Actively render low-confidence regions by applying pose perturbation to expose defects. Use the rendered depth map as a strong geometric constraint to guide the diffusion model to generate repair textures, and feed it back as pseudo-ground values ​​to the three-dimensional Gaussian field for incremental training.

[0047] Specifically, step 4 includes the following sub-steps:

[0048] Step 4-1, Active Viewpoint Sampling and Occlusion Mask Extraction: In scenes such as libraries, due to the dense bookshelves, there are many visual blind spots in the training trajectory (such as the sides and rear of the bookshelves). The keyframe poses of the original training trajectory are used... Based on its Lie algebra The internal perturbation vector is applied by sampling through a multidimensional Gaussian distribution and then randomizing the rotation and translation. Generate virtual observation pose = .

[0049] Under this virtual pose, the RGB image and cumulative opacity map are rendered using the constructed 3D Gaussian model. Set the opacity threshold. (e.g., 0.95), when < When this occurs, it indicates that the ray did not hit a sufficiently dense geometry, indicating the existence of a spatial hole caused by physical obstruction, and a binary repair mask is generated:

[0050]

[0051] Step 4-2, Depth Condition Injection and Generative Diffusion Denoising: To avoid perspective errors when filling holes in a large 2D generated model (e.g., drawing a bookshelf from the side as the front), a rendering depth map from the virtual perspective is extracted. The spatial conditional control network ControlNet maps these signals into depth control signals with spatial topology awareness through zero-convolutional network layers. The rendered RGB image with holes obtained in step 4-1 is mapped to latent spatial features using a VAE encoder. And noise is added during the forward diffusion process to obtain Simultaneously, prompt text describing the semantics of the library scene (such as "neatly arranged book spines, wooden bookshelves") is input into a pre-trained text encoder to extract text semantic feature signals. Finally, the depth control signal Text semantic feature signals Inverse denoising U-Net network co-injected into the Latent Diffusion Model In the middle, the objective function for modified diffusion denoising is:

[0052]

[0053] This step forces the generated content to not only be realistic in texture, but also to have its edges and surface undulations fit together perfectly. Provided three-dimensional physical fault edges.

[0054] Step 4-3, Closed-loop pseudo-true value feedback and Gaussian field incremental evolution: High-quality full-resolution restored image output from diffusion model decoding. It possesses perfect 3D perspective consistency. Compare it with the corresponding virtual pose. This is treated as a set of "pseudo ground truths" and re-injected into the original training data stream of the 3D Gaussian. During the incremental fine-tuning phase, to avoid overwriting existing real physical observations, a confidence-based spatially weighted loss function is constructed:

[0055]

[0056] In the formula, These are real video frame images. The image is rendered based on the Gaussian model. and All use pixel-level Loss and structural similarity The weighted combination of losses is calculated using the following formula:

[0057]

[0058] To balance the global weighting coefficient, its value ranges from [0.1, 1.0]. In this embodiment, a value of 0.2 is preferred to ensure that the accuracy of existing real observation areas is not interfered with while repairing holes. This is a Gaussian decay spatial weight map calculated based on the distance to the mask edge. Specifically, the weight value of a pixel in this weight map is positively correlated with its physical distance to the mask edge: in the center and deep within the mask region (hole), due to the complete lack of real observation, the highest generation weight is assigned to fully guide the hole filling; while closer to the mask edge (i.e., the boundary between the generated region and the real observation), the generation weight smoothly decays to zero according to a Gaussian function. Through this distance-based spatial smooth decay mechanism, the 3D Gaussian field is forced to converge to the original real physical observation at the boundary, thereby completely eliminating the rigid stitching gap between the generated region and the real region, and achieving a seamless 3D fusion of locally generated textures and the global real environment.

[0059] Based on the total loss function Driven by the reverse gradient propagation, and utilizing the unique adaptive density control mechanism of 3D Gaussian sputtering, Gaussian elements in the void region will sense the sharp increase in spatial gradient. This actively triggers the cloning and splitting of primitives. After multiple rounds of closed-loop iteration of "rendering-repairing-retraining", the missing physical space will grow into a solid structure, completely realizing high-fidelity, blind-angle-free 3D digital roaming.

[0060] Beneficial effects:

[0061] 1. Significantly improves the robustness of trajectory estimation: It abandons traditional local feature matching and utilizes the global receptive field of a large visual geometric model for temporal block segmentation, and... Global pose graph optimization within the space effectively eliminates long-sequence cumulative drift and trajectory breakage caused by single textures and repetitive structures.

[0062] 2. Achieving joint optimization and geometric smoothing: Camera extrinsic parameters are incorporated into the differentiable rendering pipeline, and scale-displacement invariant depth regularization constraints are constructed using a monocular depth map. This effectively eliminates "floating artifacts" in weakly textured areas such as white walls, ensuring that the generated Gaussian geometry accurately conforms to the real physical surface.

[0063] 3. Significantly reduces memory usage and rendering overhead: The innovative use of an octree data structure to perform spatial indexing and multi-level detail (LOD) mathematical aggregation on unordered 3D Gaussian primitives, combined with view frustum visibility pruning to remove redundant points, greatly compresses the storage volume of the model, enabling it to smoothly achieve real-time roaming of tens of millions of Gaussian scenes on regular web platforms.

[0064] 4. Achieving Physically Compliant Repair with 3D Consistency: Constructing a Closed-Loop Mechanism of "Rendering-Repair-Retraining". Utilizing the rendered depth map as a strong conditional control signal for ControlNet, the diffusion model is guided to generate textures that strictly conform to perspective rules; by injecting the repaired image as a pseudo-ground value into the Gaussian field incremental evolution, perspective errors and multi-view inconsistencies generated in 2D are completely avoided, achieving a high-fidelity digital twin without blind spots. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the overall process of a structured 3D reconstruction and generative perspective restoration method for a library setting, as provided in an embodiment of the present invention.

[0066] Figure 2 This is a schematic diagram illustrating the principle of adaptive temporal segmentation and global pose graph optimization for video streams in an embodiment of the present invention.

[0067] Figure 3 This is a schematic diagram of hierarchical spatial indexing and multi-level of detail (LOD) visibility pruning based on an octree in an embodiment of the present invention.

[0068] Figure 4 This is a schematic diagram of the deep-guided generative closed-loop repair and pseudo-truth value incremental training framework in an embodiment of the present invention.

[0069] Figure 5 This is a schematic diagram of a monocular long video sequence frame with overlapping features collected in an embodiment of the present invention.

[0070] Figure 6 This is a schematic diagram of the three-dimensional point cloud spatial structure of the library generated after global pose graph optimization in an embodiment of the present invention.

[0071] Figure 7The image shows a comparison of the process and rendering effect of the viewpoint repair mechanism in this embodiment of the invention. The left image is the original rendering image with geometric defects caused by the limited viewpoint, the middle image is the binarized repair mask generated by the system, and the right image is the final rendering result after generative closed-loop repair. Detailed Implementation

[0072] Example 1:

[0073] like Figure 1 As shown, this invention provides a structured 3D reconstruction and generative perspective restoration method for library scenarios, specifically including the following steps:

[0074] Step 1: As Figure 1 and Figure 2 As shown, the acquired monocular video stream is adaptively segmented into temporal blocks using a large visual geometric model, local dense point maps and relative poses are regressed, and the coordinate systems of each sub-map are aligned through global pose map optimization. In specific implementation, firstly, monocular long video sequences of large scenes such as libraries are acquired, and the sliding window mechanism is used to divide them into overlapping local sub-windows. A relative scale factor s is introduced in the local coordinate system to extend the optimization space to the similarity transformation group Sim(3) to solve the scale blur problem of monocular cameras. Subsequently, constraint edges are established at the overlapping shared frames of adjacent sub-windows to construct the global pose map optimization objective function. By minimizing the objective function, each local sub-window is aligned to a unified world coordinate system, effectively overcoming the trajectory breakage and drift problems that are prone to occur in traditional methods.

[0075] Step 2: Construct an anchor-driven sliding window joint optimization mechanism, incorporate camera extrinsic parameters into the differential rendering pipeline, and alternately optimize the 3D Gaussian properties and camera trajectory by combining monocular depth prior. Specifically, a 3D Gaussian sputtering model is constructed using the dense point cloud obtained in Step 1 as the initial value, and the scene is represented as a discrete 3D Gaussian primitive containing mean, covariance, opacity, and spherical harmonic function coefficients. During the differentiable rasterization rendering process, high-confidence keyframes are selected as "anchor frames" and their poses are frozen, allowing "non-anchor frames" to be fine-tuned in the Lie algebra se(3) space. In order to constrain the trajectory of non-anchor frames, a trajectory smoothing loss based on a large model prior is introduced. Simultaneously, the dense depth map output by the monocular depth estimation network is extracted as the pseudo-ground value, the optimal affine parameters are dynamically calculated, and a scale-displacement invariant depth regularization loss is constructed. Direct surface constraints are applied to the geometric position of the 3D Gaussian to eliminate "floating artifacts" in weakly textured areas.

[0076] Step 3: As Figure 3As shown, an octree data structure is used to spatially index 3D Gaussian primitives. Occlusion culling and redundant point pruning are performed in conjunction with view frustum visibility analysis, and a multi-level-of-detail (LOD) attribute aggregation model is constructed. First, the spatial bounding boxes of all Gaussian primitives are calculated to construct the root node, and the Morton codes generated based on the primitive mean coordinates are assigned to the leaf nodes of the octree. A moment matching method based on opacity weighting is used to calculate the "super Gaussian" attributes of non-leaf nodes from the bottom up to support LOD rendering. During the rendering inference phase, view frustum culling is performed, and nodes located beyond the depth of already rendered pixels or with saturated opacity are skipped early. Furthermore, in the later stages of offline training, redundant "dead Gaussians" encased within objects are physically removed by statistically analyzing the highest opacity contribution of primitives in the virtual viewpoint, thereby maximally compressing the model size and reducing GPU memory usage.

[0077] Step 4: As Figure 4 As shown, low-confidence regions are actively rendered to expose defects by applying pose perturbations. The rendered depth map is used as a strong geometric constraint to guide the diffusion model in generating repair textures, and this texture is fed back as a pseudo-ground value to a 3D Gaussian field for incremental training. Random small perturbations are applied in the keyframe tangent space of the training trajectory to generate virtual observation poses. A cumulative opacity map is rendered from this perspective, and when it falls below a set threshold, geometric defects are identified and a repair mask is generated. The rendering depth map of the virtual viewpoint is extracted and mapped to a depth control signal through a zero-convolutional layer. Along with text prompts, this is injected into the inverse denoising process of the latent diffusion model, forcing the generated texture to conform to the edges of the 3D physical fault. Finally, the generated repaired image is treated as a pseudo-ground value and re-injected into the training data stream to construct a mask-based spatially weighted loss function. By utilizing the adaptive density control mechanism of Gaussian sputtering, primitives in the void region are driven to split or clone, thereby achieving three-dimensional consistent texture evolution.

[0078] This embodiment achieves significant technical effects through the coordinated processing of steps 1 to 4 described above. The specific details are as follows:

[0079] First, such as Figure 5 As shown, for the original monocular long video stream with corridor features and weak texture areas in a library, this invention successfully overcomes monocular scale drift by dividing overlapping sequence frames through the sliding window mechanism in step 1 and performing global pose graph optimization. The aligned and reconstructed 3D point cloud spatial structure is shown below. Figure 6 As shown, the spatial distribution from multiple perspectives reveals that the corridor and bookshelves maintain a strict linear and parallel relationship in the global coordinate system, without any trajectory bending or breakage caused by long sequences, demonstrating the robustness of this method in global alignment at a large scene scale.

[0080] Secondly, for weak texture areas such as the sides of the bookshelf, the depth regularization loss introduced in step 2 effectively eliminates the "floating artifacts" common in traditional methods, making the Gaussian elements accurately attached to the physical surface.

[0081] Finally, as Figure 7 As shown, for geometrically missing areas caused by physical occlusion and limited viewing angle during video acquisition, this invention achieves accurate repair through active defect detection and closed-loop feedback training in step 4. Specifically, Figure 7 The left image shows the original image rendered from a virtual perspective, with visual artifacts and geometric defects (such as blurred discontinuities); Figure 7 The image in the middle shows the binarized repair mask automatically calculated and generated by the system based on this (corresponding to the formula in the image). ; Figure 7 The right image shows the final rendering result obtained from the same viewpoint after injecting the generated repaired image as a pseudo-ground value into the Gaussian field increment evolution. Comparing the left and right images, it can be seen that the repaired area not only completes the missing 3D geometry, but the newly generated book and bookshelf textures also achieve seamless integration with the surrounding original environment in 3D consistency, demonstrating the superior performance of this invention in handling blind spots in complex indoor scenes.

[0082] Step 1 is as follows:

[0083] Step 1-1: Divide the long video sequence corresponding to the monocular video stream into a series of partially overlapping local sub-windows. The partial overlap means that there are a preset number of shared frames between adjacent local sub-windows. Within each sub-window, two frames with temporal or spatial co-viewing relationships are defined as image pairs. The visual geometric large model VGGT is used to directly regress the dense point map and inter-frame relative transformation of all image pairs within the sub-window.

[0084] Steps 1-2 introduce a relative scale factor s in the local coordinate system to extend the optimization space to the similarity transformation group Sim(3), for the relative pose between any two frames. The optimal local pose set within the sub-window is obtained by minimizing the reprojection error of the dense point map through bundle adjustment.

[0085] Steps 1-3: Establish constraint edges in the overlapping area of ​​adjacent sub-windows, construct a global pose graph, and define the objective function for optimizing the global pose graph. To solve for the optimal similarity transformation matrix The local sub-windows are aligned to a unified world coordinate system, and the dense point maps of each local sub-window are merged to obtain a globally consistent camera trajectory and a globally dense point cloud.

[0086]

[0087] in, and Let be the global similarity transformation matrices from the k-th and (k+1)-th sub-windows to the world coordinate system. and For overlapping shared frames, the local poses within their respective local sub-windows are used. For the logarithmic mapping operator from the Sim(3) group to its corresponding Lie algebra sim(3); The square of the Mahalanobis distance. This is the covariance matrix calculated based on the confidence level of local reconstruction.

[0088] The anchor-driven sliding window joint optimization mechanism described in step 2 is as follows:

[0089] Step 2-1: Using the dense point cloud obtained after alignment in Step 1 as the initial value, construct a three-dimensional Gaussian sputtering model. The scene is represented as a set of discrete three-dimensional Gaussian primitives.

[0090] Step 2-2: Project the above Gaussian primitives onto the two-dimensional image plane and perform differentiable rendering using a differentiable rasterizer; incorporate the camera extrinsic parameters into the differentiable rendering pipeline and perform joint optimization for small pose errors through gradient backpropagation;

[0091] Steps 2-3 involve alternating optimization of 3D Gaussian properties and camera trajectory using monocular depth priors.

[0092] The optimization described in step 2-2 specifically includes:

[0093] Step 2-2-1: Treat the camera extrinsic parameters as learnable parameters and incorporate them into the aforementioned differentiable rendering pipeline, utilizing Lie algebras. The six-DOF pose of the camera, which includes three-dimensional rotation and three-dimensional translation, is mapped into an unconstrained six-dimensional vector, and the pose increment in the form of the six-dimensional vector is calculated parametrically.

[0094] Step 2-2-2: Select high-confidence keyframes from the video sequence as anchor frames. During the optimization process, impose strict constraints or freeze the pose of the anchor frames, allowing non-anchor frames to be fine-tuned in the Lie algebra space. In addition, introduce relative pose constraints between adjacent frames to construct a trajectory smoothing loss function. Specifically:

[0095]

[0096] in, and The poses to be optimized are the poses of two adjacent frames. The initial values ​​of the relative transformation are predicted by the large visual geometry model.

[0097] The step 2-3, which involves alternately optimizing the 3D Gaussian properties and camera trajectory using monocular depth prior, specifically involves:

[0098] Step 2-3-1: Obtain the depth map generated by the monocular depth estimation network for each frame of image. As a pseudo-truth value;

[0099] Step 2-3-2: Using a 3D Gaussian sputtering differentiable rasterization pipeline, the desired rendering depth value for each pixel is cumulatively calculated using the alpha blending formula. The calculation formula is:

[0100]

[0101] in, Let the depth of the k-th Gaussian element along the z-axis in the camera coordinate system be denoted as . Let k be the opacity of the k-th Gaussian element;

[0102] Step 2-3-3, Combine Depth Map With the expected rendering depth value Calculate the optimal affine parameters The optimal affine parameters for each frame are dynamically solved using the least squares method. To eliminate scale and offset uncertainties in monocular depth, a scale-displacement invariant depth regularization loss function is constructed. Specifically:

[0103]

[0104] in, This represents the valid pixel region in the image, and its definition includes the following two filtering criteria:

[0105] 1. Effective depth range: Excludes depth values ​​that exceed the preset physical measurement range. Abnormal noise, among which and Designed according to the actual physical dimensions of the library's interior;

[0106] 2. Rasterization visibility: Only includes the cumulative opacity at the current viewing angle. The number of pixels exceeding a preset threshold is used to ensure that the loss function only applies to the solid surface already covered by Gaussian elements, thus avoiding the introduction of incorrect depth constraints by background hole areas.

[0107] Step 3, which describes using an octree data structure to perform spatial indexing and attribute aggregation on 3D Gaussian primitives, specifically involves:

[0108] Step 3-1: Obtain the extreme values ​​of the 3D coordinates of all Gaussian elements, construct a global axial bounding box that encloses the entire indoor scene as the root node of the octree, and then calculate the mean center point of the Gaussian elements. Discretization and interleaving operations are performed to generate unique Morton codes, and all primitives are assigned to the leaf nodes of the octree according to the Morton codes.

[0109] Step 3-2: The properties of the super Gaussian in the non-leaf nodes of the octree are calculated using the moment matching method based on opacity weighting to construct a multi-level-of-details model.

[0110] Step 3-3: During rendering, maintain the depth buffer based on the view frustum visibility, remove nodes that are completely behind the depth of the rendered pixels or nodes whose cumulative opacity has reached saturation, and physically prune redundant points that are globally invisible.

[0111] Step 4, which describes actively rendering low-confidence regions by applying pose perturbations to expose defects, specifically involves:

[0112] Step 4-1-1: Using the camera trajectory obtained from the joint optimization in Step 2 as the original training trajectory, select the keyframe pose in this trajectory as the center, and calculate its corresponding Lie algebra. A set of virtual observation poses is generated by applying random, minute perturbations within the space.

[0113] Step 4-1-2: Under the virtual observation pose, render the RGB image and cumulative opacity using the three-dimensional Gaussian sputtering model constructed in step 2. ;

[0114] Step 4-1-3, set the opacity threshold. When the cumulative opacity is below the threshold, a geometric defect is determined to exist, and a binarized repair mask is generated. Specifically:

[0115]

[0116] in, This indicates the area to be repaired that needs to be generated.

[0117] Step 4, which uses the rendered depth map as a strong geometric constraint to guide the diffusion model in generating the repair texture, specifically involves:

[0118] Step 4-2-1: Create a depth map under the virtual observation pose. Encoded as depth control signals through zero-convolutional layers. ;

[0119] Step 4-2-5, transfer the depth control signal Along with scene prompt text Together with the inverse denoising process of the latent diffusion model, the denoising objective function is modified as follows:

[0120]

[0121] in, The potential noise representation for time step t, It is real Gaussian noise. The noise component predicted by the denoising network.

[0122] Step 4, which involves feeding the pseudo-true value back to the three-dimensional Gaussian field for incremental training, specifically involves:

[0123] Step 4-3-1, use the repaired image generated by the diffusion model. Together with the corresponding virtual pose, they are treated as pseudo-ground values ​​and injected into the original training data stream composed of the real video frame images and their corresponding camera poses acquired in step 1;

[0124] Step 4-3-2: Incremental fine-tuning is performed based on the new training data to construct a confidence-based weighted loss update strategy. The total loss function of this incremental fine-tuning is defined as the weighted sum of the loss on the real data and the loss on the generated data.

[0125]

[0126] in, These are the global weighting coefficients. It is a pixel-level spatial weight map used to limit the optimizer to focus primarily on the region within the repair mask;

[0127] Step 4-3-3 utilizes the adaptive density control mechanism of Gaussian sputtering to drive the Gaussian elements within the mask region to split or clone, thereby completing incremental geometry and texture evolution.

[0128] Example 2:

[0129] Based on the same inventive concept, this embodiment provides a structured 3D reconstruction and generative perspective restoration system for library scenarios.

[0130] The system comprises four core modules that execute the above-described methods and steps:

[0131] Temporal Blocking and Global Alignment Module: Adaptive temporal blocking is performed using the built-in visual geometric model, and the output of globally consistent dense point cloud and camera trajectory is optimized through the global pose graph in Sim(3) space.

[0132] Joint Optimization and Constraints Module: Responsible for initializing 3D Gaussian properties and attaching camera Lie algebra extrinsic parameters to the rendering computation graph. It alternately calculates photometric loss, trajectory smoothing loss, and depth regularization loss to correct drift and eliminate artifacts.

[0133] Spatial Indexing and Hierarchical Module: Constructs an octree based on Morton codes, generates super Gaussian nodes with multiple levels of detail through a moment matching algorithm, and implements real-time culling and offline physical pruning in conjunction with the view frustum depth buffer.

[0134] Closed-loop feedback and repair module: The missing mask is calculated by perturbation sampling virtual pose, and the potential diffusion model is guided by the depth map to generate a repair texture without perspective distortion, and it is encapsulated as a pseudo-true value to trigger the incremental proliferation of the Gaussian field.

[0135] This invention provides a structured 3D reconstruction and generative perspective restoration method for library scenarios. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A structured 3D reconstruction and generative perspective restoration method for library scenarios, characterized in that, Includes the following steps: Step 1: Process the acquired monocular video stream to obtain the dense point map of the target to be reconstructed; Step 2: Construct a three-dimensional Gaussian sputtering model based on the dense point cloud through an anchor-driven sliding window joint optimization mechanism; Step 3: Use the octree data structure to perform spatial indexing of the 3D Gaussian primitives, combine the view frustum visibility analysis to perform occlusion culling and redundant point pruning, and construct a multi-detail level attribute aggregation model. Step 4: Actively render low-confidence regions by applying pose perturbation to expose defects. Use the rendered depth map as a strong geometric constraint to guide the diffusion model to generate repair textures, and feed it back as pseudo-ground values ​​to the three-dimensional Gaussian field for incremental training.

2. The structured 3D reconstruction and generative perspective restoration method for library scenarios according to claim 1, characterized in that, Step 1 is as follows: Step 1-1: Divide the long video sequence corresponding to the monocular video stream into a series of partially overlapping local sub-windows. Within each sub-window, define two frames that are temporally adjacent or spatially co-viewing as image pairs. Use the large visual geometry model to directly regress the dense point map and inter-frame relative transformation of all image pairs within the sub-window. Steps 1-2 introduce a relative scale factor s in the local coordinate system to extend the optimization space to the similarity transformation group Sim(3), for the relative pose between any two frames. The optimal local pose set within the sub-window is obtained by minimizing the reprojection error of the dense point map through bundle adjustment. Steps 1-3: Establish constraint edges in the overlapping area of ​​adjacent sub-windows, construct a global pose graph, and define the objective function for optimizing the global pose graph. To solve for the optimal similarity transformation matrix The local sub-windows are aligned to a unified world coordinate system, and the dense point maps of each local sub-window are merged to obtain a globally consistent camera trajectory and a globally dense point cloud.

3. The structured 3D reconstruction and generative perspective restoration method for library scenarios according to claim 2, characterized in that, The anchor-driven sliding window joint optimization mechanism described in step 2 is as follows: Step 2-1: Using the dense point cloud obtained after alignment in Step 1 as the initial value, construct a three-dimensional Gaussian sputtering model. The scene is represented as a set of discrete three-dimensional Gaussian primitives. Step 2-2: Project the above Gaussian primitives onto the two-dimensional image plane and perform differentiable rendering using a differentiable rasterizer; incorporate the camera extrinsic parameters into the differentiable rendering pipeline and perform joint optimization for small pose errors through gradient backpropagation; Steps 2-3 involve alternating optimization of 3D Gaussian properties and camera trajectory using monocular depth priors.

4. The structured 3D reconstruction and generative perspective restoration method for library scenarios according to claim 3, characterized in that, The optimization described in step 2-2 specifically includes: Step 2-2-1: Treat the camera extrinsic parameters as learnable parameters and incorporate them into the differentiable rendering pipeline. Map the six-DOF pose of the camera, which includes three-dimensional rotation and three-dimensional translation, into an unconstrained six-dimensional vector, and perform parameterized calculation on the pose increment in the form of the six-dimensional vector. Step 2-2-2: Select high-confidence keyframes from the video sequence as anchor frames. During the optimization process, impose strict constraints or freeze the pose of the anchor frames, allowing non-anchor frames to be fine-tuned in the Lie algebra space. In addition, introduce relative pose constraints between adjacent frames to construct a trajectory smoothing loss function. .

5. The structured 3D reconstruction and generative viewpoint restoration method for library scenarios according to claim 1, characterized in that, The step 2-3, which involves alternately optimizing the 3D Gaussian properties and camera trajectory using monocular depth prior, specifically involves: Step 2-3-1: Obtain the depth map generated by the monocular depth estimation network for each frame of image. As a pseudo-truth value; Step 2-3-2: Using a 3D Gaussian sputtering differentiable rasterization pipeline, the desired rendering depth value for each pixel is cumulatively calculated using the alpha blending formula. ; Step 2-3-3, Combine Depth Map With the expected rendering depth value Calculate the optimal affine parameters The optimal affine parameters for each frame are dynamically solved using the least squares method. To eliminate scale and offset uncertainties in monocular depth, a scale-displacement invariant depth regularization loss function is constructed. Specifically: in, This represents the valid pixel area in the image.

6. The structured 3D reconstruction and generative viewpoint restoration method for library scenarios according to claim 1, characterized in that, Step 3, which describes using an octree data structure to perform spatial indexing and attribute aggregation on 3D Gaussian primitives, specifically involves: Step 3-1: Obtain the extreme values ​​of the 3D coordinates of all Gaussian elements, construct a global axial bounding box that encloses the entire indoor scene as the root node of the octree, and then calculate the mean center point of the Gaussian elements. Discretization and interleaving operations are performed to generate unique Morton codes, and all primitives are assigned to the leaf nodes of the octree according to the Morton codes. Step 3-2: The properties of the super Gaussian in the non-leaf nodes of the octree are calculated using the moment matching method based on opacity weighting to construct a multi-level-of-details model. Step 3-3: During rendering, maintain the depth buffer based on the view frustum visibility, remove nodes that are completely behind the depth of the rendered pixels or nodes whose cumulative opacity has reached saturation, and physically prune redundant points that are globally invisible.

7. The structured 3D reconstruction and generative viewpoint restoration method for library scenarios according to claim 1, characterized in that, Step 4, which describes actively rendering low-confidence regions by applying pose perturbations to expose defects, specifically involves: Step 4-1-1: Using the camera trajectory obtained from the joint optimization in Step 2 as the original training trajectory, select the keyframe pose in this trajectory as the center, and calculate its corresponding Lie algebra. A set of virtual observation poses is generated by applying random, minute perturbations within the space. Step 4-1-2: Under the virtual observation pose, render the RGB image and cumulative opacity using the three-dimensional Gaussian sputtering model constructed in step 2. ; Step 4-1-3, set the opacity threshold. When the cumulative opacity is below the threshold, a geometric defect is determined to exist, and a binarized repair mask is generated. Specifically: in, This indicates the area to be repaired that needs to be generated.

8. A structured 3D reconstruction and generative viewpoint restoration method for library scenarios according to claim 6, characterized in that, Step 4, which uses the rendered depth map as a strong geometric constraint to guide the diffusion model in generating the repair texture, specifically involves: Step 4-2-1: Create a depth map under the virtual observation pose. Encoded as depth control signals through zero-convolutional layers. ; Step 4-2-5, transfer the depth control signal Along with scene prompt text Together with the inverse denoising process of the latent diffusion model, the denoising objective function is modified as follows: in, The potential noise representation for time step t, It is real Gaussian noise. The noise component predicted by the denoising network.

9. A structured 3D reconstruction and generative perspective restoration method for library scenarios according to claim 8, characterized in that, Step 4, which involves feeding the pseudo-true value back to the three-dimensional Gaussian field for incremental training, specifically involves: Step 4-3-1, use the repaired image generated by the diffusion model. Together with the corresponding virtual pose, they are treated as pseudo-ground values ​​and injected into the original training data stream composed of the real video frame images and their corresponding camera poses acquired in step 1; Step 4-3-2: Perform incremental fine-tuning based on the new training data to construct a confidence-based weighted loss update strategy; Step 4-3-3 utilizes the adaptive density control mechanism of Gaussian sputtering to drive the Gaussian elements within the mask region to split or clone, thereby completing incremental geometry and texture evolution.

10. A structured 3D reconstruction and generative viewpoint restoration method for library scenarios according to claim 9, characterized in that, In step 4-3-2, the total loss function corresponding to the incremental fine-tuning of the weighted loss update strategy is defined as the weighted sum of the actual data loss and the generated data loss: in, These are the global weighting coefficients. It is a pixel-level spatial weight map used to limit the optimizer to focus primarily on the region within the repair mask.

Images