Online indoor intrinsic decomposition reconstruction framework based on two-dimensional Gaussian

Abstract

The invention discloses an online indoor intrinsic decomposition reconstruction framework based on two-dimensional Gaussian. The method comprises the steps that an intrinsic two-dimensional Gaussian primitive is innovatively put forward, and the potential of rapid large-scale indoor scene material and illumination decomposition of intrinsic decomposition in Gaussian splash neural inverse rendering is displayed; the front end of the frame receives an RGBD video stream, and the camera pose, the two-dimensional Gaussian covariance, the surface curvature and other representations are obtained through normal iteration nearest point positioning; the NICP-2DGS acceleration method is innovatively provided at the rear end of the framework, and the covariance and covariance scale alignment technology at the front end is utilized to initialize the intrinsic two-dimensional Gaussian, so that maximum collaborative acceleration is formed. Finally, positioning, mapping and intrinsic decomposition tasks are synchronously completed on line at the speed exceeding 12 FPS, excellent performance is shown in the aspects of precision, efficiency and practicability, the urgent requirements of real-time indoor scene editing and inverse rendering are met, and wide application prospects are shown.

Description

Technical Field

[0001] This invention belongs to the field of image 3D reconstruction technology, specifically involving online front-end localization, intrinsic 2D Gaussian primitive representation, and back-end online intrinsic 2D Gaussian neural rendering and mapping. Background Technology

[0002] Intrinsic Image Decomposition (IID) is theoretically based on Retinex theory, which posits that an object's appearance is determined by its inherent reflective properties and incident illumination. The rapid development of IID has facilitated a deeper understanding of scene interaction mechanisms by enabling the extraction of material, geometric, and lighting information from observations of scene objects. Most current mainstream IID models decouple the scene into two components: Albedo / Reflectance and Shading. Albedo reflects the object's inherent material properties, while Shading reflects the interaction between ambient light and the object's geometry.

[0003] Existing intrinsic eigenvalue decomposition (IoV) models are mainly divided into optimization-based IoV and learning-based IoV. Optimization methods often rely on physical priors abstracted from people's summaries of objective laws, using loss functions to optimize the separation of illumination and material information. Optimization-based IoV methods are highly efficient, but often fail when the prior conditions are not met. Learning methods use neural networks (such as CNNs and Transformers) or generative models (such as GANs and Diffusion) to learn the mapping relationship between images, reflectivity, and illumination, but require a large amount of ground-value data for training and have poor generalization ability.

[0004] Neural rendering and neural inverse rendering. Neural rendering provides a differentiable rendering path from 3D primitives to images, combined with optimizations to achieve inverse 3D reconstruction. Neural rendering originated from Neural Radiance Fields (Nerf) and reached its peak with 3D Gaussian Splatting (3DGS). Unlike the inefficient Nerf, the highly parallel splatting rendering framework of 3DGS has brought the possibility of real-time online reconstruction.

[0005] Inverse rendering focuses on extracting spatial relationships and semantic information of entities from a scene. Mainstream 3D decomposition methods use neural inverse rendering frameworks, based on Nerf and 3DGS, to explicitly establish the inverse process of the rendering equation to obtain physical relationships. They use physically-based rendering (PBR) to decouple the interaction between lighting and surface materials. The large amount of computation makes the construction of online systems difficult. These inverse rendering methods perform well in object-level scenes, but often fail to achieve ideal results in indoor scenes.

[0006] 2DGS aims to address the limitations of 3DGS in terms of geometric accuracy. By introducing 2D surface primitives to fit object surfaces and combining them with a novel Gaussian ray intersection method, it achieves a significant improvement in quality thanks to this more accurate geometric representation.

[0007] Dense Simultaneous Localization and Mapping (DSLAM). The goal of DSLAM using dense representations is to build dense map representations, which, unlike SLAM that focuses on front-end localization, provides a visual, high-precision map.

[0008] Early 3DGS focused on batch data input and offline processing, which spurred the combination of Gaussian splashing and Simultaneous Localization and Mapping (SLAM) to meet the localization and reconstruction needs when using arbitrary poseless streaming sequential frames as input. Existing methods typically use Gaussian-rendered images and photometric loss for localization and camera parameter calculation at the front end, and optimize mapping at the back end based on various strategies such as incremental or hierarchical approaches, but the problem of slow tracking speed still exists.

[0009] Existing technologies and their disadvantages: With the development of commercial RGBD cameras, obtaining depth maps has become simple, but existing models rarely utilize the relationship between depth maps and point clouds.

[0010] Existing dense simultaneous localization and mapping methods use dense photometric loss, which imposes a severe optimization burden on online real-time optimization.

[0011] Existing intrinsic decomposition methods are mostly limited to the decomposition of reflection and shading for a single view, and the limited research on multi-view applications also focuses on offline rendering. There is a lack of relevant models and applications for streaming multi-view input, which has significant practical implications for real-time interaction.

[0012] To obtain scene lighting and material information, existing physically based rendering decomposition methods are often limited to object-level objects and cannot be extended to complex indoor scenes. Moreover, to our knowledge, no model can decompose and obtain 3D scene reflectivity and lighting information in real time to meet the real-time interactive requirements of Augmented Reality (AR). Summary of the Invention

[0013] The core problem this invention aims to solve is to construct an online intrinsic decomposition framework. This framework enables real-time online processing of RGBD video stream frames, simultaneously performing three tasks: localization, mapping, and intrinsic decomposition. The final result is the camera pose obtained from front-end camera localization and the map based on the proposed intrinsic Gaussian elements in the back-end. The results, during real-time reconstruction and decomposition or after model optimization, can be edited and modified as applications of the framework.

[0014] The main problems that this invention needs to solve can be summarized as follows: Question Q1: For RGBD stream input of real-world indoor scenes, what primitives should be used to represent and optimize the backend map representation? Traditional 3DGS primitive representations suffer from several drawbacks. First, they lack geometric accuracy, as the actual depth of the scene may not match the depth of the 3D Gaussian primitives. Second, 3D Gaussian primitives exhibit poor multi-view consistency due to the ambiguity of their viewpoint-spherical harmonic color representation under 3D primitive representation. Second, 2D Gaussian functions offer significant improvements in geometric accuracy through 2D Gaussian rendering techniques and optimization of the consistency loss between primitive depth and splash intersection depth, providing a stable constraint on the spatial geometric distribution of Gaussians. Third, their 2D color representation exhibits better multi-view consistency, reducing overfitting to the input viewpoint. It should be noted that, to date, there is no suitable solution for the online inverse rendering problem that this invention aims to address.

[0015] Question Q2: How to solve the efficiency problem of online inverse rendering? Most current inverse rendering methods are limited to offline rendering. The first difference between online and offline systems is that offline rendering is generally based on an inverse rendering framework, explicitly establishing the inverse process of the rendering equation to obtain physical relationships. It uses Physically Based Rendering (PBR) to decouple the interaction between lighting and surface materials. The large amount of computation makes building an online system difficult. Second, with the development of commercial RGBD cameras, obtaining depth maps has become simpler, but existing models rarely utilize the relationship between depth maps and point clouds. Furthermore, another significant point is that the dense photometric loss used in existing inverse rendering methods places a heavy optimization burden on real-time online optimization. These three points pose a huge efficiency obstacle to a real-time online inverse rendering system and are problems that urgently need to be solved.

[0016] Question Q3: How to build a front-end and back-end framework to achieve online indoor intrinsic decomposition and reconstruction? The difficulties of online systems are not limited to efficiency. The second difference between online and offline systems lies in their different frameworks. Offline processing generally focuses on phased batch processing, with each phase having a sequential relationship. The input of the later phase generally depends on the previous phase. This means that the system can only start running after all inputs are known, which cannot match the real-time interactive requirements of online systems that require "input, processing, and feedback simultaneously."

[0017] This invention addresses the main problems by proposing the following objectives: Objective D1: Addressing problem Q1, this invention proposes an intrinsic Gaussian unit representation for an online indoor intrinsic decomposition and reconstruction framework, referred to as the intrinsic Gaussian unit. This representation, combined with existing splatting rendering techniques, provides efficient map representation, optimization, and storage. The aim is to efficiently achieve splatting rendering and scene representation of various map elements, including RGB values, depth, normals, shading maps, and reflection maps.

[0018] Objective D2: Addressing problem Q2, this invention proposes a NICP-2DGS covariance sharing optimization acceleration method, termed NICP-2DGS acceleration. This acceleration method leverages the spatial scale uniformity of the 2D Gaussian covariance used in NICP front-end localization and the 2DGS covariance used in the back-end. A scale alignment method enables bidirectional sharing of their covariances, achieving a mutually reinforcing synergistic acceleration effect. The goal is to provide an optimization acceleration effect based on covariance sharing when the system front-end uses NICP localization and the back-end uses 2DGS representation.

[0019] Objective D3: In response to question Q3, this invention proposes an online indoor intrinsic decomposition and reconstruction framework based on two-dimensional Gaussian. Unlike offline frameworks, the purpose of this framework is to unify the innovations proposed in objectives D1 and D2, construct an online front-end and back-end processing framework under streaming RGBD input, and advance inverse rendering towards an online system.

[0020] To address the aforementioned technical problems and in accordance with the main objectives of this invention, the technical solution adopted by this invention is as follows: Firstly, addressing the previously mentioned problem Q1 and objective D1, the present invention provides a Gaussian representation, called intrinsic Gaussian, for fast intrinsic representation and intrinsic decomposition of multi-view scenes. The specific technical solution includes the following steps: Step S1: Receive scene information from the front end. This includes the front end's localization pose, the depth and RGB values ​​of the scene input, and the 2D covariance of the point cloud generated during front end localization.

[0021] Step S2: Initialize the intrinsic Gaussian using scene information. This includes initializing the intrinsic Gaussian's position, 2D covariance, normal, spherical harmonic coefficients, reflectivity, illumination, residuals, and other elements.

[0022] Step S3: Intrinsic Gaussian rasterization and Splatting render RGB images, depth, normals, shading images, reflection images, and other two-dimensional images to represent scene representation information from the current viewpoint.

[0023] Step S4: Use loss optimization to render the graph, and combine differentiable rendering backpropagation loss to optimize and decompose scene attributes.

[0024] Step S5, frame transition, switch from the current frame to the next frame, corresponding to the time flow, repeat step S1 until the input stream ends.

[0025] The ingenuity of intrinsic Gaussian primitive representation lies in bringing intrinsic properties into 2DGS, combining the advantages of 2DGS in geometric accuracy and multi-view color representation. The intrinsic properties in Gaussian representation can be backpropagated to the decomposition loss by means of Gaussian splashing rasterization, and scene decomposition and inverse rendering can be performed under any intrinsic property initialization.

[0026] Intrinsic Gaussian representation offers two main advantages: First, it makes the model insensitive to initial intrinsic attribute values, thus ensuring the possibility of convergence; second, it makes the model not limited to specific scenarios, ensuring its good adaptability in various scenarios.

[0027] Secondly, regarding the previously mentioned problem Q2 and objective D2, the technical solution of this invention provides a NICP-2DGS covariance sharing optimization acceleration method, referred to as NICP-2DGS acceleration, for online optimization of intrinsic Gaussian maps. Its specific technical solution includes the following steps: Step S1: The front end inputs the current RGBD frame into the inverse projection to obtain the 3D point cloud.

[0028] Step S2: Extraction of statistical information of the front-end surface.

[0029] Step S3: The front end finds the corresponding relationship through projection.

[0030] Step S4: The front end determines the transformation based on the corresponding relationship.

[0031] Step S5: The backend obtains the frontend surface information.

[0032] Step S6: The backend initializes the intrinsic Gaussian based on the surface information from the frontend.

[0033] Step S7: Optimize the map on the backend.

[0034] In step S8, the backend transmits the optimized Gaussian point attributes to the frontend and adds the new points to the world coordinate system.

[0035] Step S9, proceed to step S1, read the next frame, until the input stream terminates.

[0036] A key observation of NICP-2DGS acceleration is that, under Lambert's scene assumptions, most existing depth measurement methods are based on Time of Flight (ToF)-like methods, where depth is essentially a sampling point on the object's surface, rather than penetrating deep into the object's interior. Therefore, a 2D Gaussian surface, similar to a patch, can effectively fit the object's surface and simulate scene geometry, thus allowing the backend to benefit from using a 2D Gaussian surface in the depth map.

[0037] The ingenuity of NICP-2DGS acceleration lies in two key aspects. First, the front-end localization is no longer based on dense photometric loss, but rather on a traditional iterative nearest-point method that fully integrates geometric input. This separates the front-end from the back-end rasterization, freeing the front-end from reliance on dense rasterization and enabling rapid localization. Second, and most importantly, NICP registration generates secondary assets such as point cloud normals and 2D covariance. These point cloud attributes are essentially free for the back-end, requiring no additional computation to obtain optimal initialization values. Using these initial values ​​during the back-end's intrinsic Gaussian initialization allows for rapid convergence. Conversely, the back-end's optimized Gaussian attribute values ​​can also be passed to the front-end. Using the optimized Gaussian attributes from the back-end's graph loss for front-end localization allows the back-end to incorporate joint loss optimization based on various information, influencing front-end localization and ensuring the front-end is no longer isolated from the back-end.

[0038] Thirdly, addressing the previously mentioned problem Q3 and objective D3, the technical solution of this invention provides an online indoor intrinsic decomposition and reconstruction framework based on two-dimensional Gaussian. This framework combines the first aspect of intrinsic Gaussian elements with the second aspect of the NICP-2DGS covariance sharing optimization acceleration method to construct an online indoor intrinsic decomposition and reconstruction framework. This framework completes the positioning, mapping, and intrinsic decomposition tasks online synchronously at a rate of 12 FPS, demonstrating superior performance in terms of accuracy, efficiency, and practicality. Its specific technical solution includes the following steps: Step S1: Initialize the front end, receive the first frame, and inversely project the current RGBD frame as the world coordinates.

[0039] Step S2: Run the NICP-2DGS accelerated front end.

[0040] Step S3: Initialize new intrinsic Gaussians.

[0041] Step S4: Run the NICP-2DGS accelerated backend.

[0042] Step S5: Accept the next frame and jump to step S2 until the input stream terminates.

[0043] Compared with the prior art, the present invention has at least the following technical effects: Effect E1 is a fast intrinsic representation and decomposition primitive for multi-view scenes: intrinsic Gaussian primitives. This representation, combined with existing splatting rendering techniques, enables differentiable rendering of intrinsic properties, providing efficient map representation, optimization, and storage. Intrinsic Gaussian differs from previous inverse rendering frameworks, enabling online implementation of localization, mapping, and intrinsic decomposition. It guarantees a wide convergence domain and high applicability of the model.

[0044] The E2 result implements a fast front-end and back-end collaborative acceleration optimization method: NICP-2DGS acceleration. Localization is separated from rasterization, divided into front-end and back-end. The front-end maintains high-speed NICP localization, while the back-end uses intrinsic Gaussian elements from technical solution 1. Attribute sharing between the front-end and back-end is maintained, allowing the back-end to benefit from the excellent initial attribute values ​​of the front-end, and the front-end to benefit from the attribute values ​​optimized by the back-end loss. Both mutually promote each other, accelerating optimization and doubling the model convergence speed.

[0045] Effect E3 is a 2D Gaussian online indoor intrinsic decomposition and reconstruction framework. This framework performs online synchronous localization, mapping, and intrinsic decomposition tasks at over 12 FPS on a commercial 3090 graphics card, demonstrating superior performance in accuracy, efficiency, and practicality.

[0046] The E4 effect meets the urgent needs for real-time indoor scene editing and interaction, demonstrating broad application prospects. With features including real-time editing of reflectivity and inherent color, and interactive editing of lighting intensity, it holds immense potential in real-time AR interactive scenarios. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the online indoor intrinsic decomposition and reconstruction framework based on two-dimensional Gaussian shown in this invention; Figure 2 This is an overall schematic diagram of the online indoor intrinsic decomposition and reconstruction framework based on two-dimensional Gaussian shown in this embodiment; Figure 3 This is the visualization comparison result of the dataset Replica shown in this embodiment; Figure 4 This is the result of relighting editing shown in this embodiment; Figure 5 This is the recoloring editing result shown in this embodiment; Detailed Implementation Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0048] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0049] The online intrinsic decomposition and reconstruction method of two-dimensional Gaussians provided by the present invention will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0050] Please see Figure 1 ,like Figure 1 As shown, the online indoor eigenvalue decomposition and reconstruction framework based on two-dimensional Gaussian includes the following steps: Step S1 System initialization, such as Figure 2 The left column of the overall diagram shows a schematic of system initialization. The entire system is initialized. The first frame is received, and the current RGBD frame is inversely projected as world coordinates.

[0051] Specifically as follows: Step S1.1 Initialize the entire system, including creating the mapper class for backend mapping, the tracker class for frontend localization, the dataset class for dataset loading, and the viewer class for visualization.

[0052] The hyperparameters used in subsequent systems are read. These hyperparameters include several major categories such as Gaussian scene parameters, SLAM pipeline parameters, NICP registration parameters, and intrinsic decomposition loss weight parameters. It is worth noting that there are also Boolean flag parameters used for front-end and back-end communication, controlling synchronization and asynchrony between the two, which is part of the online design of this invention.

[0053] Step S1.2 Receive the first frame, and back-project the current RGBD frame as world coordinates.

[0054] Step S1.2.1 Create a filter index and pre-computed coordinate values ​​for image downsampling.

[0055] The sampling interval is calculated using the downsampling scale, and grid point coordinates are created to determine which pixels need to be retained.

[0056] Normalized values ​​of pixel coordinates (u, v) relative to the camera center are pre-calculated, and these values ​​are then combined with depth values ​​to obtain the actual 3D point coordinates.

[0057] Returns the downsampled pixel index and pre-computed normalized x and y coordinates for quickly building point clouds.

[0058] Step S1.2.2 Generate downsampled point cloud data from RGB-D image data.

[0059] Process color and depth data, including converting them into tensors and downsampling and normalizing them.

[0060] Calculate the camera spatial coordinates. Based on the previously calculated pixel coordinates and indices, and combined with the depth map, calculate the camera spatial coordinates; this is also known as inverse projection.

[0061] Since there is no camera pose in the input, this invention uses the camera pose of the first frame as the initial pose, and sets it as follows: The identity matrix. Subsequent poses are based on this.

[0062] Step S2 Runs a front-end that is accelerated by NICP-2DGS.

[0063] NICP-2DGS accelerated front end such as Figure 2 The NICP section of the overall schematic diagram is shown.

[0064] Specifically as follows: Step S2.1 Decoding point cloud surface data.

[0065] Specifically as follows: For streaming RGBD input, this invention combines camera intrinsic parameter inverse projection to obtain the camera coordinate point cloud:

[0066] in, ,make , which serves as the world coordinate system.

[0067] This invention uses the k-nn algorithm for PCA principal component analysis to find... The covariance of each point in the equation.

[0068]

[0069] in It constitutes The set of points in the neighborhood, yes The center of mass.

[0070] Now that we have the Gaussian parameters, we can perform singular value decomposition on the pointwise covariance matrix:

[0071] in yes The eigenvalues ​​are arranged in ascending order. The curvature of each point is defined as:

[0072] The smaller the curvature, the flatter the surface at that point. This is used to measure how well the intrinsic Gaussian fits the surface.

[0073] Based on the covariance of points, the covariance of the intrinsic Gaussian is defined as:

[0074] in It is a very small number. Therefore, the normal direction is defined as the shortest axis. Direction. If the surface cannot be well approximated by a local plane, the covariance remains unchanged.

[0075] The distance sensor collects point measurements from samples of a segmented, continuous surface. This invention uses the normal and curvature of a point to locally characterize the surface around that point.

[0076] The above is about point clouds. The surface is characterized as follows:

[0077] The parameters represent the mean, curvature, Gaussian covariance, and normal of the point, respectively.

[0078] Step S2.2 Point cloud registration.

[0079] The initial correspondence is calculated using an index map, which is the projected pixel coordinates of the point clouds from different source frames in the target frame coordinates. Match the points to the corresponding coordinates of the target frame. If a point has multiple matches, select the nearest one. Consider the errors of unreliable points by comprehensively evaluating 3D coordinates, curvature, and normals.

[0080] At this point, we will obtain M pairs of initial correspondences between the two sets of points. .

[0081] Next, these point pairs are registered.

[0082] For an intrinsic Gaussian, the transformation at each Gaussian point is expressed as follows: :

[0083] The transformation involves two aspects: coordinate transformation and normal transformation. This transformation is a 6-DOF variable, requiring at least 6 sets of points to solve.

[0084] Using the least squares method, the combined error of multiple sets of points is obtained:

[0085] The relative transformation is the least squares solution of the sum of errors at a single point.

[0086] The information matrix in the middle It is a 6x6 matrix. We hope to have a... Rotate the corresponding points so that their normals are aligned, and primarily penalize the distance along the normal direction while ignoring the distance along the plane tangent. Therefore, this information matrix is ​​defined as:

[0087] in yes The inverse block matrix.

[0088] Using these information matrices, by allowing points to slide and rotate relative to each other along the tangent direction of the surface to align their normals, the correspondence between two points can be minimized. Then, the Gauss-Newton method can be used to iteratively update T.

[0089] Step S2.3 Keyframe detection.

[0090] Obtain the relative pose of the registration Least square solution Then, the current camera pose is obtained using world coordinates and relative pose.

[0091] The decision to create a keyframe is based on two criteria: Visual overlap detection: Obtain the correspondence and distance between the current frame and the target point cloud, calculate the proportion of nearby matching points, and if the proportion is lower than the threshold (default 0.7), it is considered a tracking keyframe.

[0092] Fixed frequency detection: Create mapping keyframes every fixed number of frames (default every 5 frames).

[0093] Step S2.4 Keyframe processing and maintenance.

[0094] Tracking keyframe processing: Wait for the Mapper to complete the current task, calculate the pose information (rotation, scaling) of the current point cloud, remove new points that overlap too much with the existing map, write the new point cloud data to shared memory, update camera parameters and mark them as keyframes, wait for the Mapper to prepare the new target point cloud and update the NICP target.

[0095] Mapping keyframe processing: Wait for the Mapper to complete its current task, calculate the pose information, write the data to shared memory, and mark it as a mapping keyframe.

[0096] Step S3 Initialize a new intrinsic Gaussian.

[0097] Initialize new eigenvalues ​​as follows Figure 2 The intrinsic Gaussian elements are shown in the overall schematic diagram.

[0098] Step S3.1 Modeling of intrinsic decomposition on Gaussian splash.

[0099] Traditional intrinsic decomposition of a single view can decompose pixels according to Retinex theory as follows:

[0100] R and S represent the reflectivity and illumination of a two-dimensional Gaussian, respectively. This is a pixel-by-pixel channel multiplication, where C represents the synthesized color value. However, this invention aims to extend intrinsic decomposition to multi-view scenes. To maintain consistency in reflectivity and illumination across multiple viewpoints, residuals are introduced for fitting multiple views:

[0101] The residual is represented as the truth value. and The pixel-by-pixel difference is used to smooth the two. By introducing residuals, a better multi-view consistency effect is achieved in scenes with inconsistent multi-viewpoints (such as highly reflective objects and scenes that do not satisfy the Lambert assumption). It should be noted that residuals are only an auxiliary element for multi-viewpoint smoothing consistency, and should be kept at a small scale throughout the optimization process, rather than becoming the main component. This is specifically reflected in the optimization process.

[0102] The geometry of indoor scenes is mainly dominated by planar structures, possessing relatively smooth geometric structures. In terms of representation, surface primitives and depth input information can effectively represent the scene. Therefore, this invention chooses to use 2DGS as the primitives carrying intrinsic attributes, thus obtaining the following intrinsic Gaussian primitive representation.

[0103] This invention innovatively integrates intrinsic decomposition into two-dimensional Gaussian properties for fast intrinsic decomposition property map rendering. It can be described as follows:

[0104] like Figure 2 The intrinsic Gorsky element column is shown.

[0105] Step S3.2 Initialize the intrinsic Gaussian.

[0106] Once the pose of the current frame is obtained, the intrinsic Gaussian can be initialized based on the pose of the current frame and the resolved surface data.

[0107] This invention uses shared memory technology to create shared memory between the front-end and back-end, which is used for the front-end to pass shared covariance and normal information to the back-end. The point-by-point normals and two-dimensional covariance obtained in the front-end surface analysis step are losslessly propagated to the back-end for the initialization of newly added intrinsic Gaussians.

[0108] Initialization properties include: _xyz: 3D position coordinates are directly derived from the input point cloud. _features_dc: Directly uses the input color information as the DC component. _features_rest: Initialized to zero, representing higher-order spherical harmonic coefficients. _opacity: Initialized to a near-transparent value (using the inverse sigmoid function). _scaling: Initializes the scale using the input scale information via the inverse sigmoid function. _rotation: Initializes using the input quaternion rotation information. _n: Initialize using front-end NICP normals _reflectance: Initialized to the point color _shading: Initializes a tensor to all 1s. _residual: Initialized to 0 Step S4 Runs a NICP-2DGS accelerated backend.

[0109] Thanks to the intrinsic Gaussian initialization of the NICP front-end geometric properties, the intrinsic Gaussian converges quickly during the optimization process, significantly reducing the optimization iteration time for covariance gradient transformation.

[0110] The present invention uses differentiable rendering loss to optimize the scene, including reconstruction loss and intrinsic decomposition loss.

[0111] Specifically as follows: Step S4.1 Reconstruction losses.

[0112] Luminous loss. Firstly, the synthesized image should be consistent with the true input value. This invention uses... SSIM structural loss ensures consistent perception.

[0113]

[0114] Depth loss. The input depth map provides geometric information, and this invention ensures the structural similarity between the two.

[0115]

[0116] Normal loss. The local neighborhood gradient of the ground truth depth provides normal information. This invention promotes consistency between splashed normals and depth neighborhood normals.

[0117]

[0118] This indicates the mixed weight at that point.

[0119] Step S4.2 Intrinsic decomposition loss.

[0120] Reflectivity loss.

[0121] Piecewise reflectance loss. This invention imposes constraints on reflectance based on prior knowledge. Reflectance images are typically composed of piecewise constant regions, which can be minimized by minimizing the pixel-wise reflectance gradient. The weight w is obtained by raising the value to the power of p. The weight w is determined by the gradient of the chroma; the more similar the chroma, the lower the gradient. The weight w after passing through the step function is higher, so the more similar the chroma, the more optimization is needed. The smoother the appearance and the less similar the colors, the less optimized it is. And it is presented in segments.

[0122]

[0123] in Represents the RGB color space to Color space conversion functions.

[0124] Reflectance chromaticity loss. Illumination only affects the intensity of pixels, not their chromaticity. Therefore, large changes in chromaticity come from reflectance, not illumination shading. This invention uses reflectance chromaticity loss to maintain consistency between the two.

[0125]

[0126] Lighting loss. This invention assumes the object has a diffuse reflective surface, and light is uniformly diffused. Therefore, lighting shading is shape-dependent. Assuming the smooth shape of indoor objects, the lighting shading should be smooth. That is, the lighting shading gradient needs to be minimized. Large changes in chromaticity come from reflectance rather than lighting shading, so lighting shading should be optimized more in areas with large chromaticity gradients. Therefore, [the following is used]... As gradient.

[0127]

[0128] Although and They are similar in form, but they differ in the scope of optimization. The former targets regions with smaller gradients, while the latter targets regions with larger gradients.

[0129] Residual loss. The residual represents the difference that cannot be fitted using illumination and reflectivity, and represents the non-diffuse area of ​​the scene, so this variable is minimized.

[0130]

[0131] Step 4.3 Joint optimization.

[0132] Combining all the above losses, this invention simultaneously performs multi-loss joint optimization, expressed as:

[0133] Step S5 Accept the next frame and jump to step S2 until the input stream terminates.

[0134] Example Dataset This invention is implemented on the Replica and ScanNet++ datasets, both of which are multi-view indoor datasets. The former is synthetic data, while the latter is real data.

[0135] Implementation effect Figure 3 This is the visualization comparison result of the dataset Replica shown in this embodiment.

[0136] The quantitative quantization results of this invention on the ScanNet++ dataset are shown in Table 1.

[0137] Table 1. Quantitative results of this invention on the ScanNet++ dataset.

[0138] In addition, this invention demonstrates certain application scenarios, and can be used for real-time scene relighting and recoloring editing. The editing effect is visualized as follows: Figure 4 This is the result of relighting editing shown in this embodiment.

[0139] Figure 5 This is the recoloring editing result shown in this embodiment.

[0140] Key points and advantages of the present invention The ingenuity of intrinsic Gaussian primitive representation lies in bringing intrinsic properties into 2DGS, combining the advantages of 2DGS in geometric accuracy and multi-view color representation. The intrinsic properties in Gaussian representation can be backpropagated to the decomposition loss by means of Gaussian splashing rasterization, and scene decomposition and inverse rendering can be performed under any intrinsic property initialization.

[0141] Intrinsic Gaussian representation offers two advantages: first, it makes the model insensitive to the initial intrinsic attribute values, ensuring the possibility of a convergence domain; second, it makes the model not limited to specific scenarios, ensuring adaptability in most scenarios.

[0142] A key observation of NICP-2DGS acceleration is that, under Lambert's scene assumptions, most existing depth measurement methods are based on Time of Flight (ToF)-like methods, where depth is essentially a sampling point on the object's surface, rather than penetrating deep into the object's interior. Therefore, a 2D Gaussian surface, similar to a patch, can effectively fit the object's surface and simulate scene geometry, thus allowing the backend to benefit from using a 2D Gaussian surface in the depth map.

[0143] The ingenuity of NICP-2DGS acceleration lies in two key aspects. First, the front-end localization is no longer based on dense photometric loss, but rather on a traditional iterative nearest-point method that fully integrates geometric input. This separates the front-end from the back-end rasterization, freeing the front-end from reliance on dense rasterization and enabling rapid localization. Second, and most importantly, NICP registration generates secondary assets such as point cloud normals and 2D covariance. These point cloud attributes are essentially free for the back-end, requiring no additional computation to obtain optimal initialization values. Using these initial values ​​during the back-end's intrinsic Gaussian initialization allows for rapid convergence. Conversely, the back-end's optimized Gaussian attribute values ​​can also be passed to the front-end. Using the optimized Gaussian attributes from the back-end's graph loss for front-end localization allows the back-end to incorporate joint loss optimization based on various information, influencing front-end localization and ensuring the front-end is no longer isolated from the back-end.

[0144] An intrinsic representation and decomposition primitive for fast multi-view scenes: Intrinsic Gaussian primitives. This representation, combined with existing splatting rendering techniques, enables differentiable rendering of intrinsic attributes, providing efficient map representation, optimization, and storage. Intrinsic Gaussian differs from previous inverse rendering frameworks, enabling online implementation of localization, mapping, and intrinsic decomposition. It guarantees a wide convergence domain and high applicability of the model.

[0145] A fast front-end and back-end collaborative acceleration optimization method is implemented: NICP-2DGS acceleration. Localization is separated from rasterization, divided into front-end and back-end. The front-end maintains high-speed NICP localization, while the back-end uses intrinsic Gaussian elements from technical solution 1. Attribute sharing between the front-end and back-end is maintained, allowing the back-end to benefit from the superior initial attribute values ​​of the front-end, and the front-end to benefit from the attribute values ​​optimized by the back-end loss. The two mutually promote each other, jointly accelerating optimization and doubling the model convergence speed.

[0146] A two-dimensional Gaussian online indoor intrinsic decomposition and reconstruction framework is presented. This framework achieves online synchronous localization, mapping, and intrinsic decomposition tasks at a rate exceeding 12 FPS on a commercial 3090 graphics card device, demonstrating superior performance in terms of accuracy, efficiency, and practicality.

[0147] It meets the urgent need for real-time indoor scene editing and interaction, demonstrating broad application prospects. With features including real-time editing of reflectivity and inherent color, and interactive editing of lighting intensity, it holds immense potential in real-time AR interactive scenarios.

[0148] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. An online eigendecomposition reconstruction method, characterized in that, Comprising the following steps: Step S1 System initialization; initialize the entire system; accept the first frame, inverse project the current RGBD frame as the world coordinate; In detail as follows: Step S1.1 Initialize the entire system, including creating the mapper class for back-end mapping, the tracker class for front-end positioning, the dataset class for dataset loading, and the viewer class for visualization; Read the hyperparameters for the subsequent system, including: Gaussian scene parameters, SLAM pipeline parameters, NICP registration parameters, intrinsic decomposition loss weight parameters, and several other categories; It should be noted that there is also a part of the Boolean type flag parameter for front-end and back-end communication, which is used to control the synchronization and asynchronization of the front-end and back-end, as one of the online designs of the present application; Step S1.2 Accept the first frame, inverse project the current RGBD frame as the world coordinate; Step S1.2.1 Create a filter index and pre-computed coordinate value for image downsampling; Calculate the sampling interval using the downsampling scale, create grid point coordinates to determine which pixels need to be retained; Pre-compute the normalized value of the pixel coordinate (u, v) relative to the camera center, which will be combined with the depth value to obtain the actual 3D point coordinate later; Return the down-sampled pixel index and pre-computed x, y direction normalized coordinate value for fast point cloud construction; Step S1.2.2 Generate down-sampled point cloud data from RGB-D image data; Process color data and depth data, including converting to tensors and downsampling tensors, and normalizing; Calculate camera space coordinates; calculate camera space coordinates according to previously calculated pixel coordinates and indices, combined with depth maps, which is also called inverse projection; Since there is no camera pose in the input, the camera of the first frame is set as the initial pose, that is, Identity matrix; the subsequent poses are taken as the reference; Step S2 Run the NICP-2DGS accelerated front-end; In detail as follows: Step S2.1 Point cloud surface data decoding; In detail as follows: For streaming RGBD input, the present application combines camera intrinsic inverse projection to obtain camera coordinate point cloud: wherein , let , be the world coordinates; The present application uses knn algorithm to perform PCA principal component analysis, and covariance of each point is solved ​ wherein is a set of points constituting a neighborhood, is a centroid of Now that there are Gaussian parameters, perform singular value decomposition on the point-wise covariance matrix: wherein is the eigenvalues of the feature vector, arranged in ascending order; the curvature of each point is defined as: The smaller the curvature, the flatter the point; to measure the fitting degree of the intrinsic Gaussian to the surface; According to the point covariance, the covariance of the intrinsic Gaussian is defined as: where is a small number; so the normal direction is defined as the shortest axis direction; if the surface cannot be well approximated by a local plane, the covariance is left unchanged; The point measurements collected by the distance sensor are samples from a piecewise continuous surface; the present application uses the point's normal and curvature to locally characterize the surface around the point; Above, point cloud For the characterization of surfaces: The parameters in it represent the mean, curvature, Gaussian covariance and normal of the point, respectively; Step S2.2 Point cloud registration; The initial correspondence is calculated by index map, i.e. the pixel coordinates of the projection of the different source frame point clouds in the target frame coordinates The points corresponding to the coordinates of the respective target frames are matched; when a point has multiple matches, the nearest point is taken; the errors of the unreliable points are eliminated by considering the 3D coordinates, curvature and normal; At this time, the M pairs of initial correspondence between the two sets of points are obtained ; Next, register these point pairs; For intrinsic Gaussians, the transformation of each Gaussian point is represented as follows : The transformation contains two aspects, one is coordinate transformation, and the other is normal transformation; this transformation is a 6-degree-of-freedom variable, which requires at least 6 groups of points to solve; Using the least squares method, the comprehensive error of multiple points is obtained: The relative transformation is the least squares solution of the sum of the errors of individual points; The information matrix in the middle is a 6*6 matrix; we want to have one that rotates the respective points so that their normals are aligned and mainly penalizes the distance along the normal direction and ignores the distance along the plane tangent; so this information matrix is defined as: wherein is the inverse block matrix of Using these information matrices, by allowing points to slide along the tangent direction of the surface and rotate them to align their normals, the correspondence between two points can be minimized; the Gauss-Newton method can then be used to iteratively update T; Step S2.3 Key frame judgment; Acquiring a registered relative pose least squares solution Acquiring a current camera pose using world coordinates and relative pose later Determine whether to create a key frame based on two criteria: Visual overlap detection: Get the correspondence and distance between the current frame and the target point cloud, calculate the proportion of close-range matching points, if the proportion is lower than the threshold (default 0.7), it is considered to be a tracking key frame; Fixed frequency detection: Create a mapping key frame every fixed number of frames (default every 5 frames); Step S2.4 Key frame processing and maintenance; Tracking key frame processing: Wait for the Mapper to complete the current task, calculate the pose information (rotation, scaling) of the current point cloud, remove the new points that overlap too much with the existing map, write the new point cloud data into the shared memory, update the camera parameters and mark as a key frame, wait for the Mapper to prepare the new target point cloud and update the NICP target; Mapping key frame processing: Wait for the Mapper to complete the current task, calculate the pose information, write the data into the shared memory, and mark it as a mapping key frame; Step S3 Initialize a new intrinsic Gaussian; Step S3.1 Modeling of intrinsic decomposition on Gaussian splatting; Traditional single-view intrinsic decomposition can decompose pixels into: R and S represent reflectance and illumination of a two-dimensional Gaussian, respectively, where is a pixel-wise channel multiplication, C represents the resulting color value; however, the present application aims to extend the eigen decomposition to multi-view scenes, in order to maintain consistency of the multi-view reflectance and illumination, a residual is introduced for multi-view fitting: Residual is expressed as the true value and The pixel-by-pixel difference of and is used for smoothing both; by introducing the residual, there is a better multi-view consistency effect in multi-view inconsistent scenes (such as highly reflective objects and scenes that do not meet the Lambert assumption); it should be noted that the residual is only an auxiliary element of multi-view smoothing consistency, and needs to be kept in a small scale in the whole optimization, and cannot occupy the main component, which is specifically embodied in the optimization; The geometry of indoor scenes is mainly dominated by planar structures, with relatively smooth geometry. In the sense of expression, using surface primitives and depth input information can better represent the scene. Therefore, the present application selects to use 2DGS as the primitive that carries the intrinsic properties, thus obtaining the following intrinsic Gaussian primitive representation: The present application innovatively integrates intrinsic decomposition into two-dimensional Gaussian properties for fast intrinsic decomposition attribute map rendering, which can be expressed as: Step S3.2 Initialize the intrinsic Gaussian; Once the pose of the current frame is obtained, the intrinsic Gaussian can be initialized according to the current frame pose and the parsed surface data; The present application uses shared momery technology to create shared memory in the front end and the back end, which is used to pass the shared covariance and normal information from the front end to the back end. The point-by-point normal and two-dimensional covariance obtained in the front-end surface parsing step are propagated to the back end without loss, which is used for initialization of the newly added intrinsic Gaussian; Initialization attributes include: _xyz: 3D position coordinates directly from input point cloud _features_dc: Directly use the input color information as the direct current component _features_rest: Initialized to zero, representing high-order spherical harmonic coefficients _opacity: Initialized to a value close to transparency (through inverse sigmoid function) _scaling: Use the input scale information, initialized through inverse sigmoid function _rotation: Use the input quaternion rotation information for initialization _n: Use the front-end NICP normal _reflectance: Initialized to point color _shading: Initialized to a full 1 tensor _residual: Initialized to 0 Step S4 Run the NICP-2DGS accelerated back end; Thanks to the intrinsic Gaussian initialization of the geometric properties of the NICP front end, the intrinsic Gaussian quickly converges during optimization; reducing most of the covariance gradient transformation optimization iteration time; The present application uses differentiable rendering to optimize the scene using loss, including reconstruction loss and intrinsic decomposition loss. Specifically as follows: Step S4.1 Reconstruction loss; photometric loss; the first synthesized image should agree with the input ground truth, the present invention uses and SSIM structural loss ensures perceptual agreement; Depth loss; input depth map Depth provides geometry information, the present invention ensures the structure similarity of both; Normal loss; local neighborhood gradient of ground truth depth provides normal information; the present invention promotes the consistency of splash normal and depth neighborhood normal; a mixing weight representing the point; Step S4.2 Eigen-decomposition loss; Reflectance loss; Reflectance piecewise loss; the present application makes a constraint on reflectance according to priori; a reflectance image is usually composed of piecewise constant regions, which can be obtained by minimizing the pixel-wise reflectance gradient ; the weight w is determined by the gradient of the chroma, the more similar the chroma is, the lower the gradient of the chroma is; the weight w after the step function is higher, so the more similar the chroma is, the more optimization ; the more dissimilar the chroma is, the less optimization ; wherein represents a conversion function from the RGB color space to the YUV color space; the YUV color space; Reflectance chrominance loss; illumination only affects the intensity of pixels, without changing the chrominance of pixels, so large changes in chrominance come from Reflectance rather than Shading; therefore the present invention uses reflectance chrominance loss to keep the consistency of both; Loss of illumination; the present invention assumes that the object has a diffuse surface, the light is uniformly distributed so the illumination Shading is object shape dependent, assuming the smooth shape of indoor objects, then the illumination Shading should be smooth; that is to say the gradient of the illumination Shading needs to be minimized; large changes in chrominance come from Reflectance rather than illumination Shading, so the places where the chrominance gradient is large the illumination Shading should be optimized more, therefore using the gradient as a cost function; Although and Formally similar, but different in the range of optimization, the former aiming at smaller regions of gradient, the latter aiming at larger regions of gradient; Residual loss; residual represents the fitting difference that cannot be fitted by lighting and reflectance, represents the non-diffuse reflection area of the scene, so minimize this variable; Step 4.3 Joint optimization; Combining all the losses above, the present invention performs multi-loss joint optimization at the same time, represented as: Step S5 Accept the next frame, jump to step S2, until the input stream terminates.

2. An intrinsic Gaussian scene representation primitive, characterized by, As claimed in claim 1, step S3.1 modeling on Gaussian splash and step S3.2 initialization of eigen-Gaussian.

3. A NICP-2DGS acceleration method, characterized by, As claimed in claim 1, step S2 NICP-2DGS accelerated front-end and step S4 NICP-2DGS accelerated back-end.

Images