Three-dimensional reconstruction method and system of three-dimensional gaussian splatter indoor scene based on planar prior guidance

Abstract

The present application relates to a kind of three-dimensional reconstruction method and system of three-dimensional Gaussian splashing indoor scene based on plane prior guidance, belong to three-dimensional reconstruction and computer graphics technology field.The method aims to solve the problem of inaccurate geometry reconstruction and easy to produce artifact in weak texture area in indoor scene reconstruction, especially in weak texture area.The method aims to solve the problem of inaccurate geometry reconstruction and easy to produce artifact in weak texture area in indoor scene reconstruction, especially in weak texture area.First, obtain multi-view image and its camera parameter, predict depth and normal prior, and extract plane area by multi-granularity segmentation and geometric consistency test.Density point cloud is generated by depth prior back projection, and three-dimensional Gaussian splashing model is initialized based on down-sampling point cloud.Finally, under the framework of differentiable rendering, the model is optimized by combining plane area geometric constraint, non-plane area geometric constraint and global geometric constraint, and the reconstruction is completed.The present application effectively improves the reconstruction accuracy of weak texture plane area, enhances the detail recovery ability of complex structure area, and realizes the reconstruction result with visual high fidelity and geometric accuracy.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction and computer graphics technology, and relates to a 3D reconstruction method and system for 3D Gaussian splash indoor scenes based on planar prior guidance. Background Technology

[0002] With the development of Virtual Reality (VR), Augmented Reality (AR), and automated robotics, high-precision 3D digital modeling of real-world scenes has become an important research direction. Traditional 3D reconstruction methods are mainly based on Multi-View Stereo (MVS) technology, which recovers 3D structures through image feature matching. However, in areas with weak textures or environments with significant lighting variations, these methods often struggle to operate stably.

[0003] In recent years, Neural Radiance Field (NeRF) has achieved high-quality new view synthesis through volume rendering technology. However, due to its use of Multi-Layer Perceptron (MLP) for implicit scene representation, the training and inference computation costs are high, making it difficult to achieve real-time rendering.

[0004] 3D Gaussian Splatting (3DGS) technology uses anisotropic 3D Gaussian ellipsoids as scene representation units and achieves efficient real-time rendering through differentiable rasterization, effectively solving the problem of high computational cost in NeRF. However, existing 3DGS methods mainly rely on photometric consistency for optimization and lack explicit constraints on geometric structures, which can easily lead to problems such as geometric collapse, floating-point artifacts, and surface distortion during indoor scene reconstruction.

[0005] Indoor environments typically exhibit distinct structural features, with structures such as walls, floors, and tabletops often appearing as large planar surfaces. To introduce planar geometric priors into the 3D Gaussian splash optimization process and effectively improve the geometric accuracy of scene reconstruction, this invention proposes a 3D reconstruction method and system for indoor scenes based on planar priors. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a method and system for three-dimensional reconstruction of a three-dimensional Gaussian splash indoor scene based on planar prior guidance.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A 3D reconstruction method for a 3D Gaussian splash indoor scene based on planar prior guidance, the method comprising:

[0009] Acquire a multi-view image sequence of the indoor scene to be reconstructed and the corresponding camera parameters;

[0010] Depth prior and normal prior are predicted from the input images of the multi-view image sequence using a monocular depth estimation model and a normal estimation model, respectively.

[0011] The input image and normal prior map are segmented at multiple granularities using an image segmentation model to generate candidate region masks. The candidate regions are then subjected to geometric consistency checks in conjunction with the normal prior map to extract planar prior regions that satisfy planar structural features.

[0012] Based on the depth prior and the camera parameters, the image pixels are back-projected into a three-dimensional space to generate a dense point cloud, and the dense point cloud is subjected to voxel downsampling processing. A three-dimensional Gaussian splash model is then constructed based on the downsampled point cloud.

[0013] Furthermore, within the framework of 3D Gaussian splash differentiable rendering, the 3D Gaussian splash model is optimized through planar region geometric constraints, non-planar region geometric constraints, and global geometric constraints to complete the 3D reconstruction of the scene.

[0014] Furthermore, the step of using an image segmentation model to perform multi-granularity segmentation on the input image and the normal prior map to generate candidate region masks includes:

[0015] The image segmentation model is used to perform semantic-level segmentation on the input image to generate semantic-level candidate region masks. Geometric consistency checks are performed on the semantic-level candidate region masks, and regions that pass the check are confirmed as semantic-level planar regions and removed from the remaining valid regions.

[0016] The image segmentation model is used to perform instance-level segmentation on the remaining effective regions after removing semantic-level planar regions, generating instance-level candidate region masks. Geometric consistency checks are performed on the instance-level candidate region masks, and regions that pass the check are confirmed as instance-level planar regions and removed from the remaining effective regions.

[0017] The image segmentation model is used to perform component-level segmentation on the remaining effective regions after removing semantic-level planar regions and instance-level planar regions, generating component-level candidate region masks. Geometric consistency checks are performed on the component-level candidate region masks, and regions that pass the checks are confirmed as component-level planar regions.

[0018] The semantic-level planar regions, the instance-level planar regions, and the component-level planar regions are combined to form a global planar prior mask.

[0019] Furthermore, the geometric consistency check includes:

[0020] Geometric edges are extracted based on the normal prior; for each candidate region mask, it is determined whether it crosses the geometric edge;

[0021] If the candidate region mask crosses the geometric edge, then the candidate region mask is cut with the geometric edge as the boundary to obtain independent sub-regions;

[0022] And filter out mask areas with pixel sizes smaller than a preset threshold to obtain the final planar area.

[0023] Furthermore, the geometric constraints of the planar region include planar depth coplanar constraints and planar normal consistency constraints;

[0024] The planar depth coplanar constraint is achieved by applying a planar parameter model fitting and L1 loss to the planar prior region, and the planar normal consistency constraint is achieved by applying normal map gradient smoothing inside the planar prior region.

[0025] Furthermore, the loss function of the planar depth coplanar constraint Represented as:

[0026]

[0027] in, Represents the set of pixels for all planar prior regions. u Represents pixel coordinates, D ( u The ) represents the depth value obtained from rendering the 3D Gaussian splash model. This represents the ideal coplanar depth calculated based on the planar parameters of the aforementioned a priori region.

[0028] Furthermore, the loss function of the plane normal consistency constraint Represented as:

[0029]

[0030] in N This represents the normal map obtained from Gaussian rendering. For the joint mask in the horizontal direction, This is a horizontal mask in the vertical direction.

[0031] Furthermore, the non-planar region geometric constraint achieves local geometric smoothing by constructing a weighted fusion depth based on the normals of local neighboring pixels for pixels within the non-planar region. Its loss function... Represented as:

[0032]

[0033] in, Represents a set of non-planar pixels. This represents the depth value obtained from rendering a 3D Gaussian splash model. Indicates the target pixel The ideal depth is obtained by weighted fusion of neighboring pixels, and the calculation formula is as follows: , For target pixels The set of neighboring pixels, For neighboring pixels The weight, For neighboring pixels normal and three-dimensional coordinates Calculated target pixels The predicted depth.

[0034] Furthermore, the global geometric constraints include depth prior constraints, normal prior constraints, and depth normal consistency constraints;

[0035] Among them, the loss function of depth prior constraints for Loss function of normal prior constraint for Loss function for depth normal consistency constraints for ;

[0036] in Represents the set of all pixels in an image. D This indicates the rendering of the depth map. This represents the depth prior graph. N This indicates the rendering of the normal map. Represents the a priori diagram of normals. This represents the surface normal map derived from the rendered depth map.

[0037] Furthermore, the total loss function of the joint optimization of the three-dimensional Gaussian splash model is further optimized. Represented as:

[0038]

[0039] in, This refers to the photometric consistency loss within a 3D Gaussian splash differentiable rendering framework. For planar depth coplanar constraint loss, For plane normal consistency constraint loss, For geometric constraint loss in non-planar regions, For deep prior constraint loss, For the normal prior constraint loss, For depth normal consistency constraint loss, , , , , , These are the preset weighting coefficients for each loss item.

[0040] A 3D reconstruction system for an indoor Gaussian splash scene based on planar prior guidance, the system comprising:

[0041] The image and parameter acquisition module is used to acquire multi-view image sequences of the indoor scene to be reconstructed and the corresponding camera parameters;

[0042] The prior information extraction module is used to predict depth prior and normal prior from the input image of the multi-view image sequence using a monocular depth estimation model and a normal estimation model, respectively.

[0043] The planar prior extraction module is used to perform multi-granular segmentation of the input image using an image segmentation model, generate candidate region masks, and perform geometric consistency checks on the candidate regions in conjunction with the normal prior, so as to extract planar prior regions that satisfy planar structural features.

[0044] The model initialization module is used to back-project image pixels into three-dimensional space to generate a dense point cloud based on the depth prior and the camera parameters, and to perform voxel downsampling on the dense point cloud, and to initialize and construct a three-dimensional Gaussian splash model based on the downsampled point cloud.

[0045] The system also includes a model optimization and reconstruction module, which optimizes the 3D Gaussian splash model within the 3D Gaussian splash differentiable rendering framework by using planar region geometric constraints, non-planar region geometric constraints, and global geometric constraints to complete the 3D reconstruction of the scene.

[0046] The beneficial effects of this invention are as follows:

[0047] (1) This invention extracts the planar prior region in the indoor scene and introduces the planar region geometric constraint in the optimization process of the three-dimensional Gaussian splash model, so that the Gaussian primitives are more in line with the real physical plane in spatial distribution. This effectively solves the problems of unstable geometric structure and surface distortion in weak texture areas such as walls, floors and ceilings, thereby significantly improving the geometric reconstruction accuracy of the planar region in the indoor scene.

[0048] (2) In addition to the geometric constraints of the planar region, the present invention further constructs the geometric constraints of the non-planar region. By estimating the local geometric structure through the local neighborhood normal information and establishing local geometric consistency constraints, the surface structure, object boundary and complex detail region can obtain reasonable geometric guidance in the optimization process, thereby improving the detail recovery ability and overall reconstruction quality of the non-planar region while maintaining the stability of the planar region.

[0049] (3) The present invention can output indoor scene 3D reconstruction results with both visual high fidelity and geometric accuracy. It can be widely used in indoor digital twins, virtual reality, augmented reality, robot navigation, indoor scene modeling and 3D content generation, etc., and has good engineering application prospects.

[0050] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0052] Figure 1 This is a schematic diagram of the overall process of three-dimensional reconstruction of a three-dimensional Gaussian splash indoor scene provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of a planar prior extraction method provided in an embodiment of the present invention. Detailed Implementation

[0054] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0055] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0056] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0057] Example 1

[0058] like Figure 1 As shown in the figure, this embodiment provides a 3D reconstruction method for a 3D Gaussian splash indoor scene based on planar prior guidance. The method is as follows:

[0059] 1. Obtain multi-view image sequences of the indoor scene to be reconstructed and the corresponding camera parameter information;

[0060] In this embodiment, multi-view image acquisition is first performed on the indoor scene to be reconstructed. The indoor scene can be a residence, office, laboratory, classroom, exhibition hall, shopping mall interior space, or other environments with obvious indoor structural features. The image acquisition device can be a monocular camera, RGB camera, RGB-D camera, mobile terminal camera, or other imaging devices capable of acquiring scene image information. To ensure the reliability of subsequent 3D reconstruction, during the acquisition process, it is preferable to have sufficient visual overlap between adjacent images and to cover as many major structural surfaces in the scene as possible, such as walls, floors, desktops, doors, windows, cabinets, etc. After acquiring the multi-view image sequence, the camera parameter information corresponding to each image is further obtained. The camera parameter information includes camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include focal length, principal point position, and distortion parameters when necessary; the camera extrinsic parameters describe the camera's position and attitude relationship in the world coordinate system. Camera parameters can be obtained through pre-calibration or estimated using methods such as visual structure reconstruction or motion reconstruction.

[0061] 2. Use a monocular depth estimation model and a normal estimation model to extract depth prior information and normal prior information from the input image, respectively;

[0062] Specifically, for each input image frame, a pre-trained monocular depth estimation network is used to infer the image and obtain the corresponding depth prior map. The depth estimation model can be a Depth Anything model, which can predict a depth map with good geometric consistency without scene-specific training; at the same time, a pre-trained monocular normal estimation network is used to infer the image and obtain the corresponding normal prior map, and the normal estimation model can be a StableNormal model.

[0063] 3. The input image and normal prior map are segmented at multiple granularities using an image segmentation model to generate candidate region masks. The geometric consistency of the candidate regions is then checked in conjunction with the normal prior information to extract planar prior regions that satisfy the planar structure features.

[0064] like Figure 2 The diagram illustrates the planar prior extraction process based on multi-granularity image segmentation in this embodiment. This planar prior extraction module simultaneously receives the input RGB image and the normal prior map obtained in step two as inputs. The RGB image provides texture and color information of the scene for the image segmentation model to generate two-dimensional candidate region masks; the normal prior map provides local geometric direction information of the scene surface to perform geometric consistency checks on whether the candidate regions satisfy planar structural features.

[0065] Specifically, such as Figure 2 As shown on the left, the input includes the original RGB image and a normal prior map. The RGB image mainly reflects the color distribution, edge information, and texture details in the scene, while the normal prior map reflects the local directional changes of the surface corresponding to each pixel. Since the physical planes in indoor scenes vary significantly in scale, single-granularity segmentation results can easily lead to large planar regions being fragmented into multiple pieces, or small-scale planar structures being swallowed up by surrounding areas. This embodiment uses a multi-granularity image segmentation method to generate candidate region masks at different scales, and further extracts the planar prior regions step by step through a top-down cascaded extraction mechanism. Specifically, the image segmentation model used in this embodiment can be the Semantic-SAM model, which can specify different granularities to generate semantic-level, instance-level, and part-level segmented images respectively.

[0066] like Figure 2 As shown in the middle, the planar prior extraction process in this embodiment includes three segmentation levels from coarse to fine: semantic level segmentation, instance level segmentation, and component level segmentation.

[0067] In the first level, or the Level 1 semantic segmentation stage, the input image is first coarse-grained segmented using an image segmentation model to extract macroscopic structural regions in the indoor scene, such as walls, floors, and large-scale furniture areas. The candidate region masks output from this stage then enter the geometric verification module. Regions that pass the geometric verification are confirmed as semantic-level planar regions. At the same time, the confirmed planar regions are removed from the current valid regions, resulting in the remaining region masks to be passed to the next level. Figure 2 The black areas represent regions that have been identified and removed in the previous level, while the white areas represent regions that still need to be further segmented and verified.

[0068] In the second level, the Level 2 instance-level segmentation stage, only the remaining areas not identified as planar regions in the first level are segmented with medium granularity to obtain instance-level candidate regions, such as individual furniture surfaces, door and window areas, or local component areas. The candidate region masks generated in this stage also enter the geometric verification module; regions that pass the verification are confirmed as instance-level planar regions and are further eliminated from the remaining valid regions. Figure 2 As shown, Figure 2 middle" The symbol "" indicates a region culling operation, which is used to remove planar regions that have been identified in the previous level from the current image region, so that subsequent segmentation is performed only on the remaining unidentified regions.

[0069] In the third level, the Level 3 component-level segmentation stage, fine-grained segmentation is performed on the remaining areas after the previous level to extract local small-scale planar structures. This stage is mainly used to detect detailed planar regions that were not fully identified in the first two levels, such as local components on furniture surfaces and local decorative structures. After geometric verification, regions that meet the planar structural characteristics are output as component-level planar regions.

[0070] like Figure 2 As shown on the right, the semantic-level segmentation stage, the instance-level segmentation stage, and the component-level segmentation stage each output corresponding planar region results. Finally, the planar regions that pass the geometric consistency check in the three levels are summarized to form a global planar prior mask. This planar prior mask serves as an important input for constructing the geometric constraints of the planar regions in subsequent steps, guiding the 3D Gaussian model to perform accurate optimization on the planar regions of the indoor scene.

[0071] Geometric Verification: For candidate segmentation masks, this invention verifies each candidate mask using geometric edges, which are calculated from the normal prior map using an edge detection operator. Specifically, in this embodiment, the Canny operator is used to extract geometric edges from the normal prior map obtained in step 2, which represent the physical geometric boundaries of the scene. For each candidate segmentation mask, it is evaluated whether each mask region incorrectly crosses the physical geometric boundaries in the scene; if the detected segmentation mask crosses the geometric boundary, the geometric edge is directly used as a hard boundary to cut the original mask and decompose it into multiple independent sub-regions. Finally, mask regions smaller than a specified threshold pixel size are filtered out to obtain the final planar region prior.

[0072] 4. Based on depth prior, back-project the image pixels into three-dimensional space to generate a dense point cloud, and perform voxel downsampling on the point cloud. Then, initialize a three-dimensional Gaussian model based on the downsampled point cloud.

[0073] After obtaining the depth prior map and camera parameters predicted in step two, this step uses the depth back projection method to restore the two-dimensional image pixels into three-dimensional spatial points, thereby generating a dense three-dimensional point cloud, and initializes the three-dimensional Gaussian model based on the point cloud.

[0074] Specifically, let the pixel coordinates in the input image be... Its corresponding depth value The camera intrinsic parameter matrix is First, based on the camera imaging model, the pixel coordinates are back-projected onto the 3D camera coordinate system, and their 3D spatial points... The calculation method is as follows:

[0075]

[0076] in This represents the inverse of the camera intrinsic parameter matrix. Represents pixels The corresponding depth value.

[0077] By sampling a certain number of images from different viewpoints and performing the aforementioned back-projection operation on all pixels in the images, a dense set of 3D point clouds covering the entire field of view of the image can be obtained: ,in This indicates the number of point clouds generated. Since directly generating a large number of point clouds using all pixels would significantly increase the computational complexity of the subsequent 3D Gaussian optimization process, downsampling of the dense point cloud is necessary. This embodiment uses a voxel mesh downsampling method to simplify the point cloud. Specifically, the 3D space is divided into a fixed-resolution voxel mesh, with the side length of each voxel unit set to... For a set of point clouds falling within the same voxel cell, only one representative point is retained as the center point of that voxel, thus obtaining the downsampled point cloud set: ,in This represents the number of point clouds retained after downsampling. The obtained point cloud set... Then, for each point This serves as the central location of a 3D Gaussian primitive, used to initialize the 3D Gaussian splash model. For each 3D Gaussian primitive... Its parameter set is initialized as follows:

[0078]

[0079] in This indicates the location of the Gaussian center, corresponding to the point cloud coordinates. , This represents the Gaussian covariance matrix, used to describe the scale and orientation of Gaussian elements in space. This represents the color attribute of Gaussian elements. This represents the opacity parameter of the Gaussian element.

[0080] Through the above initialization process, a set of 3D Gaussian primitives can be constructed in 3D space to represent the initial geometric structure and appearance information of the scene. Compared with traditional initialization methods based on sparse feature point clouds, this embodiment utilizes depth priors to generate dense point clouds, thereby providing more complete geometric coverage in weakly textured regions and providing a more stable initial structure for subsequent 3D Gaussian model optimization.

[0081] 5. Construct geometric constraints for planar regions, geometric constraints for non-planar regions, and global geometric consistency constraints, and perform joint optimization on the 3D Gaussian model to complete the 3D reconstruction of the indoor scene.

[0082] After initializing the 3D Gaussian model, its geometric parameters need further optimization to obtain a more accurate scene geometry. This embodiment introduces planar region geometric constraints, non-planar region geometric constraints, and global geometric consistency constraints within the 3D Gaussian splash differentiable rendering framework to optimize the 3D Gaussian model, thereby improving the geometric accuracy of the 3D reconstruction of indoor scenes. Specific constraints are as follows:

[0083] (1) Geometric constraints of planar regions

[0084] Indoor scenes typically contain numerous planar areas with well-defined structural features, such as walls, floors, and tabletops. For these areas, planar depth coplanarity constraints and planar normal consistency constraints are introduced to ensure that the 3D reconstruction results satisfy the geometric properties of the real physical plane.

[0085] Planar Depth Coplanarity Constraint: Guided by the extracted planar priors, this embodiment applies a coplanarity penalty to the depth map generated by 3D Gaussian rendering. Given a rendered depth map... First, combining the camera intrinsic parameter matrix of the current viewpoint, the depth values ​​of these pixels are back-projected into the 3D camera coordinate system to obtain 3D spatial points. Then, in 3D space, the local physical plane is parametrically modeled using the following algebraic form:

[0086]

[0087] in, Indicates the first Planar parameter vectors for a planar region Indicates the first A three-dimensional point in a planar region.

[0088] To efficiently and robustly estimate plane parameters This embodiment uses the least squares method to obtain the closed-form solution. The objective constant vector is defined. With 3D point cloud matrix By minimizing the orthogonal projection residuals, the plane parameters can be calculated using the following equations:

[0089]

[0090] in, The identity matrix, The minimum regularization coefficient is used to prevent the covariance matrix from becoming singular due to the approximate collinear distribution of local point clouds, thus ensuring the numerical stability of matrix inversion.

[0091] After obtaining the globally optimal plane parameters, the ideal coplanar depth within the plane region is recovered. :

[0092]

[0093] in These are the plane parameters of the m-th plane obtained by the least squares method. The inverse of the camera intrinsic parameter matrix. For pixels The homogeneous coordinate representation.

[0094] Finally, introduce optimization into the pipeline. The penalty term minimizes the absolute error between the ideal planar depth and the 3D Gaussian rendered depth. Planar depth consistency loss. Defined as:

[0095]

[0096] in Represents the pixels of all planar prior regions.

[0097] Planar normal consistency constraint: While relying solely on planar depth coplanar constraints can correct the spatial position of Gaussian elements to some extent, the rotation parameters of these elements may still undergo unstable optimization in weakly textured regions, leading to high-frequency fluctuations in the rendered normal map. Applying global smoothing constraints directly to the entire image can easily cross realistic geometric boundaries, causing over-smoothing of structural boundaries such as corners. Therefore, this invention introduces a planar normal consistency constraint, applying smoothing constraints only to normals within the detected planar regions.

[0098] For the A planar region, whose corresponding binary mask is denoted as . When pixel When located within a planar region ,otherwise To avoid smoothing across planar boundaries, constraints are applied only to adjacent pixels located within the same planar region when calculating the normal difference. Specifically, the joint mask in the horizontal and vertical directions is defined as follows: The joint mask in the vertical direction is When two adjacent pixels are both located in the same planar region, the above mask value is 1; otherwise, it is 0. This method can effectively avoid normal smoothing that crosses physical boundaries.

[0099] Based on this, a planar normal consistency loss is constructed by calculating the normal difference between adjacent pixels:

[0100]

[0101] in N This represents the normal map obtained from Gaussian rendering. For the joint mask in the horizontal direction, This is a horizontal mask in the vertical direction.

[0102] By applying the above constraints, normal variations are smoothed only within the planar region, thus ensuring the consistency of normals in the planar region and avoiding over-smoothing of the real geometric boundaries, thereby further improving the stability of the scene's geometric structure.

[0103] (2) Geometric constraints of non-planar regions

[0104] After constraining large structured areas in the indoor scene using planar priors, a large number of non-planar areas still exist, such as furniture, decorative objects, and curved surfaces. For these areas lacking explicit macroscopic priors, geometric constraints based on local tangent plane approximation are introduced. By utilizing neighborhood normal information to locally correct the depth, geometric smoothing of non-planar areas is achieved while preserving the object boundary structure.

[0105] Specifically, the planar prior extracted in step three can be used to extract non-planar regions in the scene. For any pixel within a non-planar region... According to the camera projection model, its line of sight direction ,in For pixels The homogeneous representation of the pixel is then used. Subsequently, the 3x3 pixel local neighborhood of that pixel is extracted. For neighboring pixels Its rendering depth is The normal is The corresponding three-dimensional space point is Under the assumption of local surface continuity, a local tangent plane can be constructed using the normals of neighboring pixels. This can be achieved using the normals of neighboring pixels. The target pixel is calculated from the intersection of the viewing direction and the tangent plane. Ideal depth :

[0106]

[0107] in, For pixels The direction of the normal line is represented. For pixels The corresponding three-dimensional points in space, For pixels The corresponding line of sight.

[0108] To improve the stability of the predicted depth, a weighted fusion of the predicted depths of neighboring pixels is performed. For neighboring pixels... Its weight is defined as ,in Indicates the input image. The color difference attenuation coefficient, The larger the value, the faster the weight of neighboring pixels with large color differences decays; The exponential parameter is used to control geometric similarity. The larger the value, the higher the weight of neighboring pixels with the same normal direction. In this embodiment, and These are manually set hyperparameters, set to 0.5 and 3 respectively in this embodiment. By normalizing and weighting the neighborhood prediction depths, the local tangent plane depth of the target pixel can be obtained. Finally, a geometric consistency loss for non-planar regions is constructed during the optimization process:

[0109]

[0110] in Represents a set of non-planar pixels. This represents the depth value in Gaussian rendering. This represents the depth value obtained by weighted fusion of local neighboring pixels.

[0111] By introducing the aforementioned geometric constraints on non-planar regions, local geometric smoothing of non-planar regions can be performed while maintaining the object's boundary structure, thereby further improving the geometric accuracy of 3D reconstruction of indoor scenes.

[0112] (3) Global geometric constraints

[0113] To constrain the absolute geometric scale of the entire scene, this paper introduces three basic global geometric prior constraints in the global image domain: depth prior constraint, normal prior constraint, and depth-normal consistency constraint.

[0114] Depth Prior Constraints: Based on the depth prior extracted in step two, the depth map rendered by Gaussian is... Apply depth prior loss :

[0115]

[0116] in Represents the set of all pixels in an image. Depth map rendered using Gaussian rendering. This is a depth prior graph.

[0117] Normal prior constraints: Based on the normal priors extracted in step two, the normal map rendered by Gaussian is constrained. N Apply normal prior loss :

[0118]

[0119] in Represents the set of all pixels in an image. N Depth map rendered using Gaussian rendering. This is a depth prior graph.

[0120] Depth normal consistency constraint: In order to ensure that the rendered normal map is consistent N Surface normals derived from the rendered depth map Alignment was performed, and a depth normal consistency constraint was introduced:

[0121]

[0122] in Represents the set of all pixels in an image. N Normal map rendered for Gaussian. This is a surface normal map derived from a depth map rendered by Gaussian rendering.

[0123] Finally, based on the 3D Gaussian splashing differentiable rendering optimization framework, the geometric constraints of planar regions, non-planar regions, and global geometric constraints, along with the photometric consistency loss in the original 3D Gaussian splashing, are combined to form the overall optimization objective function. Let the photometric consistency loss be , then the overall optimization objective function is... It can be represented as:

[0124]

[0125] in These are the weight parameters for each constraint term, used to balance the impact of different constraint terms in the overall optimization. In this embodiment, the aforementioned weight coefficients are manually set hyperparameters. The values ​​are 0.5, 0.1, 0.3, 0.15, 0.2, and 0.3 respectively.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for 3D reconstruction of an indoor Gaussian splash scene based on planar prior guidance, characterized in that: The method includes: Acquire a multi-view image sequence of the indoor scene to be reconstructed and the corresponding camera parameters; Depth prior and normal prior are predicted from the input images of the multi-view image sequence using a monocular depth estimation model and a normal estimation model, respectively. The input image and normal prior map are segmented at multiple granularities using an image segmentation model to generate candidate region masks. The candidate regions are then subjected to geometric consistency checks in conjunction with the normal prior map to extract planar prior regions that satisfy planar structural features. Based on the depth prior and the camera parameters, the image pixels are back-projected into a three-dimensional space to generate a dense point cloud, and the dense point cloud is subjected to voxel downsampling processing. A three-dimensional Gaussian splash model is then constructed based on the downsampled point cloud. Furthermore, within the framework of 3D Gaussian splash differentiable rendering, the 3D Gaussian splash model is optimized through planar region geometric constraints, non-planar region geometric constraints, and global geometric constraints to complete the 3D reconstruction of the scene; The geometric constraints of the planar region include planar depth coplanarity constraints and planar normal consistency constraints; The planar depth coplanar constraint is achieved by applying a planar parameter model fitting and L1 loss to the planar prior region, and the planar normal consistency constraint is achieved by applying normal map gradient smoothing inside the planar prior region.

2. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The steps of using an image segmentation model to perform multi-granularity segmentation on the input image and normal prior map, and generating candidate region masks, include: The image segmentation model is used to perform semantic-level segmentation on the input image to generate semantic-level candidate region masks. Geometric consistency checks are performed on the semantic-level candidate region masks, and regions that pass the check are confirmed as semantic-level planar regions and removed from the remaining valid regions. The image segmentation model is used to perform instance-level segmentation on the remaining effective regions after removing semantic-level planar regions, generating instance-level candidate region masks. Geometric consistency checks are performed on the instance-level candidate region masks, and regions that pass the check are confirmed as instance-level planar regions and removed from the remaining effective regions. The image segmentation model is used to perform component-level segmentation on the remaining effective regions after removing semantic-level planar regions and instance-level planar regions, generating component-level candidate region masks. Geometric consistency checks are performed on the component-level candidate region masks, and regions that pass the checks are confirmed as component-level planar regions. The semantic-level planar regions, the instance-level planar regions, and the component-level planar regions are combined to form a global planar prior mask.

3. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The geometric consistency check includes: Geometric edges are extracted based on the normal prior; for each candidate region mask, it is determined whether it crosses the geometric edge; If the candidate region mask crosses the geometric edge, then the candidate region mask is cut with the geometric edge as the boundary to obtain independent sub-regions; And filter out mask areas with pixel sizes smaller than a preset threshold to obtain the final planar area.

4. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The loss function of the planar depth coplanar constraint Represented as: in, Represents the set of pixels for all planar prior regions. u Represents pixel coordinates, D ( u The ) represents the depth value obtained from rendering the 3D Gaussian splash model. This represents the ideal coplanar depth calculated based on the planar parameters of the aforementioned a priori region.

5. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The loss function of the plane normal consistency constraint Represented as: in N This represents the normal map obtained from Gaussian rendering. For the joint mask in the horizontal direction, This is a horizontal mask in the vertical direction.

6. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The non-planar region geometric constraint achieves local geometric smoothing by constructing a weighted fusion depth based on the normals of local neighboring pixels for pixels within the non-planar region. Its loss function is... Represented as: in, Represents a set of non-planar pixels. This represents the depth value obtained from rendering a 3D Gaussian splash model. Indicates the target pixel The ideal depth is obtained by weighted fusion of neighboring pixels, and the calculation formula is as follows: , For target pixels The set of neighboring pixels, For neighboring pixels The weight, For neighboring pixels normal and three-dimensional coordinates Calculated target pixels The predicted depth.

7. The three-dimensional reconstruction method for a three-dimensional Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The global geometric constraints include depth prior constraints, normal prior constraints, and depth normal consistency constraints. Among them, the loss function of depth prior constraints for Loss function of normal prior constraint for Loss function for depth normal consistency constraints for ; in Represents the set of all pixels in an image. D This indicates the rendering of the depth map. This represents the depth prior graph. N This indicates the rendering of the normal map. Represents the a priori diagram of normals. This represents the surface normal map derived from the rendered depth map.

8. The method for 3D reconstruction of a 3D Gaussian splash indoor scene based on planar prior guidance as described in claim 1, characterized in that: The total loss function for joint optimization of the three-dimensional Gaussian splash model Represented as: in, This refers to the photometric consistency loss within a 3D Gaussian splash differentiable rendering framework. For planar depth coplanar constraint loss, For plane normal consistency constraint loss, For geometric constraint loss in non-planar regions, For deep prior constraint loss, For the normal prior constraint loss, For depth normal consistency constraint loss, , , , , , These are the preset weighting coefficients for each loss item.

9. A 3D reconstruction system for an indoor Gaussian splash scene based on planar prior guidance, characterized in that: The system includes: The image and parameter acquisition module is used to acquire multi-view image sequences of the indoor scene to be reconstructed and the corresponding camera parameters; The prior information extraction module is used to predict depth prior and normal prior from the input image of the multi-view image sequence using a monocular depth estimation model and a normal estimation model, respectively. The planar prior extraction module is used to perform multi-granular segmentation of the input image using an image segmentation model, generate candidate region masks, and perform geometric consistency checks on the candidate regions in conjunction with the normal prior, so as to extract planar prior regions that satisfy planar structural features. The model initialization module is used to back-project image pixels into three-dimensional space to generate a dense point cloud based on the depth prior and the camera parameters, and to perform voxel downsampling on the dense point cloud, and to initialize and construct a three-dimensional Gaussian splash model based on the downsampled point cloud. And a model optimization and reconstruction module, which is used to optimize the three-dimensional Gaussian splash model under the three-dimensional Gaussian splash differentiable rendering framework through planar region geometric constraints, non-planar region geometric constraints and global geometric constraints, so as to complete the three-dimensional reconstruction of the scene; The geometric constraints of the planar region include planar depth coplanarity constraints and planar normal consistency constraints; The planar depth coplanar constraint is achieved by applying a planar parameter model fitting and L1 loss to the planar prior region, and the planar normal consistency constraint is achieved by applying normal map gradient smoothing inside the planar prior region.

Images