Three-dimensional scene reconstruction method and system based on pixel-level Gaussian representation

Abstract

The invention provides a three-dimensional scene reconstruction method and system based on pixel-level Gaussian characterization, and the method comprises the steps: obtaining a multi-view image set of a target scene; inputting the multi-view image set of the target scene into a preset double-branch encoder network, and determining a three-dimensional space position coordinate of each pixel point; generating Gaussian representation of each pixel point according to the three-dimensional space position coordinate of each pixel point; the Gaussian representation of each pixel point is optimized by adopting a surface continuity prior and forced Alpha fusion strategy, and a Gaussian representation set of the optimized pixel points is determined; and according to the optimized Gaussian representation set of the pixel points, a Gaussian splash renderer is adopted to generate an image and a depth map of the reconstructed target scene under the target visual angle. According to the method and the device, the physical constraint and the surface continuity prior constraint are introduced, and the forced Alpha fusion strategy is combined, so that the multi-resolution high-fidelity three-dimensional scene reconstruction is realized under the condition of sparse input.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method and system for three-dimensional scene reconstruction based on pixel-level Gaussian representation. Background Technology

[0002] In existing techniques for 3D scene reconstruction under sparse viewpoint conditions, common methods mainly rely on voxel-based, implicit neural field-based, or Gaussian splash-based representation methods.

[0003] One type of method employs voxel or neural volumetric rendering, such as the Neural Radiance Fields (NeRF) series. These methods typically recover the geometric and appearance information of a scene through dense sampling and volume integration in 3D space. Under sufficient viewpoint constraints, these methods can achieve relatively fine reconstruction results. However, when the number of input viewpoints is limited, insufficient ray constraints can easily cause the model to get trapped in local optima, leading to problems such as geometric blurring and texture floating. Furthermore, volumetric rendering methods have extremely high computational costs, making them difficult to meet the demands of real-time inference and high-resolution rendering.

[0004] Another approach attempts to use 3D Gaussian splatting as a 3D representation, representing the scene as a set of Gaussian primitives with spatial position, scale, rotation, and opacity parameters, and then rendering it quickly using rasterization. For example, in patent "CN119180900A; A 3D Gaussian Splashing Block Rendering Method and System," the overall point cloud is divided into multiple different point cloud blocks, and the points to be rendered are calculated based on these blocks. The layer of the point cloud blocks to be rendered is adaptively adjusted based on hardware computing performance, thus flexibly selecting appropriate points to be rendered and avoiding repeated rendering. It uses Gaussian splatting as the 3D representation, and subsequently employs a block rendering strategy, with each point cloud block trained independently to ensure the size, accuracy, and quality of the point cloud in each block, maintaining smoothness at block junctions. This type of method can significantly improve rendering efficiency and achieve high image fidelity under multi-view conditions. However, existing Gaussian splatting methods often rely on dense multi-view supervision to optimize Gaussian parameters, making stable convergence difficult under sparse input. If the scale, rotation, and opacity parameters of the Gaussian model are directly regressed, the model is prone to degradation problems, that is, all Gaussians gradually converge to high opacity, thereby obscuring the spatial hierarchy and causing the 3D scene to lack the correct depth relationship.

[0005] In addition, existing methods lack constraints on Gaussian continuity, and gaps or discontinuous surfaces exist between Gaussians, which can easily lead to voids or fragmentation in the rendering results.

[0006] Another prominent issue in existing research is that geometric defects and holes within the scene are more pronounced under high-resolution rendering. Due to the uncertainty of depth prediction and the redundancy of representation, the rendering results may exhibit localized overly dark areas or abnormal colors, especially manifesting as unnatural color distributions in the depth map. These problems severely impact the reliability and generalization ability of the model in practical applications.

[0007] In summary, existing technologies have significant shortcomings in terms of sparse input, parameter regression stability, and surface continuity. There is an urgent need for a new method to improve the stability and fidelity of 3D scene reconstruction, while also taking into account the effects and real-time performance of high-resolution rendering. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the purpose of this application is to provide a three-dimensional scene reconstruction method and system based on pixel-level Gaussian representation.

[0009] A first aspect of this application provides a method for reconstructing a 3D scene based on pixel-level Gaussian representation, comprising: Acquire a set of multi-view images of the target scene, wherein the set of multi-view images can be either a sparse set of multi-view images or a dense set of multi-view images; The multi-view image set of the target scene is input into a preset dual-branch encoder network to determine the three-dimensional spatial coordinates of each pixel. The preset dual-branch encoder network includes a multi-view branch network, a monocular branch network, and a depth regression module. Generate a Gaussian representation of each pixel based on its three-dimensional spatial coordinates. The Gaussian representation of each pixel is optimized by employing a surface continuity prior and a forced alpha fusion strategy, thereby determining the optimized Gaussian representation set of the pixels. Based on the optimized Gaussian representation set of the pixels, a Gaussian splash renderer is used to generate an image and depth map of the reconstructed target scene from the target viewpoint.

[0010] Optionally, the step of inputting the multi-view image set of the target scene into a preset dual-branch encoder network to determine the three-dimensional spatial coordinates of each pixel includes: The multi-view image set of the target scene is input into the multi-view branch network to construct the cost body and perform multi-view depth prior prediction to determine the multi-view depth prior prediction result of the multi-view image set of the target scene. The multi-view image set of the target scene is input into the monocular branch network, and monocular depth prior prediction is performed on the image of each view to determine the monocular depth prior prediction result of the multi-view image set of the target scene. The multi-view depth prior prediction results of the multi-view image set of the target scene and the monocular depth prior prediction results of the multi-view image set of the target scene are weighted and fused to determine the scene input features; The scene input features are input into the depth regression module to determine the three-dimensional spatial coordinates of each pixel.

[0011] Optionally, the step of inputting the multi-view image set of the target scene into the multi-view branch network, constructing a cost volume and performing multi-view depth prior prediction, and determining the multi-view depth prior prediction result of the multi-view image set of the target scene includes: Perform viewpoint alignment on the multi-view image set of the target scene to determine the image set of each viewpoint after viewpoint alignment; Construct a cost volume based on the image set of each viewpoint after viewpoint alignment; Based on the cost body, a preset 2D U-Net network is used to perform multi-view depth prior prediction on the image set of each view after view alignment, and the multi-view depth prior prediction result of the multi-view image set of the target scene is determined.

[0012] Optionally, generating the reconstructed image and depth map of the target scene from the target viewpoint using a Gaussian splash renderer based on the optimized Gaussian representation set of the pixels includes: The optimized Gaussian representation set of the pixels is converted into a renderable 3D scene representation, and a Gaussian set with the attribute of the regression result is constructed. The Gaussian set whose attributes are the regression results is input into the Gaussian splash renderer to generate an image and depth map of the reconstructed target scene from the target viewpoint. A second aspect of this application provides a 3D scene reconstruction system based on pixel-level Gaussian representation, comprising: A multi-view image acquisition module is used to acquire a set of multi-view images of a target scene, wherein the set of multi-view images can be a sparse set of multi-view images or a dense set of multi-view images. The three-dimensional spatial position determination module for pixels is used to input the multi-view image set of the target scene into a preset dual-branch encoder network to determine the three-dimensional spatial position coordinates of each pixel. The preset dual-branch encoder network includes a multi-view branch network, a monocular branch network, and a depth regression module. A pixel-level Gaussian characterization determination module is used to generate a Gaussian characterization of each pixel based on the three-dimensional spatial coordinates of each pixel. A pixel-level Gaussian representation optimization module is used to optimize the Gaussian representation of each pixel by employing a surface continuity prior and a forced Alpha fusion strategy, and to determine the set of Gaussian representations of the optimized pixels. The 3D scene reconstruction module is used to generate an image and depth map of the reconstructed target scene from the target viewpoint using a Gaussian splash renderer, based on the optimized Gaussian representation set of the pixels.

[0013] A third aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0014] A fourth aspect of this application provides an electronic device comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of any of the methods provided in the first aspect of this application.

[0015] The pixel-level Gaussian representation-based 3D scene reconstruction method of this application determines the 3D spatial coordinates of each pixel by using a pre-defined dual-branch coding network, and then derives Gaussian properties through physical consistency constraints. This eliminates the need for direct regression of Gaussian parameters, improving the cross-view geometric consistency of 3D scene reconstruction under sparse input conditions. It introduces surface continuity priors to enhance the structural constraints of adjacent points in space, generating a coherent scene surface, reducing noise and holes in the reconstructed target scene, and improving geometric stability. Combined with a forced alpha fusion strategy, it unifies the Gaussian representation of the projection plane, improves the spatial consistency of images generated from different viewpoints, reduces the risk of misalignment, ghosting, and other issues from different viewpoints, and improves the consistency between the rendered image and the depth map.

[0016] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a three-dimensional scene reconstruction method based on pixel-level Gaussian representation according to an exemplary embodiment.

[0018] Figure 2 This is a schematic diagram illustrating the structure of a 3D scene reconstruction system based on pixel-level Gaussian representation according to an exemplary embodiment. Detailed Implementation

[0019] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0020] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0021] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0022] Existing methods for 3D scene reconstruction under sparse viewpoint conditions mainly employ volumetric rendering and rasterization rendering using Gaussian splashing as the 3D representation. The former, when the number of input viewpoints is limited, is prone to getting trapped in local optima, leading to geometric blurring and texture floating, making it difficult to meet the requirements of real-time inference and high-resolution rendering. The latter relies on dense multi-view supervision to optimize Gaussian parameters, which struggles to converge stably under sparse input. Furthermore, existing methods lack constraints on Gaussian continuity, and geometric defects and holes exist within the scene under high-resolution rendering. To address these issues, this application provides a 3D scene reconstruction method based on pixel-level Gaussian representation to resolve these problems.

[0023] Figure 1 This is a flowchart illustrating a three-dimensional scene reconstruction method based on pixel-level Gaussian representation according to an exemplary embodiment.

[0024] Reference Figure 1 As shown in the figure, one embodiment of this application provides a three-dimensional scene reconstruction method based on pixel-level Gaussian representation, including S11 to S15.

[0025] S11, Obtain a set of multi-view images of the target scene.

[0026] Specifically, the multi-view image set can be either a sparse or a dense set. The number of images in the multi-view image set can be either sparse or relatively dense.

[0027] S12: Input the multi-view image set of the target scene into the preset dual-branch encoder network to determine the three-dimensional spatial coordinates of each pixel.

[0028] Specifically, the preset dual-branch encoder network includes a multi-view branch network, a monocular branch network, and a depth regression module.

[0029] Among them, the multi-view branch network is used to construct the cost volume and predict the multi-view depth prior, the monocular branch network is used to predict the monocular depth prior, and the depth regression module is used to directly obtain the three-dimensional spatial coordinates of each pixel point from the two-dimensional image, realizing the mapping from two-dimensional visual information to three-dimensional spatial information.

[0030] S13 generates a Gaussian representation of each pixel based on its three-dimensional spatial coordinates.

[0031] Specifically, Gaussian representations of pixels are derived based on their three-dimensional spatial coordinates through physical consistency constraints.

[0032] S14. The Gaussian representation of each pixel is optimized by using a surface continuity prior and a forced Alpha fusion strategy to determine the set of Gaussian representations of the optimized pixels.

[0033] S15, based on the optimized Gaussian representation set of pixels, uses a Gaussian splash renderer to generate an image and depth map of the target scene from the target viewpoint.

[0034] The embodiments described above in this application determine the three-dimensional spatial coordinates of each pixel by employing a preset dual-branch coding network, and then derive Gaussian properties through physical consistency constraints. This eliminates the need for direct regression of Gaussian parameters, improving cross-view geometric consistency in 3D scene reconstruction under sparse input conditions. By introducing surface continuity priors, the structural constraints of adjacent points in space are enhanced, generating a coherent scene surface, reducing noise and holes in the reconstructed target scene, and improving geometric stability. Combined with a forced alpha fusion strategy, the Gaussian representation of the projection plane is uniformly constrained, improving the spatial consistency of images generated from different viewpoints, reducing the risk of misalignment, ghosting, and other issues from different viewpoints, and improving the consistency between the rendered image and the depth map.

[0035] To obtain multi-view input for 3D scene reconstruction of the target scene, in some specific embodiments of this application, for S11, obtaining a set of multi-view images of the target scene can be achieved by: A sparse or dense set of multi-view images is used as the multi-view image set for the target scene.

[0036] Among them, a sparse multi-view image set means that the number of images contained in the multi-view image set is sparse, and a dense multi-view image set means that the number of images contained in the multi-view image set is dense.

[0037] The 3D scene reconstruction method based on pixel-level Gaussian representation provided in this application is applicable to 3D scene reconstruction under sparse input conditions as well as 3D scene reconstruction under dense input conditions.

[0038] To enable subsequent data processing, in some specific embodiments of this application, normalization processing is performed on the images in the multi-view image set of the target scene. Specifically, the normalization operation may include pixel value normalization operation, resolution unification operation, and data augmentation operation.

[0039] In order to unify the multi-view images in a multi-view image set into a unified coordinate system, in some specific embodiments of this application, the intrinsic and extrinsic parameters of the image acquisition device for acquiring multi-view image sets of the target scene are standardized, thereby ensuring that the multi-view images are comparable in a unified coordinate system.

[0040] In order to achieve feature extraction from multiple and single perspectives and prediction of the three-dimensional spatial position corresponding to dense pixels, in some specific embodiments of this application, for S12, the multi-view image set of the target scene is input into a preset dual-branch encoder network to determine the three-dimensional spatial position coordinates of each pixel, which can be implemented using S121 to S124.

[0041] S121, input the multi-view image set of the target scene into the multi-view branch network, construct the cost volume and perform multi-view depth prior prediction, and determine the multi-view depth prior prediction result of the multi-view image set of the target scene.

[0042] Specifically, in some embodiments of this application, S121 may include S1211 to S1213.

[0043] S1211, perform viewpoint alignment on the multi-view image set of the target scene, and determine the image set of each viewpoint after viewpoint alignment.

[0044] For example, a traditional feature matcher based on manually designed feature extraction logic and matching rules is used to extract and detect features from images in a multi-view image set of a target scene, match images with the same viewpoint, and establish a correspondence to align the images in the multi-view image set.

[0045] For example, a learned feature matcher can be used instead of a traditional feature matcher. The feature matcher is trained by annotating images using deep learning methods, automatically matching images from the same viewpoint and establishing a correspondence to perform viewpoint alignment on images in a multi-view image set.

[0046] S1212, construct the cost volume based on the set of images for each viewpoint that has been aligned with the viewpoint.

[0047] Specifically, a shallow CNN module with a residual structure is used to extract 4x downsampled image features from the image set of each viewpoint. Then, a multi-view deformer, including self-attention and cross-attention layers, is used to exchange image features between different viewpoints to obtain image features F. Based on the known camera projection matrix, a depth candidate layer is used... d Image features F from the perspective j Project to i Obtain the mean of the features .

[0048] Specifically, the cost body is represented as follows: ; in, The pixel coordinates of the feature map in the multi-view image set are represented, and d represents the candidate depth layer. Indicates the first i Image features from various perspectives Represents the mean of image features. Indicates the depth of the candidate layer d The cost body, Indicates the first i Image features at candidate depth layer d from various perspectives This represents the mean of image features at candidate depth layer d.

[0049] The cost volume grid can be used to represent the distance between feature layers. The smaller the distance, the more reliable the depth candidate value d of the depth candidate layer.

[0050] In this embodiment, the cost body is used to capture geometric consistency across viewpoints.

[0051] S1213, Based on the cost volume, a pre-defined 2D U-Net network is used to perform multi-view depth prior prediction on the image set of each view after view alignment, and the multi-view depth prior prediction result of the multi-view image set of the target scene is determined.

[0052] Specifically, the multi-view depth prior prediction results are as follows: .

[0053] S122, input the multi-view image set of the target scene into the monocular branch network, perform monocular depth prior prediction on the image of each view, and determine the monocular depth prior prediction result of the multi-view image set of the target scene.

[0054] Specifically, the monocular branch network can employ a pre-trained deep neural network, taking a monocular image as input (i.e., an image from a multi-view image set of the target scene) and outputting a monocular depth image as output. .

[0055] For example, a pre-trained deep neural network can use depth anything v2. This embodiment uses a monocular branch network to perform monocular depth prior prediction and obtain the monocular depth prior prediction result, which can enhance the robustness of the pixel-level Gaussian representation-based 3D scene reconstruction method proposed in this application under sparse input conditions.

[0056] S123, weighted fusion of the multi-view depth prior prediction results of the multi-view image set of the target scene and the monocular depth prior prediction results of the multi-view image set of the target scene to determine the scene input features.

[0057] Specifically, the multi-view depth prior prediction results and the monocular depth prior prediction results are fused based on preset dynamic adjustment coefficients, and the specific formula is as follows: ; Where D represents the scene input features, This represents the preset dynamic adjustment coefficient. This represents the multi-view depth prior prediction results. This represents the monocular depth prior prediction result. Represents the pixel coordinates of the feature map in a multi-view image set.

[0058] S124, input the scene input features into the depth regression module to determine the three-dimensional spatial coordinates of each pixel.

[0059] Specifically, the depth regression module in this embodiment can be used to directly predict the three-dimensional spatial coordinates p = (X,Y,Z) of a pixel.

[0060] The embodiments described above employ a multi-view branch network in a pre-defined dual-branch encoder network to construct a cost volume and extract multi-view depth prior features from a multi-view image set of the target scene. Cross-view geometric consistency is explicitly encoded. A monocular branch network is used to extract monocular depth prior features from the image of each viewpoint, compensating for insufficient geometric cues from the viewpoint. The two are then deeply fused, and consistency supervision is applied across multiple resolutions. This enables steady regression of the 3D spatial position of pixels within an end-to-end framework, directly predicting the 3D spatial position corresponding to dense pixels. Various properties of Gaussians are derived through physical consistency constraints, thus avoiding direct regression of Gaussian parameters and effectively preventing instability caused by direct parameter regression. This improves the stability of the geometric structure of Gaussian distribution in 3D space, stably recovering continuous geometry and realistic appearance under sparse input conditions. It significantly reduces distortion phenomena such as "scattering upon close inspection," "color bias," and "surface perforation," while maintaining high consistency and detail fidelity in new viewpoint rendering.

[0061] To achieve pixel-level Gaussian representation, in some specific embodiments of this application, for S13, generating a Gaussian representation for each pixel based on its three-dimensional spatial coordinates may include: The three-dimensional spatial coordinates of each pixel are used as the initial center position of its corresponding Gaussian point; Based on the continuity of 3D assets, that is, the three-dimensional positions of adjacent pixels are continuously distributed in space, neighboring pixels are selected for each pixel to construct a set of neighboring pixels for each pixel. The orientation attribute of the Gaussian is obtained by taking the derivative of the position attribute of the neighboring Gaussian based on the set of neighboring pixels of each pixel, and the size attribute of the Gaussian is obtained by taking the cross product. The Gaussian representation of each pixel is determined based on the initial center position, orientation attribute, and size attribute of the Gaussian corresponding to each pixel.

[0062] The Gaussian representation of each pixel is as follows: ; in, Indicates the first i Gaussian representation of each pixel. Indicates the first i The opacity represented by a Gaussian symbol for each pixel. Indicates the first i The spatial position of each pixel Indicates the first i The Gaussian representation of the center position coordinates of each pixel.

[0063] The Gaussian properties of each pixel include the Gaussian scale parameter, the Gaussian rotation direction, the Gaussian opacity, and the Gaussian color or appearance coefficient. The generation of Gaussian properties is constrained by physical consistency, thereby avoiding instability caused by direct regression parameters.

[0064] To optimize the Gaussian representation of each pixel, surface continuity prior constraints and transparency constraints are introduced. In some specific embodiments of this application, for S14, a surface continuity prior and forced alpha fusion strategy is used to optimize the Gaussian representation of each pixel, and the optimized set of Gaussian representations of the pixels is determined. This can be achieved by: The prior for surface continuity is as follows: ; in, Represents the surface continuity loss function. This represents the Gaussian representation index of the selected pixel within the neighborhood, where N represents the spatial neighborhood. Indicates the first i The Gaussian representation of the center position coordinates of each pixel Indicates the first j The Gaussian representation of the center position coordinates of each pixel; A surface continuity prior is used, i.e., the surface continuity loss function is used to optimize the Gaussian representation of each pixel: The local tangent vector is calculated in the pixel neighborhood and the normal vector is obtained by cross product. The rotation parameters of Gauss are aligned to the normal vector using the Rodriguez formula. The scale is estimated by the variance of the local tangent vector and a bounded scale multiplier is introduced for learning, replacing direct regression with a physical consistency derivation from "position → (rotation, scale)". This achieves a Gaussian representation where the orientation and shape change smoothly with the surface, naturally satisfying spatial continuity.

[0065] Compared to approximating a scene as an isotropic point cloud, this embodiment uses anisotropic two-dimensional Gaussian (2DGS) as the basic unit, explicitly aligned to the local surface tangent space. This can accurately capture high-frequency geometric elements such as thin slices, texture directions, and elongated structures. The reconstructed 3D scene exhibits more coherent surfaces, fewer holes and jagged edges, and maintains geometric stability even under close-range and high-resolution rendering.

[0066] The Gaussian representation of each pixel is optimized using a forced alpha fusion strategy: ; Where C represents the rendered color. Indicates the first i The color represented by a Gaussian of each pixel Indicates the first iThe opacity represented by a Gaussian symbol for each pixel. Indicates the first position on the rendering path j The opacity represented by a Gaussian symbol for each pixel. Indicates rendering opacity. This indicates the preset threshold for rendering opacity.

[0067] In the embodiments described above, the Gaussian representation of each pixel is optimized using a surface continuity prior, resulting in stronger structural constraints on adjacent points in space. This generates a coherent scene surface, reduces noise and holes in the reconstructed 3D scene, and ensures a smooth and coherent geometric surface of the reconstructed target scene. A forced alpha blending strategy is employed to uniformly constrain the coverage of Gaussian primitives on the projection plane, ensuring spatial consistency of images generated from different viewpoints and preventing all Gaussian representations from converging to opacity. This ensures the depth and spatial structure integrity of the reconstructed target scene. The introduction of the forced alpha blending strategy in the representation optimization and rendering stages significantly reduces the risk of misalignment, ghosting, or asymmetry under different viewpoints, thereby improving the consistency between the rendered image and the depth map.

[0068] In the rendering stage, the three-dimensional scene is reconstructed. In some specific embodiments of this application, for S15, an image and depth map of the reconstructed target scene from the target viewpoint are generated using a Gaussian splash renderer based on the optimized set of Gaussian representations of the pixels. This can be achieved using S151 to S152.

[0069] S151 converts the optimized set of Gaussian representations of pixels into a renderable 3D scene representation.

[0070] Specifically, the obtained Gaussian properties are stored in the form of a Gaussian set.

[0071] It should be noted that the Gaussian opacity is further modulated before storage, polarizing it into a completely opaque Gaussian and a completely invisible Gaussian. This operation can prevent multiple layers of aliasing when the scene is viewed from an ood perspective.

[0072] A renderable 3D scene is represented as a reconstructed 3D scene.

[0073] S152 inputs a renderable 3D scene into the Gaussian splash renderer to generate an image and depth map of the reconstructed target scene from the target viewpoint.

[0074] Specifically, the Gaussian splash renderer in this embodiment introduces a multi-resolution rendering mechanism during the training phase and designs a high-resolution reconstruction consistency index to effectively constrain the Gaussian splash renderer, so that the reconstructed 3D scene maintains robustness and consistency at different resolutions, avoiding the problem of geometric defect magnification under high-resolution rendering.

[0075] The embodiments described above generate a reconstructed 3D scene representation based on an optimized Gaussian representation set of pixels, maintaining high fidelity in geometric structure while possessing real-time rendering capabilities. The Gaussian splash renderer generates images and depth maps from the target viewpoint, which can be directly applied to virtual reality, augmented reality, robot navigation, and 3D modeling application scenarios.

[0076] This application proposes a 3D scene reconstruction method based on pixel-level Gaussian representation. By introducing physical constraints and continuity priors, combined with a forced alpha fusion strategy, it achieves high-fidelity and generalizable 3D scene reconstruction at multiple resolutions.

[0077] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0078] Figure 2 This is a schematic diagram illustrating the structure of a 3D scene reconstruction system based on pixel-level Gaussian representation according to an exemplary embodiment.

[0079] Reference Figure 2 As shown in one embodiment of this application, a three-dimensional scene reconstruction system 100 based on pixel-level Gaussian representation is provided, including: a multi-view image acquisition module 110, a three-dimensional spatial position determination module 120 for pixels, a pixel-level Gaussian representation determination module 130, a pixel-level Gaussian representation optimization module 140, and a three-dimensional scene reconstruction module 150.

[0080] The multi-view image acquisition module 110 is used to acquire a set of multi-view images of the target scene. The multi-view image set can be a sparse multi-view image set or a dense multi-view image set. The pixel three-dimensional spatial position determination module 120 is used to input a set of multi-view images of the target scene into a preset dual-branch encoder network to determine the three-dimensional spatial position coordinates of each pixel. The preset dual-branch encoder network includes a multi-view branch network, a monocular branch network and a depth regression module. The pixel-level Gaussian characterization determination module 130 is used to generate a Gaussian characterization of each pixel based on the three-dimensional spatial coordinates of each pixel. The pixel-level Gaussian representation optimization module 140 is used to optimize the Gaussian representation of each pixel by adopting a surface continuity prior and a forced Alpha fusion strategy, and to determine the set of Gaussian representations of the optimized pixels. The 3D scene reconstruction module 150 is used to generate an image and depth map of the reconstructed target scene from the target viewpoint using a Gaussian splash renderer, based on an optimized set of Gaussian representations of pixels.

[0081] The embodiments described above in this application determine the three-dimensional spatial coordinates of each pixel by employing a preset dual-branch coding network, and then derive Gaussian properties through physical consistency constraints. This eliminates the need for direct regression of Gaussian parameters, improving cross-view geometric consistency in 3D scene reconstruction under sparse input conditions. By introducing surface continuity priors, the structural constraints of adjacent points in space are enhanced, generating a coherent scene surface, reducing noise and holes in the reconstructed target scene, and improving geometric stability. Combined with a forced alpha fusion strategy, the Gaussian representation of the projection plane is uniformly constrained, improving the spatial consistency of images generated from different viewpoints, reducing the risk of misalignment, ghosting, and other issues from different viewpoints, and improving the consistency between the rendered image and the depth map.

[0082] Regarding the embodiments of the above system, the specific ways in which each module performs operations have been described in detail in the embodiments of the method, and will not be elaborated here.

[0083] Based on the same technical concept, in some specific embodiments of this application, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and a method that the processor can use to execute when executing the program.

[0084] Based on the same technical concept, in some specific embodiments of this application, a computer-readable storage medium is provided on which a computer program is stored, which can be used to execute a method when the program is executed by a processor.

[0085] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0086] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0087] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0088] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A method for reconstructing a 3D scene based on pixel-level Gaussian representation, characterized in that, include: Acquire a set of multi-view images of the target scene, wherein the set of multi-view images can be either a sparse set of multi-view images or a dense set of multi-view images; The multi-view image set of the target scene is input into a preset dual-branch encoder network to determine the three-dimensional spatial coordinates of each pixel. The preset dual-branch encoder network includes a multi-view branch network, a monocular branch network, and a depth regression module. Generate a Gaussian representation of each pixel based on its three-dimensional spatial coordinates. The Gaussian representation of each pixel is optimized by employing a surface continuity prior and a forced alpha fusion strategy, thereby determining the optimized Gaussian representation set of the pixels. Based on the optimized Gaussian representation set of the pixels, a Gaussian splash renderer is used to generate an image and depth map of the reconstructed target scene from the target viewpoint.

2. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 1, characterized in that, The step of inputting the multi-view image set of the target scene into a preset dual-branch encoder network to determine the three-dimensional spatial coordinates of each pixel includes: The multi-view image set of the target scene is input into the multi-view branch network to construct the cost volume and perform multi-view depth prior prediction to determine the multi-view depth prior prediction result of the multi-view image set of the target scene. The multi-view image set of the target scene is input into the monocular branch network, and monocular depth prior prediction is performed on the image of each view to determine the monocular depth prior prediction result of the multi-view image set of the target scene. The multi-view depth prior prediction results of the multi-view image set of the target scene and the monocular depth prior prediction results of the multi-view image set of the target scene are weighted and fused to determine the scene input features; The scene input features are input into the depth regression module to determine the three-dimensional spatial coordinates of each pixel.

3. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 2, characterized in that, The step of inputting the multi-view image set of the target scene into the multi-view branch network, constructing a cost volume and performing multi-view depth prior prediction, and determining the multi-view depth prior prediction result of the multi-view image set of the target scene includes: Perform viewpoint alignment on the multi-view image set of the target scene to determine the image set of each viewpoint after viewpoint alignment; Construct a cost volume based on the image set of each viewpoint after viewpoint alignment; Based on the cost body, a preset 2D U-Net network is used to perform multi-view depth prior prediction on the image set of each view after view alignment, and the multi-view depth prior prediction result of the multi-view image set of the target scene is determined.

4. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 3, characterized in that, The step of constructing a cost body based on the set of images for each viewpoint aligned by viewpoint includes: ； in, The pixel coordinates of the feature map in the multi-view image set are represented by d, where d represents the candidate depth layer. Indicates the first i Image features from various perspectives Represents the mean of image features. Indicates the depth of the candidate layer d The cost body, Indicates the first i Image features at candidate depth layer d from various perspectives This represents the mean of image features at candidate depth layer d.

5. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 1, characterized in that, The step of generating a Gaussian representation of each pixel based on its three-dimensional spatial coordinates includes: ； in, Indicates the first i Gaussian representation of pixels. Indicates the first i The opacity represented by a Gaussian symbol for each pixel. Indicates the first i The spatial position of each pixel Indicates the first i The Gaussian representation of the center position coordinates of each pixel.

6. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 1, characterized in that, The optimization of the Gaussian representation of each pixel using a surface continuity prior and a forced alpha fusion strategy, to determine the optimized Gaussian representation set of the pixels, includes: The Gaussian representation of each pixel is optimized using the aforementioned surface continuity prior: ； in, Represents the surface continuity loss function. This represents the Gaussian representation index of the selected pixel within the neighborhood, where N represents the spatial neighborhood. Indicates the first i The Gaussian representation of the center position coordinates of each pixel Indicates the first j The Gaussian representation of the center position coordinates of each pixel; The Gaussian representation of each pixel is optimized using the forced alpha fusion strategy: ； Where C represents the rendered color. Indicates the first i The color represented by a Gaussian of each pixel Indicates the first i The opacity represented by a Gaussian symbol for each pixel. Indicates the first position on the rendering path j The opacity represented by a Gaussian symbol for each pixel. Indicates rendering opacity. This indicates the preset threshold for rendering opacity.

7. The 3D scene reconstruction method based on pixel-level Gaussian representation according to claim 1, characterized in that, The step of generating a reconstructed image and depth map of the target scene from the target viewpoint using a Gaussian splash renderer based on the optimized Gaussian representation set of the pixels includes: The optimized Gaussian representation set of the pixels is converted into a renderable 3D scene representation, and a Gaussian set with the attribute of the regression result is constructed. The Gaussian set whose attributes are the regression results is input into the Gaussian splash renderer to generate an image and depth map of the reconstructed target scene from the target viewpoint.

8. A 3D scene reconstruction system based on pixel-level Gaussian representation, characterized in that, include: A multi-view image acquisition module is used to acquire a set of multi-view images of a target scene, wherein the set of multi-view images can be a sparse set of multi-view images or a dense set of multi-view images. The three-dimensional spatial position determination module for pixels is used to input the multi-view image set of the target scene into a preset dual-branch encoder network to determine the three-dimensional spatial position coordinates of each pixel. The preset dual-branch encoder network includes a multi-view branch network, a monocular branch network, and a depth regression module. A pixel-level Gaussian characterization determination module is used to generate a Gaussian characterization of each pixel based on the three-dimensional spatial coordinates of each pixel. A pixel-level Gaussian representation optimization module is used to optimize the Gaussian representation of each pixel by employing a surface continuity prior and a forced Alpha fusion strategy, and to determine the set of Gaussian representations of the optimized pixels. The 3D scene reconstruction module is used to generate an image and depth map of the reconstructed target scene from the target viewpoint using a Gaussian splash renderer, based on the optimized Gaussian representation set of the pixels.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images