Three-dimensional scene reconstruction method, system, device, and program product

Abstract

This application provides a method, system, device, and program product for 3D scene reconstruction. The method includes: performing Gaussian 3D reconstruction based on a set of 2D images to obtain an initial Gaussian point cloud, wherein the parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector; rendering a rendered image based on the initial Gaussian point cloud; constructing a total loss, which includes reconstruction loss, scaling constraint loss, and normal consistency loss; and optimizing the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss to obtain an optimized Gaussian point cloud. Specifically, optimizing the scaling constraint loss reduces the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratios, and optimizing the normal consistency loss improves the consistency between the predicted normal vector and the geometric normal vector. This application is beneficial for improving the geometric modeling accuracy of 3D scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a three-dimensional scene reconstruction method, system, device and program product. Background Technology

[0002] In recent years, significant progress has been made in novel view synthesis techniques based on Neural Radiance Fields (NeRF). Among them, 3D Gaussian Splatting (3DGS) technology (e.g., "3D Gaussian Splatting for Real-Time Radiance Field Rendering" presented by Kerbl et al. at the 2023 SIGGRAPH conference) represents a major breakthrough in this field due to its superior rendering quality and unprecedented real-time rendering speed (e.g., achieving >100 FPS on consumer-grade GPUs). 3DGS technology explicitly represents 3D scenes using millions of colored 3D Gaussian ellipsoids and is optimized through a differentiable rendering pipeline, achieving photorealistic rendering effects.

[0003] However, despite the success of 3DGS in terms of visual fidelity and rendering speed, its underlying representation methods suffer from some serious, unresolved fundamental flaws that severely limit its deployment in interactive applications. The optimization objective function of 3DGS only cares about whether the rendered 2D pixel colors match the training image, without imposing any explicit constraints on the scene's "realistic 3D geometry." To fit colors, the optimizer tends to generate fluffy Gaussian ellipsoids that are loosely overlapped in space rather than tightly fitted to the object's true surface. This "fluffy" volumetric representation results in noticeable "ghosting" or "fog" artifacts when viewed up close, at close-up, or from a grazing angle, making the scene lack a sense of "solidity." Existing 3DGS struggles to represent sharp edges, fine grooves, or smooth surfaces, and its geometric quality is far inferior to traditional mesh models. While existing techniques (such as 2DGS) attempt to improve the geometry of 3DGS by introducing geometric priors (such as surface normals or depth maps), they typically sacrifice rendering quality. Summary of the Invention

[0004] In view of the problems in the prior art, the purpose of this application is to provide a three-dimensional scene reconstruction method, system, device and program product, which is beneficial to improving the geometric modeling accuracy of three-dimensional scene reconstruction.

[0005] The first aspect of this application provides a three-dimensional scene reconstruction method, including the following steps: Gaussian 3D reconstruction is performed based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. The rendered image is obtained based on the initial Gaussian point cloud rendering; The total loss is constructed, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the 2D image set. The scaling constraint loss is constructed based on the scaling factor of each Gaussian point. The normal consistency loss is constructed based on the predicted normal vector and the geometric normal vector of each Gaussian point. The parameters of each Gaussian point in the initial Gaussian point cloud are optimized based on the total loss to obtain the optimized Gaussian point cloud. Specifically, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss.

[0006] In some embodiments, the scaling constraint loss is configured to reduce the difference between the z-axis scaling ratio and the target scaling ratio of each Gaussian point, the target scaling ratio being obtained based on the product of the minimum of the x-axis scaling ratio and the y-axis scaling ratio of the Gaussian point and a preset coefficient, the preset coefficient being less than 1.

[0007] In some embodiments, the method further includes: rendering a depth map based on an initial Gaussian point cloud, wherein the pixel value of each pixel in the depth map represents the depth of the Gaussian point corresponding to that pixel; The geometric normal vectors of each Gaussian point are obtained by calculating the gradient based on the depth map.

[0008] In some embodiments, the parameters of each Gaussian point in the initial Gaussian point cloud further include an instance feature vector; the method further includes the following steps: An instance segmentation model is used to segment a set of two-dimensional images, and an instance ID is assigned to each pixel in each two-dimensional image of the set. The total loss also includes instance semantic consistency loss, which optimizes instance feature vectors by optimizing instance semantic consistency loss, so that instance feature vectors of Gaussian points with the same instance ID are aggregated, while instance feature vectors of Gaussian points with different instance IDs are far apart.

[0009] In some embodiments, the instance semantic consistency loss is constructed using the following steps: A feature map is obtained by rendering based on the instance feature vectors in the initial Gaussian point cloud. The pixel value of each pixel in the feature map represents the rendering feature of that pixel. An instance semantic consistency loss is constructed based on the rendering feature relationship of pixels with the same instance ID and the rendering feature relationship of pixels with different instance IDs. The instance feature vector is optimized by optimizing the instance semantic consistency loss, so that the rendering features of pixels with the same instance ID are aggregated and the rendering features of pixels with different instance IDs are far apart.

[0010] In some embodiments, an instance segmentation model is used to segment the two-dimensional image set into instances, assigning an instance ID to each pixel in each two-dimensional image of the two-dimensional image set, including the following steps: The two-dimensional image set is segmented using an instance segmentation model, and an initial instance ID is assigned to each pixel in each two-dimensional image of the two-dimensional image set. For each Gaussian point, record the two-dimensional image to which the Gaussian point is projected and the coordinates of the projected pixels in the corresponding two-dimensional image, and determine the initial instance ID corresponding to the projected pixel coordinates. For each Gaussian point, if there are multiple different initial instance IDs, the relationship between the multiple initial IDs corresponding to the same Gaussian point is scored, and the relationship scores between all pairs of initial instance IDs are calculated. All initial instance IDs are used as graph nodes. An edge is built based on the relationship score between each pair of initial instance IDs to construct a global association graph. Nodes with relationship scores greater than a preset relationship threshold are connected. Perform connectivity component analysis on the global association graph, map all initial instance IDs within the same connectivity component to the same instance ID, and obtain the updated final instance ID.

[0011] In some embodiments, after optimizing the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss, the following steps are further included: The current view from the user's perspective is rendered based on the optimized Gaussian point cloud; Upon receiving the user's selection of pixels in the current view, determine the selected Gaussian point corresponding to the selected pixel; The similarity between the instance feature vector of the selected Gaussian point and the instance feature vector of other Gaussian points is calculated, and Gaussian points with similarity greater than a preset similarity threshold are selected from other Gaussian points to form a set of Gaussian points. Based on the pose transformation information input by the user, the poses of the selected Gaussian point and each Gaussian point in the set of Gaussian points are updated, and the updated 3D scene is re-rendered.

[0012] A second aspect of this application also provides a three-dimensional scene reconstruction system for implementing the three-dimensional scene reconstruction method of the first aspect, the system comprising: The reconstruction module is used to perform Gaussian 3D reconstruction based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. The rendering module is used to obtain a rendered image based on the initial Gaussian point cloud; The construction module is used to construct the total loss, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the 2D image set, the scaling constraint loss is constructed based on the scaling factor of each Gaussian point, and the normal consistency loss is constructed based on the predicted normal vector and geometric normal vector of each Gaussian point. The training module is used to optimize the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss, so as to obtain the optimized Gaussian point cloud. Specifically, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss.

[0013] A third aspect of this application also provides a three-dimensional scene reconstruction device, comprising: processor; Memory, which stores the processor's executable instructions; The processor is configured to execute the steps of the aforementioned 3D scene reconstruction method by executing executable instructions.

[0014] The fourth aspect of this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described three-dimensional scene reconstruction method.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0016] The three-dimensional scene reconstruction method, system, equipment, and program products of this application have the following beneficial effects: In the 3D scene reconstruction method of this application, during the training process, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratios is reduced by optimizing the scaling constraint loss, making the z-axis as short as possible relative to the x-axis and y-axis. This abandons the assumption in traditional 3DGS that the Gaussian volume can arbitrarily expand into a sphere, and forcibly constrains the Gaussian volume to be extremely compressed in the z-axis direction of the local coordinate system, forming an ellipsoidal structure. The Gaussian sphere is then converted into a quasi-planar Gaussian primitive to obtain a 3D Gaussian sphere cake, making the Gaussian sphere cake closer to the real surface of the object while maintaining high fidelity and high rendering quality; simultaneously, through Optimizing the normal consistency loss improves the consistency between predicted and geometric normals, forcing the Gaussian orientation to align with the object's true surface normal. This avoids the problems of arbitrary Gaussian ellipsoid orientation and loose spatial overlap, making the Gaussian points geometrically infinitely close to the object's surface. This ensures the Gaussian point cloud closely fits the solid surface, eliminating the fog / ghost artifacts of traditional 3DGS and restoring details such as sharp edges and fine grooves. This lays the core geometric foundation for subsequent extraction of smooth, high-precision watertight meshes, resulting in smooth, detailed mesh surfaces with significantly reduced geometric errors. This application's method balances rendering quality and geometric accuracy, integrating the geometric advantages of 2DGS with the rendering efficiency of 3DGS. The Z-axis compression strategy maintains the rasterization pipeline without sacrificing rendering speed, while also improving model interpretability. Attached Figure Description

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart of a three-dimensional scene reconstruction method according to an embodiment of this application; Figure 2 This is a structural block diagram of a three-dimensional scene reconstruction system according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device according to an embodiment of this application. Detailed Implementation

[0019] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0020] Furthermore, the accompanying drawings are merely illustrative of this application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0021] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0022] To address the technical problems in existing technologies, this application proposes a novel 3D scene creation method based on 3D Gaussian sputtering (3DGS) technology. For example... Figure 1 As shown, in this embodiment, the three-dimensional scene reconstruction method includes the following steps: S100: Gaussian 3D reconstruction is performed based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. The two-dimensional image set is a collection of multiple images used for three-dimensional scene reconstruction. For example, the two-dimensional image set includes a sequence of multi-view images taken for a static scene. ; In this embodiment, Gaussian 3D reconstruction based on a set of 2D images is performed to obtain an initial Gaussian point cloud. This includes: using a Structure-from-Motion (SfM) algorithm (such as the COLMAP algorithm, or the commonly used SfM (Structure from Motion) + MVS (Multi-View Stereo) 3D reconstruction toolkit) to calculate the camera intrinsic parameters K and camera extrinsic parameters for each 2D image. And generate sparse point clouds. The initial Gaussian point cloud, used as initialization input, is represented as follows: ; The scaling ratios for each axis at each Gaussian point include the scaling ratios for the x-axis, y-axis, and z-axis; S200: Rendered image obtained based on the initial Gaussian point cloud; In this embodiment, the tile-based rasterization method of 3DGS is used for rendering to obtain the rendered image; S300: Construct the total loss, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the 2D image set. The scaling constraint loss is constructed based on the scaling factor of each Gaussian point. The normal consistency loss is constructed based on the predicted normal vector and the geometric normal vector of each Gaussian point. Among them, the reconstruction loss is used to characterize the reconstruction error between each rendered image and the corresponding input 2D image. By optimizing the reconstruction loss, the reconstructed 3D scene can be made to fit the input 2D image better. The scaling constraint loss is used to characterize the relationship between the z-axis scaling factor and the x-axis and y-axis scaling factors of each Gaussian point. The normal consistency loss is used to characterize the relationship between the predicted normal vector and the geometric normal vector of each Gaussian point. The geometric normal vector is the object's true surface normal vector obtained from the scene depth map or SfM point cloud through geometric calculation. The predicted normal vector is the normal vector defined by the third column of the quasi-plane Gaussian point rotation matrix and conforms to the orientation of the Gaussian sphere. This application uses the normal consistency loss to keep the two aligned, thereby improving the geometric accuracy of 3D reconstruction. S400: Based on the total loss, optimize the parameters of each Gaussian point in the initial Gaussian point cloud to obtain the optimized Gaussian point cloud; among which, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss. Traditional 3DGS allows independent optimization of the x, y, and z axes. This application forces the z-axis to a minimum value to compress the z-axis. In this embodiment, instead of directly optimizing the z-axis scaling ratio, the z-axis scaling ratio is optimized by defining a loss function that compares the z-axis scaling ratio with the scaling ratios of the other two axes, or by applying strong regularization. By optimizing the scaling loss, the ratio of the z-axis scaling ratio to the x and y-axis scaling ratios is reduced, making the z-axis as short as possible relative to the x and y axes.

[0023] By adopting the three-dimensional scene reconstruction method of this application, firstly, Gaussian three-dimensional reconstruction is performed based on a two-dimensional image set in step S100 to obtain the input initial Gaussian point cloud. After rendering in step S200, a total loss including reconstruction loss, scaling constraint loss and normal consistency loss is constructed in step S300. Then, the parameters of each Gaussian point are optimized based on the total loss in step S400 to obtain the optimized Gaussian point cloud.

[0024] In the 3D scene reconstruction method of this application, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratios is reduced by optimizing the scaling constraint loss during the training process. This makes the z-axis as short as possible relative to the x-axis and y-axis, abandoning the assumption in traditional 3DGS that the Gaussian volume can arbitrarily expand into a sphere. Instead, the Gaussian volume is forced to be extremely compressed along the z-axis direction of the local coordinate system, forming an ellipsoid-like structure. This transforms the Gaussian sphere into a quasi-planar Gaussian primitive. Gaussian Formulation (GRF) yields a 3D Gaussian ellipsoid that more closely approximates the real surface of an object while maintaining high fidelity and rendering quality. Simultaneously, optimizing the normal consistency loss improves the consistency between predicted and geometric normals, forcing the Gaussian orientation to align with the object's real surface normals. This avoids the problems of arbitrary orientation and loose spatial overlap of the Gaussian ellipsoid, making the Gaussian points geometrically infinitely close to the object's surface. This ensures the Gaussian point cloud tightly adheres to the solid surface, eliminating the fog / ghost artifacts of traditional 3DGS and restoring details such as sharp edges and fine grooves. This lays the core geometric foundation for subsequent extraction of smooth, high-precision watertight meshes, resulting in smooth, detailed mesh surfaces with significantly reduced geometric errors. This method balances rendering quality and geometric accuracy, integrating the geometric advantages of 2DGS with the rendering efficiency of 3DGS. The Z-axis compression strategy maintains the rasterization pipeline without sacrificing rendering speed, while also improving model interpretability.

[0025] In this embodiment, the Gaussian covariance matrix is ​​defined. Here, S represents the scaling matrix and R represents the rotation matrix. A scaling constraint loss is further introduced as a geometric flattening constraint. During the optimization of the scaling loss function, the scaling matrix is ​​forced to scale through the loss function. In Much smaller than the other two axes, i.e. .in, These represent the scaling ratios for the x-axis, y-axis, and z-axis, respectively. The target scaling factor is determined by multiplying the minimum of the x-axis and y-axis scaling factors of the Gaussian points by a preset coefficient. This represents the reciprocal of a preset coefficient, where the preset coefficient is a value less than 1. The value must be greater than 1. The preset coefficient can be set as needed; for example, setting the preset coefficient to 0.1 will... If the value is 10, then it needs to be optimized to... , here The value is for illustrative purposes only and can be adjusted to other values ​​as needed, such as 8, 15, 20, etc. The reciprocal of these values ​​is the preset coefficient.

[0026] In this embodiment, the scaling constraint loss is configured to reduce the difference between the z-axis scaling ratio and the target scaling ratio at each Gaussian point. For example, the scaling constraint loss function corresponds to the following formula:

[0027] in, Let represent the scaling constraint loss, k represent the k-th Gaussian point, and K represent the set of Gaussian points in the Gaussian point cloud.

[0028] Every plane has a normal vector. For the _i_th A Gaussian point, whose shortest axis direction (i.e., the local Z-axis direction) in the world coordinate system is determined by the rotation matrix. The third column It was decided that the third column of the rotation matrix in the Gaussian point cloud would be... Defined as the prediction normal vector of the Gaussian point.

[0029] To support geometric constraints, a depth map is rendered simultaneously with the rendered image. The 3D scene reconstruction method also includes: rendering a depth map based on the initial Gaussian point cloud, where the pixel value of each pixel in the depth map represents the depth of the corresponding Gaussian point. Specifically, the geometric normal vector of each Gaussian point... Based on depth map Calculate gradient In another embodiment, the geometric normal vectors can be obtained from the SfM point cloud via a depth estimation network, without needing to render the depth map. In this embodiment, the normal map is rendered simultaneously with the rendered map. During rendering, the predicted normal vectors of each Gaussian point are... It is rendered as color.

[0030] To ensure that the Gaussian pie adheres tightly to the surface, the normal consistency loss function can be expressed by the following formula:

[0031] in, For normal consistency loss, geometric normal vector It can be obtained from the rendered depth map Calculate gradient The prior normals are obtained, either through a depth estimation network or by means of a prior normal.

[0032] Existing 3DGS methods suffer from serious drawbacks, including a lack of semantics and editability. They represent the entire scene as a flat, unstructured collection of Gaussian point clouds. These millions of Gaussian points are independent of each other, lacking any semantic or structural connections. This representation is inherently "static" and "fixed." It is impossible to combine two or more independent 3DGS scenes in real-time and physically accurate. For example, loading a high-fidelity 3DGS model of a "chair" into a 3DGS scene of a "room" and allowing the user to freely drag the chair within the room is extremely difficult with current technology. Existing methods that incorporate semantics (such as LangSplat) are typically computationally intensive and often neglect the mutually reinforcing relationship between semantic segmentation and geometric reconstruction.

[0033] In this embodiment, to address the technical problem of lack of semantics and editability, the 3D Gaussian data structure is extended by adding a D-dimensional instance feature vector to each Gaussian point, in addition to traditional attributes. This enables them to have self-awareness and distinguish the object instances to which they belong. The parameters of each Gaussian point in the initial Gaussian point cloud also include instance feature vectors. By introducing semantic consistency loss (i.e., cross-view instance consistency loss), the instance feature vectors can be optimized, so that the instance feature vectors of Gaussian points corresponding to the same object instance are aggregated, while the instance feature vectors of Gaussian points corresponding to different object instances are far apart.

[0034] Between steps S100 and S200, the 3D scene reconstruction method further includes the following steps: An instance segmentation model is used to segment a set of two-dimensional images. Each pixel in each two-dimensional image of the set is assigned an instance ID, and each instance ID corresponds to an independent object instance.

[0035] In this embodiment, Gaussian points learn the correct object classification, and an instance segmentation model is used to segment the 2D image set. Each pixel in each 2D image of the set is assigned an instance ID. This includes processing all input 2D images using an existing pre-trained high-performance 2D instance segmentation model (such as the YOLOv11 algorithm). For each 2D image, generate an instance mask image. Each pixel A discrete instance ID label is assigned. This generates a multi-view 2D instance segmentation mask.

[0036] In this embodiment, since the instance segmentation model assigns inconsistent IDs to the same object from different viewpoints, the perspective geometry of SfM sparse point clouds is utilized to address the issue of inconsistent semantic labels across views. Specifically, an association graph based on common viewpoints is established, using 2D segmentation masks from different viewpoints as nodes. If a 3D sparse point falls within the mask regions of two different viewpoints after projection, the association weight between these two masks is increased. By performing spectral clustering or connected component analysis on the association graph, multiple 2D masks belonging to the same physical object are merged into the same global ID, which is then used as a supervision signal to train the instance feature vector of the 3D Gaussian point.

[0037] Based on this, in this embodiment, an instance segmentation model is used to segment the two-dimensional image set into instances, and an instance ID is assigned to each pixel in each two-dimensional image of the two-dimensional image set as an initial instance ID. Assigning an instance ID to each pixel further includes the following steps: (1) Input data preparation, including: obtaining the sparse 3D point set reconstructed by COLMAP. Get each image Corresponding 2D segmentation mask set ,in It is the first The ID in the image is The masked area, where each Mask ID corresponds to the initial instance ID; (2) Constructing the correlation statistics matrix (Voting / Statistics): Traversing each sparse 3D point Query In the COLMAP data, the observation frame index is recorded. For each Gaussian point, the two-dimensional image projected onto that Gaussian point and its projected pixel coordinates in the corresponding two-dimensional image are recorded, and the initial instance ID corresponding to the projected pixel coordinates is determined; assuming... Simultaneously projected onto the image and images Above, General Projection Back and The pixel coordinates are checked to see which of these pixels fall within the range of the pixel coordinates. Which Mask ID (denoted as) )and Which Mask ID (denoted as) )middle; For each Gaussian point, if multiple distinct initial instance IDs exist, a score is added to the relationship between the multiple initial IDs corresponding to the same Gaussian point, and the pairwise relationship scores among all initial instance IDs are calculated; if the same 3D point falls on multiple instances... and In this case, it is assumed that the two Mask IDs may belong to the same object, and a global "association counter" is used to assign them to the same object. and This relationship gets 1 point.

[0038] (3) Constructing a Global Associative Graph: Using the initial instance IDs in all two-dimensional images as graph nodes, construct a global associative graph by establishing edges based on the relationship scores between each pair of initial instance IDs (statistical scores in the association counter in the previous step). Connect nodes whose relationship scores are greater than a preset relationship threshold. The preset relationship threshold can be set as needed. For example, if it is set to 10, two nodes need to share at least 10 3D points before connecting the two nodes. In the global associative graph, the relationship score is used as the weight of the corresponding edge. The weight of the edge can be the number of shared 3D points or the IoU (Intersection over Union) score. (4) Graph clustering and ID unification: Perform connected component analysis on the global association graph. Each connected component represents a real 3D object. Map all initial instance IDs within the same connected component to the same instance ID to obtain the updated final instance ID. For each 3D object, each corresponding pixel has a unified global instance ID.

[0039] In this embodiment, mathematical modeling of instance-aware quasi-plane Gaussian primitives is added, and the data structure of 3D Gaussian primitives is redefined. Each Gaussian point... Includes the following parameter set :

[0040] in: : The center position (mean) in the world coordinate system.

[0041] Rotation quaternions are used to construct rotation matrices. .

[0042] Scaling factor .

[0043] Opacity.

[0044] : Spherical harmonic coefficient (SH), used to represent color with respect to viewing angle.

[0045] The newly added D-dimensional instance feature vector, the value of D can be selected as needed, for example, set to 16, but this application is not limited to this.

[0046] In this embodiment, the total loss also includes instance semantic consistency loss. By optimizing the instance semantic consistency loss, the instance feature vector is optimized so that the instance feature vectors of Gaussian points corresponding to the same instance ID are aggregated, while the instance feature vectors of Gaussian points corresponding to different instance IDs are moved away.

[0047] In this embodiment, in the rendering pipeline, in addition to rendering the rendered image, a feature map is also rendered synchronously using the volume rendering formula to calculate the instance semantic consistency loss. Based on this, the instance semantic consistency loss is constructed using the following steps: A feature map is obtained by rendering based on the instance feature vectors in the initial Gaussian point cloud. The pixel value of each pixel in the feature map represents the rendering feature of that pixel. An instance semantic consistency loss is constructed based on the rendering feature relationship of pixels with the same instance ID and the rendering feature relationship of pixels with different instance IDs. The instance feature vector is optimized by optimizing the instance semantic consistency loss, so that the rendering features of pixels with the same instance ID are aggregated and the rendering features of pixels with different instance IDs are far apart.

[0048] In summary, in step S200 of this embodiment, the rendering pipeline simultaneously obtains a rendering map, a feature map, a depth map, and a normal map during rendering. The rendering map is the RGB image of each viewpoint obtained during rendering, used to construct the reconstruction loss. When rendering the RGB image, for a pixel p on the screen, its color... From the cone It is formed by mixing Gaussian points sorted by depth:

[0049] in, The final rendered color of a single pixel p on the screen is represented by a three-dimensional RGB vector. N represents the set of all Gaussian points within the view frustum that cover the current pixel p. i is the index of the Gaussian point in the set N. j represents the index of all Gaussian points before (further away from) index i, used to calculate the cumulative occlusion of the forward transparency.

[0050] Feature maps are used to construct semantic consistency loss. When rendering feature maps, the same mixture weights are used to render high-dimensional instance feature maps. :

[0051] in, It is the first The instance feature vectors of Gaussian points, due to It has high dimensionality and is implemented in the CUDA (Compute Unified Device Architecture) kernel through MRT (Multiple Render Targets) technology or multi-channel rendering, as well as high-dimensional feature maps.

[0052] In step S300 of this embodiment, the constructed total loss includes photometric loss, quasi-plane geometric loss (including scaling constraint loss and normal consistency loss), semantic consistency loss, and a regularization term. In step S400, end-to-end training is performed by weighting the individual losses to obtain the total loss. The total loss includes:

[0053] Among them, (1) photometric reconstruction loss ( It is constructed using the following formula:

[0054] Ensure that the rendered image matches the real image.

[0055] (2) Quasi-plane geometric loss Scaling constraint loss in ) The Z-axis length is directly penalized, and its calculation formula has been illustrated in the previous text.

[0056] Normal consistency loss in quasi-planar geometric loss The Gaussian point is used to constrain and ensure that its orientation is consistent with the scene's geometric surface. Its calculation formula has been illustrated in the previous text.

[0057] (3) Semantic consistency loss based on semantic instances )

[0058] Assuming at pixel At this point, the globally unique instance ID corresponding to the object given by the instance segmentation model is It is necessary to make It is close to the feature center corresponding to the object.

[0059] The semantic consistency loss is constructed using InfoNCE Contrastive Loss (Information Noise Contrastive Estimation Contrastive Loss):

[0060] Among them, positive sample pairs ( ): pixel The rendering features, and the features of other pixels within the same mask area.

[0061] Negative sample pairs ( ): pixel The rendering characteristics of the pixels in the masked region (different objects).

[0062] Through this semantic consistency loss, the feature vectors of all 3D Gaussian points belonging to the same object (such as "table") are obtained. They will cluster together in the feature space.

[0063] (4) Regularization term ( )

[0064] Includes opacity regularization (encourages binarization, i.e., the opacity parameter of the Gaussian point cloud). Approaching 0 or 1) and total variation loss, further reducing noise.

[0065] In this embodiment, after training, a set of "flat" Gaussian point clouds with rich semantic information is obtained, and mesh extraction can be performed based on the optimized Gaussian point cloud for post-processing. Since the scaling factor of the forced z-axis is very small in this embodiment, and the predicted normal vector is aligned with the true geometric normal vector, the obtained Gaussian points are actually approximately oriented point elements (surfels).

[0066] In this embodiment, after step S400, the method further includes using the optimized positions and normal vectors of the Gaussian points on the quasi-plane to directly generate a watertight 3D mesh through a Poisson Surface Reconstruction or TSDF (Truncated Signed Distance Function) fusion algorithm. Specifically, the following steps are used for mesh extraction: Filter out the confidence level from the Gaussian point cloud obtained during training. The low confidence point; Using the center position of Gauss and prediction normal vector As input, run the Poisson surface reconstruction algorithm or the TSDF fusion algorithm to generate a mesh, with depth values... The value ranges from 8 to 10; The generated mesh can be texture-mapped or directly bound to Gaussian points for rendering.

[0067] Traditional 3DGS Gaussian point cloud mesh extraction typically uses Marching Cubes to process the density field, resulting in extremely poor performance. This embodiment directly utilizes the explicit geometric information of the Gaussian point cloud, generating a mesh surface with smooth and sharp edges. Furthermore, due to the more compact geometry and the removal of redundant interiors and floating Gaussian points, the final model's storage volume is effectively controlled, and storage efficiency is optimized.

[0068] In this embodiment, precise segmentation and decoupling of discrete objects are achieved in a continuous Gaussian field using instance feature vectors. This allows users to rearrange the neural radiation field scene as if operating CAD software, enabling fine-grained editing. This embodiment further provides a feature-based interactive system. When a user clicks a pixel on the screen, the system obtains the corresponding instance features through ray projection, then retrieves all Gaussian points with similar features across the entire scene, constructs an "ObjectGroup," and applies an affine transformation matrix. Perform spatial operations on groups of objects to move and combine the entire object.

[0069] In this embodiment, after step S400, the following step is further included: The current view from the user's perspective is rendered based on the optimized Gaussian point cloud; Upon receiving the user's selection of pixels in the current view, determine the selected Gaussian point corresponding to the selected pixel; The similarity between the instance feature vector of the selected Gaussian point and the instance feature vector of other Gaussian points is calculated. Gaussian points with similarity greater than a preset similarity threshold are selected from the other Gaussian points to form a Gaussian point set. Each Gaussian point in the Gaussian point set corresponds to the same physical entity as the selected Gaussian point. Based on the pose transformation information input by the user, the poses of the selected Gaussian point and each Gaussian point in the set of Gaussian points are updated, and the updated 3D scene is re-rendered. The pose transformation information input by the user may include one or more of the following: position update information, rotation matrix update information, and normal vector update information.

[0070] In this embodiment, the transformed scene is rendered in real time after updating the pose of each Gaussian point. Since the Gaussian points are independent, the background does not need to be retrained, enabling real-time object-level editing. Therefore, this embodiment, based on instance semantics and quasi-planar geometry, allows users to decouple, translate, rotate, and combine independent objects in the scene in real time, while ensuring the rendering quality and geometric consistency after editing.

[0071] In summary, this embodiment provides a complete closed-loop system whose input is a set of two-dimensional images, such as a sequence of photographs of a static scene. Core processing includes constructing a quasi-planar Gaussian model, embedding semantic vectors, and performing joint optimization through a multi-task loss function. The output includes an optimized Gaussian point cloud, enabling real-time rendering of a high-fidelity 3D scene. Furthermore, a high-quality scene geometry mesh is extracted from the optimized Gaussian point cloud. The scene can be easily edited using independently operable semantic objects (Gaussian groups). This embodiment innovatively resolves the contradiction between "seeing" and "using" in neural rendering, allowing users to both "see" photorealistic image quality and "use" it for geometric editing and physical interaction, thus possessing significant industrial application value.

[0072] The following section uses two specific application examples to illustrate the optional implementation methods of the 3D scene reconstruction method in this embodiment in specific application scenarios.

[0073] Application Example 1: Large Scene Reconstruction Example.

[0074] (1) Data collection: A collection of two-dimensional images was acquired using drones and mobile phones (or other handheld cameras). Aerial images were captured using oblique photography drones, taking both top-down and 45-degree tilted views, primarily covering building rooftops and overall terrain layout. Ground-based images were captured using handheld cameras, primarily covering building facades, street facilities, and close-up textures.

[0075] (2) Data preprocessing: Initial sparse point cloud reconstruction was performed using COLMAP to obtain the camera pose. and sparse point clouds This provides an initial 3D Gaussian point cloud for 3DGS training.

[0076] A pre-trained 2D instance segmentation model (such as the YOLOv11 instance segmentation model or an instance segmentation model trained on the Cityscapes dataset) is used to segment each image in the 2D image set. Then, the perspective geometry properties of the SfM sparse point cloud are used to construct a global association graph to solve the problem of inconsistent semantic labels for the same object in different images across views. This provides instance semantic labels for 3DGS training.

[0077] (3) Training the model: During initial training, each Gaussian point is initialized, and the parameter attributes of the Gaussian point cloud are determined by... The Gaussian parameters are continuously updated during training. During inference, the Gaussian point cloud is projected, splashed, and rasterized for rendering according to step S200. The inverse optimization loss is the loss shown in step S300. For large-scale scene training, a block-based parallel training approach can be used to avoid the impact on training efficiency and results caused by a single GPU being unable to handle the large number of Gaussian points.

[0078] (4) Exporting 3D Gaussian models and custom combinations: The trained 3D Gaussian model includes an optimized 3D Gaussian point cloud, which can perform real-time rendering at more than 60fps, supports free movement of instance objects in the scene, can be combined in any 3DGS scene, and finally can be exported as a high-precision Mesh model through TSDF.

[0079] Application Example 2: Face Reconstruction in a Meeting.

[0080] (1) Data collection: During the meeting, facial capture or mobile phone surround shooting is used to obtain a set of two-dimensional images. The more images there are, the better the final three-dimensional reconstruction model will be generated.

[0081] (2) Data preprocessing: Initial sparse point cloud reconstruction was performed using COLMAP to obtain the camera pose. and sparse point clouds This provides an initial 3D Gaussian point cloud for 3DGS training.

[0082] A pre-trained 2D instance segmentation model (such as the YOLOv11 face instance segmentation model or a face-optimized segmentation network (such as BiSeNet)) is used to segment each image in the 2D image set, generating a semantic mask $M_{face}$ containing the following category labels (corresponding instance IDs): {0: background, 1: skin, 2: left eyebrow, 3: right eyebrow, 4: left eye, 5: right eye, 6: nose, 7: upper lip, 8: lower lip, 9: inside of the mouth, 10: hair, 11: glasses}. Then, the perspective geometry of the SfM sparse point cloud is used to construct a global association graph to solve the problem of inconsistent semantic labels for the same object across different images, providing instance semantic labels for 3DGS training.

[0083] (3) Training the model: During initial training, to accelerate convergence, this embodiment does not directly use randomly initialized point clouds. Instead, it uses the FLAME model (a parametric face model) to fit the image and generate a coarse face mesh as the initial Gaussian center position. This ensures that the Gaussian points are initially distributed near the facial manifold surface. The parametric properties of the Gaussian point cloud are determined by... These Gaussian parameters are defined and continuously updated during training. During inference, the Gaussian point cloud is projected, splashed, and rasterized for rendering according to the method in step S200. The inverse optimization loss is the loss shown in step S300.

[0084] (4) Exporting models and custom combinations: The trained 3D Gaussian model includes an optimized 3D Gaussian point cloud, which can perform real-time rendering at more than 60fps, supports free movement of instance objects in the scene, can be combined in any 3DGS scene, and finally can be exported as a high-precision Mesh model through TSDF.

[0085] like Figure 2 As shown, this application embodiment also provides a three-dimensional scene reconstruction system for implementing the three-dimensional scene reconstruction method of the first aspect. The system includes: The reconstruction module M100 is used to perform Gaussian 3D reconstruction based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. The M200 rendering module is used to obtain a rendering image based on the initial Gaussian point cloud. The M300 module is used to construct the total loss, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the 2D image set. The scaling constraint loss is constructed based on the scaling factor of each Gaussian point. The normal consistency loss is constructed based on the predicted normal vector and the geometric normal vector of each Gaussian point. The training module M400 is used to optimize the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss, so as to obtain the optimized Gaussian point cloud. Specifically, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss.

[0086] In the three-dimensional scene reconstruction system of this application, the functions of each module can be implemented by the specific implementation method of the three-dimensional scene reconstruction method described above, and the three-dimensional scene reconstruction system can achieve the technical effects of the three-dimensional scene reconstruction method described above, which will not be elaborated here.

[0087] This application embodiment also provides a three-dimensional scene reconstruction device, including a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the three-dimensional scene reconstruction method by executing the executable instructions.

[0088] Those skilled in the art will understand that various aspects of this application can be implemented as systems, methods, or computer program products. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "platform."

[0089] The following reference Figure 3 To describe an electronic device 600 according to this embodiment of the present application. Figure 3 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0090] like Figure 3 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0091] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the above-described section on the three-dimensional scene reconstruction method according to various exemplary embodiments of this application. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0092] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0093] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0094] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0095] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] In the aforementioned 3D scene reconstruction device, when the program in the memory is executed by the processor, it implements the steps of the aforementioned 3D scene reconstruction method. Therefore, the device can also achieve the technical effects of the aforementioned 3D scene reconstruction method.

[0097] An exemplary embodiment of this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described three-dimensional scene reconstruction method.

[0098] In one embodiment, the computer program product can be a tangible product containing a computer program, such as a computer-readable storage medium storing the computer program. The readable storage medium can be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory, hard disk drive (HDD), solid-state drive (SSD), etc. Exemplarily, the computer program product can be implemented as a non-volatile storage medium storing a computer program, such as read-only memory, NAND flash memory, etc.

[0099] In one implementation, the computer program product can be an intangible product containing a computer program. For example, the computer program product can be implemented as a virtual digital product, such as an executable file, installation package, or other digital file storing the computer program.

[0100] Computer program code can be written in one or more programming languages. Examples of programming languages ​​include C, Java, C++, and Python. Program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, such as a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via an internet connection provided by a mobile network operator).

[0101] Computer programs can be carried or transmitted via signals such as electrical, magnetic, optical, electromagnetic, and infrared rays. Electronic devices can convert the signals carrying computer programs into digital signals, thereby running the computer programs. When a computer program runs on an electronic device, its code is used to cause the electronic device to execute (more specifically, the processor of the electronic device to execute) the method steps of various exemplary embodiments of this application, such as the steps of the three-dimensional scene reconstruction method described above.

[0102] When the computer program is executed by the processor, it implements the steps of the above-described three-dimensional scene reconstruction method. Therefore, the computer program product can also achieve the technical effects of the above-described three-dimensional scene reconstruction method.

[0103] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of this application and should not be construed as limiting the specific implementation of this application to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of this application, and all such modifications or substitutions should be considered within the scope of protection of this application.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, Includes the following steps: Gaussian 3D reconstruction is performed based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. A rendered image is obtained based on the initial Gaussian point cloud rendering; Construct a total loss, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the set of two-dimensional images. The scaling constraint loss is constructed based on the scaling factor of each Gaussian point. The normal consistency loss is constructed based on the predicted normal vector and geometric normal vector of each Gaussian point. The parameters of each Gaussian point in the initial Gaussian point cloud are optimized based on the total loss to obtain an optimized Gaussian point cloud; wherein, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The scaling constraint loss is configured to reduce the difference between the z-axis scaling ratio and the target scaling ratio of each Gaussian point. The target scaling ratio is obtained by multiplying the minimum of the x-axis scaling ratio and the y-axis scaling ratio of the Gaussian point with a preset coefficient, where the preset coefficient is less than 1.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that, Also includes: Based on the initial Gaussian point cloud rendering depth map, the pixel value of each pixel in the depth map represents the depth of the Gaussian point corresponding to that pixel; The geometric normal vector of each Gaussian point is obtained by calculating the gradient based on the depth map.

4. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The parameters of each Gaussian point in the initial Gaussian point cloud also include the instance feature vector; The method further includes the following steps: An instance segmentation model is used to segment the two-dimensional image set into instances, and an instance ID is assigned to each pixel in each two-dimensional image of the two-dimensional image set. The total loss also includes instance semantic consistency loss, which optimizes the instance feature vector by optimizing the instance semantic consistency loss, so that the instance feature vectors of Gaussian points with the same instance ID are aggregated, and the instance feature vectors of Gaussian points with different instance IDs are far apart.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that, The instance semantic consistency loss is constructed using the following steps: A feature map is obtained by rendering based on the instance feature vectors in the initial Gaussian point cloud, and the pixel value of each pixel in the feature map represents the rendering feature of that pixel; An instance semantic consistency loss is constructed based on the rendering feature relationship of pixels with the same instance ID and the rendering feature relationship of pixels with different instance IDs. The instance feature vector is optimized by optimizing the instance semantic consistency loss, so that the rendering features of pixels with the same instance ID are aggregated and the rendering features of pixels with different instance IDs are far apart.

6. The three-dimensional scene reconstruction method according to claim 4, characterized in that, The two-dimensional image set is segmented using an instance segmentation model, and an instance ID is assigned to each pixel in each two-dimensional image of the two-dimensional image set. The steps include: The two-dimensional image set is segmented using an instance segmentation model, and an initial instance ID is assigned to each pixel in each two-dimensional image of the two-dimensional image set. For each Gaussian point, record the two-dimensional image to which the Gaussian point is projected and the coordinates of the projected pixels in the corresponding two-dimensional image, and determine the initial instance ID corresponding to the corresponding projected pixel coordinates; For each Gaussian point, if there are multiple different initial instance IDs, the relationship between the multiple initial IDs corresponding to the same Gaussian point is scored, and the relationship scores between all pairs of initial instance IDs are obtained. All initial instance IDs are used as graph nodes. An edge is built based on the relationship score between each pair of initial instance IDs to construct a global association graph. Nodes with relationship scores greater than a preset relationship threshold are connected. Perform connectivity component analysis on the global association graph, and map all initial instance IDs within the same connectivity component to the same instance ID to obtain the updated final instance ID.

7. The three-dimensional scene reconstruction method according to claim 4, characterized in that, After optimizing the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss, the following steps are also included: The current view from the user's perspective is rendered based on the optimized Gaussian point cloud. Upon receiving the user's selection of pixels in the current view, determine the selected Gaussian point corresponding to the selected pixel; The similarity between the instance feature vector of the selected Gaussian point and the instance feature vector of other Gaussian points is calculated, and Gaussian points with similarity greater than a preset similarity threshold are selected from other Gaussian points to form a set of Gaussian points. Based on the pose transformation information input by the user, the poses of the selected Gaussian point and each Gaussian point in the set of Gaussian points are updated, and the updated 3D scene is re-rendered.

8. A three-dimensional scene reconstruction system, characterized in that, The system for implementing the three-dimensional scene reconstruction method according to any one of claims 1 to 7 includes: The reconstruction module is used to perform Gaussian 3D reconstruction based on a set of 2D images to obtain an initial Gaussian point cloud. The parameters of each Gaussian point in the initial Gaussian point cloud include the scaling ratio of each axis and the predicted normal vector. The rendering module is used to obtain a rendered image based on the initial Gaussian point cloud; A construction module is used to construct the total loss, which includes reconstruction loss, scaling constraint loss and normal consistency loss. The reconstruction loss is constructed based on the rendered image and the two-dimensional image set. The scaling constraint loss is constructed based on the scaling factor of each Gaussian point. The normal consistency loss is constructed based on the predicted normal vector and geometric normal vector of each Gaussian point. The training module is used to optimize the parameters of each Gaussian point in the initial Gaussian point cloud based on the total loss to obtain an optimized Gaussian point cloud; wherein, the ratio of the z-axis scaling ratio to the x-axis and y-axis scaling ratio is reduced by optimizing the scaling constraint loss, and the consistency between the predicted normal vector and the geometric normal vector is improved by optimizing the normal consistency loss.

9. A three-dimensional scene reconstruction device, characterized in that, include: processor; A memory in which executable instructions of the processor are stored; The processor is configured to perform the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional scene reconstruction method according to any one of claims 1 to 7.

Images