Three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium

Abstract

The embodiments of the present application propose a three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium, and the method includes: first, obtaining multiple three-dimensional Gaussian distributions of the target scene; then, obtaining the two-dimensional projection weight of each three-dimensional Gaussian distribution, and selecting a key Gaussian from the multiple three-dimensional Gaussian distributions based on the two-dimensional projection weight; next, performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, and performing deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions; finally, selecting multiple retained Gaussian distributions from the multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, and performing scene reconstruction based on the multiple retained Gaussian distributions to obtain a reconstructed scene matching the target scene, thereby significantly improving the overall efficiency of three-dimensional scene reconstruction and the compactness and quality of the final reconstructed scene.

Description

Technical Field

[0001] The present application relates to the field of image data processing technology, and in particular to a three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium. Background Art

[0002] With the development of large models and big data, pre-trained image segmentation technology for general objects has gradually become a practical engineering technology, and provides extremely useful prior information for downstream visual tasks such as image / video understanding and 3D reconstruction.

[0003] In related technologies, 3D Gaussian splatting is typically used when reconstructing a 3D scene. This technique uses a sparse point cloud generated from a Structure-from-Motion (SfM) algorithm to initialize a Gaussian distribution. This Gaussian distribution is then optimized. During the optimization process, a pre-set fixed density control strategy is used to dynamically adjust the number of Gaussian distributions, allowing the optimized Gaussian distribution to be used for 3D scene reconstruction. Because the number of 3D Gaussian distributions used in 3D scene reconstruction is often numerous and complex, a pre-set fixed density strategy cannot provide appropriate density control in such complex situations. Consequently, the existing 3D Gaussian splatting technique still suffers from low reconstruction efficiency. Summary of the Invention

[0004] The embodiments of the present application provide a three-dimensional Gaussian scene reconstruction method, device, electronic device, and storage medium, which can improve the reconstruction efficiency of the three-dimensional Gaussian scene.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a three-dimensional Gaussian scene reconstruction method, the method comprising:

[0006] Obtain multiple three-dimensional Gaussian distributions of the target scene;

[0007] Obtaining a two-dimensional projection weight of each of the three-dimensional Gaussian distributions, and selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weight;

[0008] Performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, and performing deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions;

[0009] Based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, multiple retained Gaussian distributions are selected from the multiple dense three-dimensional Gaussian distributions, and scene reconstruction is performed based on the multiple retained Gaussian distributions to obtain a reconstructed scene matching the target scene.

[0010] In some embodiments, selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weight comprises:

[0011] For each image plane point of each viewing angle, from multiple three-dimensional Gaussian distributions corresponding to the image plane point, select the three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the important Gaussian corresponding to the image plane point;

[0012] Obtaining an effective projected area of ​​each of the three-dimensional Gaussian distributions, and selecting, from the multiple three-dimensional Gaussian distributions, the three-dimensional Gaussian distribution whose effective projected area exceeds a preset area threshold as a fuzzy Gaussian;

[0013] The key Gaussian is obtained based on all the important Gaussians and all the fuzzy Gaussians.

[0014] In some embodiments, obtaining the effective projected area of ​​each of the three-dimensional Gaussian distributions includes:

[0015] Obtaining a two-dimensional projection result corresponding to each viewing angle of each three-dimensional Gaussian distribution;

[0016] When the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian, obtaining a valid projection value corresponding to each of the viewing angles;

[0017] The effective projection values ​​of all the viewing angles are accumulated to obtain the effective projection area of ​​each of the three-dimensional Gaussian distributions.

[0018] In some embodiments, the three-dimensional Gaussian distribution includes at least a center position, opacity, and a covariance matrix, and performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions includes:

[0019] Based on the center position of the key Gaussian, the center position of the split Gaussian distribution is obtained;

[0020] Based on the difference between the opacity of the key Gaussian and the opacity of the key Gaussian, a root opening process is performed to obtain a difference root, and based on the difference between the opacity of the key Gaussian and the opacity of the split Gaussian distribution is obtained;

[0021] Based on the difference between twice the opacity of the split Gaussian distribution and the square of the opacity of the split Gaussian distribution, a quadratic inverse process is performed to obtain a split square inverse;

[0022] A covariance matrix of the split Gaussian distribution is obtained based on the split square inverse, the covariance matrix of the key Gaussian, and the square of the opacity of the key Gaussian.

[0023] In some embodiments, performing deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain a dense three-dimensional Gaussian distribution includes:

[0024] Obtaining the Gaussian distribution depth corresponding to the important Gaussian corresponding to each of the image plane points;

[0025] Obtaining a depth map based on all of the Gaussian distribution depths;

[0026] Based on the depth map, the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions are reprojected into a coordinate system to obtain the multiple dense three-dimensional Gaussian distributions.

[0027] In some embodiments, selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions comprises:

[0028] Among the multiple dense three-dimensional Gaussian distributions corresponding to each image plane point, selecting the dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the first retained Gaussian distribution corresponding to the image plane point;

[0029] Accumulating the two-dimensional projection weights corresponding to each of the dense three-dimensional Gaussian distributions at each viewing angle to obtain the Gaussian importance of each of the dense three-dimensional Gaussian distributions;

[0030] Accumulating all the Gaussian importances to obtain a sum of importances, and obtaining a sampling probability corresponding to each of the dense three-dimensional Gaussian distributions based on a ratio of each Gaussian importance to the sum of importances;

[0031] Sampling each of the dense three-dimensional Gaussian distributions based on the sampling probability to obtain a second retained Gaussian distribution;

[0032] The retained Gaussian distribution is obtained based on the first retained Gaussian distribution and the second retained Gaussian distribution.

[0033] In some embodiments, before selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weights, the method further includes:

[0034] During each iteration, image rendering is performed based on the multiple three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a rendered image, and the opacity, center position, and covariance matrix of the multiple three-dimensional Gaussian distributions are updated based on the difference between the rendered image and the real image of the target scene, until the current iteration number reaches a first preset iteration number;

[0035] Before selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions, the method further includes:

[0036] During each iteration, image rendering is performed based on the multiple dense three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a dense rendered image, and based on the difference between the dense rendered image and the real image, the opacity, center position and covariance matrix of the multiple dense three-dimensional Gaussian distributions are updated until the current iteration number reaches a second preset iteration number.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a three-dimensional Gaussian scene reconstruction device, the device comprising:

[0038] A Gaussian acquisition module, configured to acquire a three-dimensional Gaussian distribution of a target scene, wherein the three-dimensional Gaussian distribution includes multiple three-dimensional Gaussian distributions;

[0039] a key Gaussian screening module, configured to obtain a two-dimensional projection weight of each of the three-dimensional Gaussian distributions, and select a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weight;

[0040] a Gaussian densification module, configured to perform Gaussian splitting on the key Gaussian to obtain a plurality of split Gaussian distributions, and perform deep reinitialization based on the plurality of split Gaussian distributions and the plurality of three-dimensional Gaussian distributions to obtain a dense three-dimensional Gaussian distribution, wherein the dense three-dimensional Gaussian distribution includes a plurality of dense three-dimensional Gaussian distributions;

[0041] A Gaussian sparsification module is used to select multiple retained Gaussian distributions from the multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, and to reconstruct the scene based on the multiple retained Gaussian distributions to obtain a reconstructed scene that matches the target scene.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the three-dimensional Gaussian scene reconstruction method described in the first aspect.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the three-dimensional Gaussian scene reconstruction method described in the first aspect above.

[0044] The embodiments of the present application propose a three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium, and the method includes: first, obtaining multiple three-dimensional Gaussian distributions of the target scene; then, obtaining the two-dimensional projection weight of each three-dimensional Gaussian distribution, and selecting a key Gaussian from the multiple three-dimensional Gaussian distributions based on the two-dimensional projection weight; next, performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, and performing deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions; finally, selecting multiple retained Gaussian distributions from the multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, and performing scene reconstruction based on the multiple retained Gaussian distributions to obtain a reconstructed scene matching the target scene. The embodiment of the present application first intelligently selects key Gaussians that contribute significantly to the scene expression or may cause blur based on the two-dimensional projection weights of the three-dimensional Gaussian distribution, then performs targeted Gaussian splitting on these key Gaussians and reinitializes them in combination with depth information to achieve effective densification, and finally selects important retained Gaussians based on the two-dimensional projection weights of the dense three-dimensional Gaussian distribution for simplification and final scene reconstruction. By utilizing a dynamic density adjustment strategy driven by data importance, the limitation of using a pre-set fixed density control strategy that is difficult to perform appropriate density control when processing complex and numerous three-dimensional Gaussian distributions is effectively overcome, so that the Gaussian density can be increased more accurately in areas where details are required, while removing unnecessary Gaussians in redundant areas, avoiding computational waste and representation redundancy caused by blind adjustment, thereby significantly improving the overall efficiency of three-dimensional scene reconstruction and the compactness and quality of the final reconstructed scene.

[0045] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flowchart of a three-dimensional Gaussian scene reconstruction method provided in one embodiment of the present application.

[0047] Figure 2 yes Figure 1 Flowchart of step 102 in FIG.

[0048] Figure 3 yes Figure 2 Flowchart of step 202 in FIG.

[0049] Figure 4 yes Figure 1 Flowchart of step 103 in FIG.

[0050] Figure 5 yes Figure 1 Another flow chart of step 103 in FIG.

[0051] Figure 6 yes Figure 1 Flowchart of step 104 in FIG.

[0052] Figure 7 This is a flowchart of Gaussian densification and then thinning provided in another embodiment of the present application.

[0053] Figure 8 This is a flowchart of Gaussian optimization update provided by another embodiment of the present application.

[0054] Figure 9 This is a schematic diagram of collecting image data of a three-dimensional scene provided by yet another embodiment of the present application.

[0055] Figure 10 This is a schematic diagram of reconstruction of a three-dimensional scene provided by yet another embodiment of the present application.

[0056] Figure 11 3D Gaussian scene reconstruction device provided in one embodiment of the present application is shown in FIG.

[0057] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] With the development of large models and big data, pre-trained image segmentation technology for general objects has gradually become a practical engineering technology, and provides extremely useful prior information for downstream visual tasks such as image / video understanding and 3D reconstruction.

[0062] In related technologies, 3D Gaussian splatting is typically used when reconstructing a 3D scene. This technique uses a sparse point cloud generated from a Structure-from-Motion (SfM) algorithm to initialize a Gaussian distribution. This Gaussian distribution is then optimized. During the optimization process, a pre-set fixed density control strategy is used to dynamically adjust the number of Gaussian distributions, allowing the optimized Gaussian distribution to be used for 3D scene reconstruction. Because the number of 3D Gaussian distributions used in 3D scene reconstruction is often numerous and complex, a pre-set fixed density strategy cannot provide appropriate density control in such complex situations. Consequently, the existing 3D Gaussian splatting technique still suffers from low reconstruction efficiency.

[0063] In order to improve the reconstruction efficiency of three-dimensional Gaussian scenes, the embodiment of the present application uses the pose point cloud estimation information obtained from two-dimensional scene images from multiple perspectives to perform three-dimensional reconstruction to obtain a three-dimensional Gaussian point cloud corresponding to the three-dimensional structural information used to restore the target scene. After the target segmented objects in the target scene are segmented in response to the segmentation prompt instruction, the Gaussian points corresponding to the target segmented objects in each perspective in the three-dimensional Gaussian point cloud are filtered out, so that the target segmented objects are accurately segmented from the target scene from a three-dimensional perspective. Then, the pose point cloud estimation information is used to perform two-dimensional projection on the filtered three-dimensional Gaussian point set after three-dimensional segmentation, so as to accurately obtain the two-dimensional segmentation results corresponding to multiple perspectives of the target segmented objects, thereby overcoming the defects of relying on manual adjustment and frequent over / under segmentation in the existing technology, thereby greatly improving the accuracy of two-dimensional image segmentation of objects in the target scene.

[0064] It can be understood that the 3D Gaussian Splatting (3DGS) technology models the scene as a set of 3D Gaussian distributions. Each three-dimensional Gaussian distribution By opacity , central location , the covariance matrix in world space (used to represent the scale and rotation of the three-dimensional Gaussian), each three-dimensional Gaussian distribution is represented as ,in is the image plane point. For a given viewing angle, the two-dimensional image can be recovered from all Gaussian representations ,in , is a three-dimensional Gaussian Projection to a 2D image. Rendered image by minimizing and the true value image The error (such as root mean square error and structural similarity) is used to optimize the scene, that is, all Gaussians.

[0065] This solution proposes a multi-view reconstruction system based on Gaussian representation, which mainly includes two parts: densification and simplification. Figure 1 As shown in the figure, first, the system uses a sparse point cloud as the initial input and converts the sparse point cloud into a dense Gaussian model through densification to achieve efficient scene reconstruction. After obtaining the dense Gaussian model, the system further performs a simplification operation, simplifying the dense Gaussian model into a sparse Gaussian model to achieve a minimum scene representation. Specifically, the system is in the optimization process (minimizing the rendered image and the true value image In the error), by setting a specific number of iterations, repeatedly densifying (such as every 100 or 200) until a dense Gaussian is obtained, and then performing simplification (such as 3000 or 8000), and continuing to optimize the model until convergence, the final sparse Gaussian is obtained, as described below.

[0066] The following will further describe the three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium provided by the present application. First, the three-dimensional Gaussian scene reconstruction method in the embodiment of the present application is specifically described. Figure 1 , which is an optional flow chart of the three-dimensional Gaussian scene reconstruction method provided in an embodiment of the present application, Figure 1 The method may include but is not limited to steps 101 to 104. It is also understood that this embodiment is for Figure 1 The order of steps 101 to 104 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs. The three-dimensional Gaussian scene reconstruction method provided in this application can be applied to any server or intelligent terminal with processing and computing capabilities.

[0067] Step 101: Acquire multiple three-dimensional Gaussian distributions of the target scene.

[0068] Step 101 is described in detail below.

[0069] In some embodiments, after responding to a scene reconstruction request of a target scene, firstly, a plurality of three-dimensional Gaussian distributions of the target scene are obtained. These initial multiple 3D Gaussian distributions can come from, for example, sparse point clouds generated by the Structure-from-Motion (SfM) technique from a set of image sequences. Together, they constitute a preliminary 3D representation of the target scene and serve as the basic input for the subsequent reconstruction process.

[0070] It can be understood that a three-dimensional Gaussian distribution refers to a mathematical model used to characterize the geometry and appearance characteristics of a scene in three-dimensional space. Each three-dimensional Gaussian distribution is usually defined by parameters such as its center position, covariance matrix (used to characterize its shape, size and direction), and opacity.

[0071] Step 102: Obtain a two-dimensional projection weight of each three-dimensional Gaussian distribution, and select a key Gaussian from multiple three-dimensional Gaussian distributions based on the two-dimensional projection weight.

[0072] Step 102 is described in detail below.

[0073] In some embodiments, after obtaining multiple three-dimensional Gaussian distributions of the target scene, a specific algorithm is further used to identify key Gaussians in the scene from the multiple three-dimensional Gaussian distributions to determine the focus of the area to be reconstructed. In this embodiment, for the image plane point corresponding to a given viewing angle, there is a specific algorithm for each three-dimensional Gaussian distribution. The projection mixing result is shown in the following formula (1).

[0074] (1)

[0075] Then further determine the two-dimensional projection weight of each three-dimensional Gaussian distribution under each virtual camera perspective. The two-dimensional projection weight represents the contribution or influence of each three-dimensional Gaussian distribution on the final color value of each pixel in the image when it is projected onto the two-dimensional image plane through the differentiable renderer.

[0076] The system then screens and identifies key Gaussians from multiple 3D Gaussian distributions based on these calculated 2D projection weights. These key Gaussians are those that contribute significantly to the scene representation or whose projection characteristics may affect the reconstruction quality (for example, they occupy a significant position in the image or may cause blurry rendering or lack of detail), providing targeted targets for subsequent densification processing.

[0077] How to determine the key Gaussian from multiple three-dimensional Gaussian distributions will be further described below.

[0078] Reference Figure 2 , selecting a key Gaussian from multiple three-dimensional Gaussian distributions based on the two-dimensional projection weight, including the following steps 201 to 203.

[0079] Step 201: For each image plane point of each viewing angle, from multiple three-dimensional Gaussian distributions corresponding to the image plane point, select the three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the important Gaussian corresponding to the image plane point.

[0080] Step 202: Obtain the effective projection area of ​​each three-dimensional Gaussian distribution, and select a three-dimensional Gaussian distribution whose effective projection area exceeds a preset area threshold from multiple three-dimensional Gaussian distributions as a fuzzy Gaussian.

[0081] Steps 201 to 202 are described in detail below.

[0082] In some embodiments, in order to identify the three-dimensional Gaussian distribution that has a core contribution to the scene expression, the system analyzes each image plane point (i.e., pixel point in the image) under each given camera perspective. For each image plane point, there are generally multiple three-dimensional Gaussian distributions projected through the rendering pipeline and affecting it. The system compares the two-dimensional projection weights of these three-dimensional Gaussian distributions and selects the one with the largest two-dimensional projection weight, and identifies it as the important Gaussian corresponding to the image plane point. That is, for each image plane point , you can get the Gaussian index , the system selects the three-dimensional Gaussian distribution corresponding to all indices , as the important Gaussian. This process means that for each position in the image, the system finds the 3D Gaussian basis that contributes most to its color.

[0083] Next, in order to further identify those 3D Gaussian distributions that may not be the largest contributor to each pixel, but whose projection range is too large and may cause blurred rendering results or loss of details, the system first obtains each 3D Gaussian distribution Effective projected area Then the effective projection area of ​​each three-dimensional Gaussian distribution is compared with a pre-set area threshold For comparison, if a three-dimensional Gaussian distribution Effective projected area Exceeds this preset area threshold , then the three-dimensional Gaussian distribution is selected and marked as fuzzy Gaussian.

[0084] Among them, the effective projected area can be understood as a metric, which characterizes the size of the area actually effectively covered by a three-dimensional Gaussian distribution on the two-dimensional image plane. For example, it can be counted how many image plane points the three-dimensional Gaussian distribution is identified as an important Gaussian, or the number of pixels covered by its projected two-dimensional Gaussian.

[0085] The following further describes how to obtain the effective projection area of ​​each three-dimensional Gaussian distribution.

[0086] Reference Figure 3 , obtaining the effective projection area of ​​each three-dimensional Gaussian distribution, including the following steps 301 to 303.

[0087] Step 301: Obtain the two-dimensional projection result corresponding to each three-dimensional Gaussian distribution at each viewing angle.

[0088] Step 302: When the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian, the effective projection value corresponding to each viewing angle is obtained.

[0089] Step 303: Accumulate the effective projection values ​​of all viewing angles to obtain the effective projection area of ​​each three-dimensional Gaussian distribution.

[0090] Steps 301 to 303 are described in detail below.

[0091] In some embodiments, in order to calculate the effective projected area of ​​each three-dimensional Gaussian distribution, the system first needs to obtain each three-dimensional Gaussian distribution The corresponding 2D projection results at each available or selected camera view The two-dimensional projection result refers to the two-dimensional representation formed by projecting a three-dimensional Gaussian distribution onto a two-dimensional image plane through a rendering pipeline (usually differentiable). It can reflect the potential impact range and shape of the three-dimensional Gaussian distribution on the image area at a specific viewing angle.

[0092] After obtaining each three-dimensional Gaussian distribution Two-dimensional projection results at each viewing angle Afterwards, the system will determine whether the two-dimensional projection result of a specific three-dimensional Gaussian distribution on a certain image plane point at a certain viewing angle is consistent with the two-dimensional projection result of the important Gaussian corresponding to the image plane point; when the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian (that is, the three-dimensional Gaussian distribution currently being evaluated is the important Gaussian of the pixel), the system will record a "valid projection value" for the current three-dimensional Gaussian distribution at this viewing angle, for example, the count is 1; if they are inconsistent, it will be counted as 0, that is, , , is the indicator function. By performing this judgment on all image plane points within a viewing angle and accumulating them, the effective projection value of the three-dimensional Gaussian distribution at that viewing angle can be obtained. This value reflects the number of pixels covered by it as an important Gaussian at that viewing angle.

[0093] After that, the system will accumulate the effective projection values ​​obtained at all different viewing angles for each 3D Gaussian distribution. That is, the effective projection values ​​corresponding to each viewing angle calculated for each specific 3D Gaussian distribution will be summed up. This cumulative sum across all viewing angles ultimately constitutes the specific 3D Gaussian distribution. Effective projected area As shown in the following formula (2).

[0094] (2)

[0095] This area value summarizes the overall extent to which the Gaussian is the dominant contributor at all observation angles.

[0096] Through the above steps 301 to 303, the overall influence of a three-dimensional Gaussian distribution as a key image contributor in all view angles of the entire observation data set can be comprehensively quantified, so as to provide a reliable quantitative basis for the subsequent accurate identification of fuzzy Gaussians that occupy significant projection areas under multiple view angles and may cause redundancy or blurring, thereby making the selection of key Gaussians more accurate, helping to optimize the density control strategy and improve the efficiency and quality of three-dimensional scene reconstruction.

[0097] Step 203: Obtain key Gaussians based on all important Gaussians and all fuzzy Gaussians.

[0098] Steps 201 to 203 are described in detail below.

[0099] In some embodiments, after completing the identification of important Gaussians and fuzzy Gaussians, the system merges these two identified Gaussian sets to determine the key Gaussians required for subsequent processing (ie, Gaussian splitting and densification).

[0100] Through the above steps 201 to 203, by selecting the three-dimensional Gaussian distribution with the maximum two-dimensional projection weight on each image plane point as the important Gaussian, it is ensured that the Gaussian that contributes most to the main structure and color of the scene is identified, and then by comparing the effective projection area of ​​each three-dimensional Gaussian distribution with the preset area threshold, the fuzzy Gaussian that may cause rendering blur is identified, and these two types of Gaussians are merged to obtain the final key Gaussian set. It is possible to comprehensively identify the Gaussian distributions that have a significant impact on the quality and efficiency of scene reconstruction, including both Gaussians that contribute greatly to scene details and Gaussians that may affect rendering clarity, thereby providing more accurate and targeted input for subsequent intensive operations, avoiding invalid operations on unimportant or sufficiently fine areas, and thus helping to improve the efficiency of the overall three-dimensional scene reconstruction and the quality of the final reconstructed scene.

[0101] Step 103: Perform Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, and perform deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions.

[0102] Step 103 is described in detail below.

[0103] After determining the key Gaussian, the system will perform a Gaussian splitting operation on the selected key Gaussian, that is, splitting a three-dimensional Gaussian distribution identified as key into multiple smaller and more numerous split Gaussian distributions according to preset rules, so as to increase the number and density of Gaussian basis units in the key area, thereby improving the detail expression ability of the scene model. Then, the system will use the depth information carried by all current three-dimensional Gaussian distributions (including the original unsplit three-dimensional Gaussian distribution and the newly generated multiple split Gaussian distributions) to perform depth reinitialization. The depth reinitialization process involves extracting or estimating a depth map from the existing Gaussian distribution, and then reprojecting this depth map back to three-dimensional space according to the camera parameters to generate a new point cloud, and using these new three-dimensional points to optimize or reinitialize the parameters of the Gaussian distribution, thereby generating multiple denser three-dimensional Gaussian distributions that are more dense and can more accurately capture the geometric details of the scene.

[0104] How to perform Gaussian splitting on a key Gaussian is further described below.

[0105] Reference Figure 4 , performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, including the following steps 401 to 404.

[0106] Step 401: Based on the center position of the key Gaussian, obtain the center position of the split Gaussian distribution.

[0107] Step 402: Based on the difference between the opacity of the key Gaussian and the opacity of the key Gaussian, a root opening process is performed to obtain a difference root, and based on the difference between the opacity of the key Gaussian and the opacity of the split Gaussian distribution is obtained.

[0108] Step 403: Based on the difference between twice the opacity of the split Gaussian distribution and the square of the opacity of the split Gaussian distribution, a quadratic inverse process is performed to obtain the split square inverse.

[0109] Step 404: Obtain a covariance matrix of the split Gaussian distribution based on the splitting square inverse, the covariance matrix of the key Gaussian, and the square of the opacity of the key Gaussian.

[0110] Steps 401 to 404 are described in detail below.

[0111] In some embodiments, for each selected key Gaussian , which includes opacity , central location , the covariance matrix in world space . Then split it to generate multiple split Gaussian distributions (such as and ), the center position of the newly generated split Gaussian distribution is determined based on the original center position of the key Gaussian, that is, This means that the initial positioning of the split Gaussian distribution in 3D space will inherit or be closely related to the geometric center of its parent key Gaussian, ensuring the continuity and locality of the splitting operation in space.

[0112] Then, to determine the opacity of the split Gaussian, first calculate the difference between one and the opacity of the key Gaussian itself ; Then, perform root (i.e. square root) processing on this difference result to obtain an intermediate value , which is the difference root; then, the difference between one and the difference root is calculated, and the final difference result is set as the opacity of the split Gaussian distribution, that is, This step derives the opacity parameters of its child split Gaussians from the opacity of the key Gaussian through a series of precise mathematical operations.

[0113] Then, to calculate the covariance matrix of the split Gaussian distribution, the system first obtains a basis based on twice the opacity of the split Gaussian distribution minus the ratio of the square of the opacity of the split Gaussian distribution to the square root of 2, that is, ; Then, the reciprocal quadratic processing is performed on this basis, that is, the negative quadratic power of the basis is calculated (or it can be understood as the reciprocal of the square of the basis), and the result is the split square reciprocal. This specific calculation step provides a key scaling or adjustment factor for the subsequent adjustment of the covariance matrix; then, based on the split square reciprocal, the covariance matrix of the key Gaussian and the square of the opacity of the key Gaussian, the covariance matrix of the split Gaussian distribution is obtained, as shown in the following formula (3).

[0114] (3)

[0115] Through the above steps 401 to 404, the inheritance of the spatial position of the split Gaussian is ensured, and a new opacity is set for the split Gaussian distribution through specific mathematical transformations. Then, based on a covariance matrix calculation method of the original parameters of the key Gaussian and the new opacity of the split Gaussian distribution, the number of three-dimensional Gaussian basis elements can be increased in the identified key areas. By carefully setting the center position, opacity and covariance matrix of the newly generated Gaussian, it is helpful to improve the expression ability of local details while maintaining the continuity of the scene structure, laying the foundation for the subsequent generation of high-quality, high-precision dense three-dimensional Gaussian distribution and the final reconstruction of the scene.

[0116] The following will further describe how to reinitialize based on depth.

[0117] Reference Figure 5, deep reinitialization is performed based on multiple split Gaussian distributions and multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions, including the following steps 501 to 503.

[0118] Step 501: Obtain the Gaussian distribution depth corresponding to the important Gaussian corresponding to each image plane point.

[0119] Step 502: Obtain a depth map based on all Gaussian distribution depths.

[0120] Step 503: Based on the depth map, reproject the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions into a coordinate system to obtain multiple dense three-dimensional Gaussian distributions.

[0121] Steps 501 to 503 are described in detail below.

[0122] In some embodiments, in order to utilize the depth information of the scene to optimize and densify the Gaussian distribution, the system first needs to obtain each image plane point (i.e., image pixels) corresponding to the important Gaussian (i.e., the Gaussian distribution depth of the three-dimensional Gaussian distribution with the largest contribution identified for each image plane point as shown in step 201 above) The depth of the Gaussian distribution refers to the depth value of the center point of the important Gaussian under the corresponding camera perspective, that is, the distance from the center point of the camera along the camera line of sight, that is, the important Gaussian Center This process is equivalent to extracting the most relevant depth information for each pixel from the 3D Gaussian representation of the current scene.

[0123] After obtaining the Gaussian distribution depths corresponding to all (or selected) image plane points, the system will generate one or more depth maps based on all these obtained Gaussian distribution depth values. A depth map is a two-dimensional image in which the value of each pixel represents the depth information of the corresponding point in the scene from the camera's perspective. By collecting all Gaussian distribution depths, a complete scene depth image inferred from the Gaussian representation can be constructed for each perspective, that is, .

[0124] Next, using the generated depth map, all 3D Gaussian distributions in the current scene—including the newly generated multiple split Gaussian distributions and the existing multiple 3D Gaussian distributions (other 3D Gaussian distributions before or after splitting)—are reprojected into a coordinate system. This reprojection process involves reversely projecting the depth information in the depth map back into the 3D world coordinate system to form a set of 3D point clouds. The system then uses these newly generated, denser, and more realistic 3D point clouds to reinitialize or adjust the parameters of the existing 3D Gaussian distributions (i.e., the multiple split Gaussian distributions and the multiple 3D Gaussian distributions), particularly their center positions, to more accurately align them with the scene surface indicated by the depth map. This depth-information-based optimization ultimately results in a larger set of dense 3D Gaussian distributions that are more reasonably distributed and can more accurately capture the scene's geometric details.

[0125] Through the above steps 501 to 503, the depth of the Gaussian distribution is first extracted from the important Gaussian, and then a depth map is constructed based on this depth information. Finally, the depth map is used to reproject all existing three-dimensional Gaussian distributions (including the new Gaussian generated by splitting and the original Gaussian) into the coordinate system to optimize their parameters and generate a dense three-dimensional Gaussian distribution. The scene geometric prior (depth information) extracted from the Gaussian representation is effectively fed back into the optimization process of the Gaussian parameters, so that the Gaussian basis element can fit the real surface structure of the scene more closely, which helps to correct the geometric deviation that may be introduced by the initial sparse point cloud or splitting operation, and can also increase the density and coverage integrity of the Gaussian distribution while maintaining details, thereby providing a more solid and accurate geometric foundation for subsequent high-quality three-dimensional scene reconstruction. The multiple dense three-dimensional Gaussian distributions finally obtained can more accurately characterize the target scene.

[0126] Step 104: Select multiple retained Gaussian distributions from the multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, and reconstruct the scene based on the multiple retained Gaussian distributions to obtain a reconstructed scene that matches the target scene.

[0127] Step 104 is described in detail below.

[0128] In some embodiments, after generating multiple dense three-dimensional Gaussian distributions, in order to obtain an efficient and accurate final scene representation, the system will again select multiple retained Gaussian distributions that are most important or contribute the most to the final scene expression based on the two-dimensional projection weights of each of these dense three-dimensional Gaussian distributions, thereby removing dense three-dimensional Gaussian distributions that contribute less to the scene or are redundant, thereby simplifying the model and reducing storage and rendering overhead. Ultimately, the system uses these carefully selected multiple retained Gaussian distributions to complete the three-dimensional scene reconstruction process, that is, by optimizing the parameters of these retained Gaussian distributions until convergence, it generates a reconstructed scene that is highly matched with the target scene and has a more efficient and compact representation.

[0129] How to select multiple preserved Gaussian distributions will be further described below.

[0130] Reference Figure 6 , selecting multiple retained Gaussian distributions from multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of multiple dense three-dimensional Gaussian distributions, including the following steps 601 to 605.

[0131] Step 601: From a plurality of dense three-dimensional Gaussian distributions corresponding to each image plane point, select a dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as a first retained Gaussian distribution corresponding to the image plane point.

[0132] Step 602: Accumulate the two-dimensional projection weights of each dense three-dimensional Gaussian distribution corresponding to each viewing angle to obtain the Gaussian importance of each dense three-dimensional Gaussian distribution.

[0133] Step 603: Accumulate all Gaussian importances to obtain the total importance, and obtain the sampling probability corresponding to each dense three-dimensional Gaussian distribution based on the ratio of each Gaussian importance to the total importance.

[0134] Step 604: sampling each dense three-dimensional Gaussian distribution based on the sampling probability to obtain a second retained Gaussian distribution.

[0135] Step 605: Obtain a retained Gaussian distribution based on the first retained Gaussian distribution and the second retained Gaussian distribution.

[0136] Steps 601 to 605 are described in detail below.

[0137] In some embodiments, in order to preliminarily screen out the most direct and significant part of the scene visual contribution from the multiple dense three-dimensional Gaussian distributions obtained after densification, the system will perform a (i.e. each ray or each pixel coordinate ) for analysis. At each image plane point , there are usually multiple dense three-dimensional Gaussian distributions that affect its projection. The system will compare the two-dimensional projection weights of these dense three-dimensional Gaussian distributions and select the dense three-dimensional Gaussian distribution with the largest two-dimensional projection weight. It is marked as the first retained Gaussian distribution corresponding to the image plane point, that is, the corresponding index Three-dimensional Gaussian distribution of , ensuring that for each pixel position in the image, its main contributor (i.e., the main source of color information) is preferentially retained.

[0138] In order to more comprehensively evaluate the overall importance of each dense 3D Gaussian distribution to the reconstruction of the entire scene, rather than just its maximum contribution on a single pixel, the system will perform a , the corresponding two-dimensional projection weights at all contributing image plane points under each available camera perspective The sum of the two-dimensional projection weights of a dense three-dimensional Gaussian distribution on all view angles and all relevant pixels is defined as the Gaussian importance of the dense three-dimensional Gaussian distribution. The higher the Gaussian importance value, the greater the contribution of the dense three-dimensional Gaussian distribution to the overall visual presentation of the scene under multiple perspectives.

[0139] After calculating each dense three-dimensional Gaussian distribution Gaussian importance of After that, the system first accumulates all these Gaussian importance values , thus obtaining a global sum of importance , the sum represents the total contribution of all current dense three-dimensional Gaussian distributions to the scene. Then, for each dense three-dimensional Gaussian distribution , the system will divide its own Gaussian importance by the sum of this importance and calculate a ratio This ratio is determined as the dense three-dimensional Gaussian distribution The corresponding sampling probability , which reflects the importance of this Gaussian relative to all other Gaussians and is used in the subsequent random sampling process.

[0140] Afterwards, using each dense three-dimensional Gaussian distribution The sampling probability of , a sampling selection process is performed on these dense three-dimensional Gaussian distributions. This means that each dense three-dimensional Gaussian distribution has a certain probability of being selected and retained, and its probability of being selected is proportional to its own sampling probability. Through this importance-based probabilistic sampling method, the system can select a part from a large number of dense three-dimensional Gaussian distributions to form a second set of retained Gaussian distributions. This method tends to retain those dense three-dimensional Gaussian distributions with higher Gaussian importance values, while also giving Gaussians with lower importance a certain chance of retention to maintain the integrity of the scene representation. For example, according to the sampling probability, the number of Gaussians of the sparse Gaussian model is preset (such as 10,000), or the ratio to the original dense Gaussian (such as 50%), and the sampled Gaussians are retained to complete the sampling, thereby obtaining multiple second retained Gaussian distributions.

[0141] Finally, by integrating the union of the first retained Gaussian distribution and the second retained Gaussian distribution obtained by the two aforementioned screening strategies, we ensure that both the Gaussian that directly contributes the most to each pixel and those that are of high importance overall are retained, thus obtaining a streamlined Gaussian set that can both ensure details and reflect global importance.

[0142] Through the above steps 601 to 605, it is ensured that the Gaussian with the greatest visual contribution to each pixel is directly retained. Then, a probabilistic sampling mechanism based on global importance is introduced to comprehensively evaluate and retain the Gaussians that contribute significantly to the overall scene. This dual-guarantee selection mechanism can effectively reduce the number of three-dimensional Gaussian distributions, reduce model redundancy and computational complexity, and maximize the retention of information that is critical to the quality of scene reconstruction. The resulting retained Gaussian distribution can achieve high-quality three-dimensional scene reconstruction with a more compact representation, thereby improving reconstruction efficiency and the practicality of the final model.

[0143] Reference Figure 7 , is a schematic diagram of a Gaussian densification and re-sparseness process provided in an embodiment of the present application. Figure 7The figure illustrates the core process of an efficient 3D Gaussian-based scene reconstruction method provided by an embodiment of the present invention. This method uses a sparse point cloud (or multiple 3D Gaussian distributions initialized from it) as initial input. Subsequently, a densification process (i.e., selecting key Gaussians based on the 2D projection weights of the 3D Gaussian distribution, performing Gaussian splitting on the key Gaussians to obtain multiple split Gaussian distributions, and performing deep reinitialization based on the multiple split Gaussian distributions and the multiple 3D Gaussian distributions) is performed to obtain the "dense Gaussians" shown in the accompanying figure. Next, a "simplification" operation is performed on these "dense Gaussians" (i.e., selecting multiple retained Gaussian distributions from multiple dense 3D Gaussian distributions based on the 2D projection weights of the multiple dense 3D Gaussian distributions), ultimately resulting in an optimized number of "sparse Gaussians" for scene reconstruction. This achieves efficient and high-quality 3D scene reconstruction, addressing the low reconstruction efficiency issue of the existing technology due to the fixed density control strategy.

[0144] Reference Figure 8 Before selecting key Gaussians from multiple three-dimensional Gaussian distributions based on two-dimensional projection weights, the three-dimensional Gaussian scene reconstruction method also includes the following steps 801; and before selecting multiple retained Gaussian distributions from multiple dense three-dimensional Gaussian distributions based on two-dimensional projection weights of multiple dense three-dimensional Gaussian distributions, the three-dimensional Gaussian scene reconstruction method also includes the following steps 802.

[0145] Step 801: During each iteration, image rendering is performed based on multiple three-dimensional Gaussian distributions corresponding to the current number of iterations to obtain a rendered image, and based on the difference between the rendered image and the real image of the target scene, the opacity, center position and covariance matrix of the multiple three-dimensional Gaussian distributions are updated until the current number of iterations reaches the first preset number of iterations.

[0146] Step 802: During each iteration, image rendering is performed based on multiple dense three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a dense rendered image, and based on the difference between the dense rendered image and the real image, the opacity, center position and covariance matrix of the multiple dense three-dimensional Gaussian distributions are updated until the current iteration number reaches the second preset iteration number.

[0147] Steps 801 to 802 are described in detail below.

[0148] In some embodiments, when performing a two-dimensional projection weighting Before selecting the key Gaussian from multiple three-dimensional Gaussian distributions, the initial multiple three-dimensional Gaussian distributions are also obtained. Perform multiple iterative optimizations. Specifically, during each iteration, the system performs image rendering based on multiple three-dimensional Gaussian distributions corresponding to the current iteration number n, that is, uses these three-dimensional Gaussian distributions to generate one or more rendered images from a specific virtual camera perspective through a differentiable rendering pipeline. Subsequently, the system calculates the difference between the rendered image and the real image of the target scene at the corresponding perspective (usually the input original photo or video frame) (which is quantified by a commonly used loss function) to measure the degree of consistency between the current three-dimensional Gaussian distribution representation and the real scene. Based on this difference, the system adjusts and updates the parameters of multiple three-dimensional Gaussian distributions, namely their respective opacity, center position, and covariance matrix, through backpropagation and optimization algorithms (such as gradient descent) in order to reduce the difference. This iterative rendering, comparison, and update process will continue until the current number of iterations n reaches a preset first preset number of iterations (such as 100 or 200), thereby obtaining a set of multiple three-dimensional Gaussian distributions that have been preliminarily optimized, providing a more accurate basis for the subsequent selection of key Gaussians; and then a densification process will be performed again, that is, after every 100 or 200 iterative optimization processes, a densification process will be performed again.

[0149] Before selecting multiple preserved Gaussian distributions from multiple dense 3D Gaussian distributions based on their 2D projection weights, the 3D Gaussian scene reconstruction method also performs another iterative optimization on the multiple dense 3D Gaussian distributions obtained after Gaussian splitting and depth reinitialization. Similar to step 801, during each iteration, the system renders an image based on the multiple dense 3D Gaussian distributions corresponding to the current iteration number n to generate a dense rendered image. The system then calculates the difference between the dense rendered image and the true image. Based on this difference, the system also updates and optimizes the opacity, center position, and covariance matrix of the multiple dense 3D Gaussian distributions. This iterative optimization process continues until the current iteration number n reaches a predetermined second preset iteration number (e.g., 3003 or 8000). This step aims to further refine these densified Gaussian distributions so that they can fit the target scene more accurately and provide high-quality input for the subsequent selection of retained Gaussians for final scene reconstruction; then the densification process is performed again, that is, after every 3000 or 8000 iterative optimization processes, a simplification process is performed.

[0150] Through the above steps 801 and 802, the three-dimensional Gaussian scene reconstruction method effectively adjusts and optimizes the parameters of the Gaussian distribution in the key control stage (i.e., before selecting the key Gaussian and before selecting the retained Gaussian), ensuring that these dense Gaussians have represented the scene details as accurately as possible before the final simplification and selection of the retained Gaussian, so that the Gaussian distribution can continuously and phasedly evolve to a better state in the entire reconstruction process, significantly improving the quality of the Gaussian representation in the intermediate stage, and ultimately improving the accuracy, completeness and visual fidelity of the overall reconstructed scene.

[0151] Reference Figure 9 and Figure 10 , is a schematic diagram of the acquisition of image data of a three-dimensional scene and a schematic diagram of the reconstruction of a three-dimensional scene provided by an embodiment of the present application. Figure 9 and Figure 10 As shown in , the key stages and final effects of the three-dimensional Gaussian scene reconstruction method disclosed in the embodiment of the present invention are demonstrated. Figure 9 The input of the reconstruction process is illustrated, showing multiple camera perspectives distributed around the target scene (represented as a preliminary sparse point cloud). These perspectives correspond to the sources for obtaining multi-view images of the target scene and provide the basis for subsequent generation of sparse point clouds from motion structure (SfM) and initialization of multiple 3D Gaussian distributions. Figure 10 The three-dimensional Gaussian scene reconstruction method provided in this application is applied to demonstrate that a high-quality reconstructed scene that matches the target scene is finally obtained. The reconstructed scene has rich details and realistic visual effects, which intuitively demonstrates the beneficial effects of the three-dimensional Gaussian scene reconstruction method provided in this application in improving reconstruction efficiency and scene quality.

[0152] The three-dimensional Gaussian scene reconstruction method, device, electronic device and storage medium proposed in the embodiments of the present application include: first, obtaining multiple three-dimensional Gaussian distributions of the target scene; then, obtaining the two-dimensional projection weight of each three-dimensional Gaussian distribution, and for each image plane point of each perspective, selecting the three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight from the multiple three-dimensional Gaussian distributions corresponding to the image plane point as the important Gaussian corresponding to the image plane point, obtaining the two-dimensional projection result corresponding to each three-dimensional Gaussian distribution at each perspective, and when the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian, the effective projection corresponding to each perspective is obtained. value, accumulate the effective projection values ​​of all viewing angles to obtain the effective projection area of ​​each three-dimensional Gaussian distribution, and select the three-dimensional Gaussian distribution whose effective projection area exceeds the preset area threshold from multiple three-dimensional Gaussian distributions as the fuzzy Gaussian, and obtain the key Gaussian based on all important Gaussians and all fuzzy Gaussians; next, based on the center position of the key Gaussian, obtain the center position of the split Gaussian distribution, based on the difference between the opacity of one and the key Gaussian, perform root opening processing to obtain the difference root, and based on the difference between one and the difference root, obtain the opacity of the split Gaussian distribution, based on twice the opacity of the split Gaussian distribution and the square of the opacity of the split Gaussian distribution. The difference is then processed by the quadratic inverse to obtain the split square inverse. Based on the split square inverse, the covariance matrix of the key Gaussian and the square of the opacity of the key Gaussian, the covariance matrix of the split Gaussian distribution is obtained, and the Gaussian distribution depth corresponding to the important Gaussian corresponding to each image plane point is obtained. A depth map is obtained based on the depth of all Gaussian distributions. Based on the depth map, multiple split Gaussian distributions and multiple three-dimensional Gaussian distributions are reprojected into the coordinate system to obtain multiple dense three-dimensional Gaussian distributions; finally, among the multiple dense three-dimensional Gaussian distributions corresponding to each image plane point, the dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight is selected as the image plane point. Corresponding to the first retained Gaussian distribution, the two-dimensional projection weights corresponding to each dense three-dimensional Gaussian distribution at each perspective are accumulated to obtain the Gaussian importance of each dense three-dimensional Gaussian distribution, all Gaussian importances are accumulated to obtain the sum of the importances, and based on the ratio of each Gaussian importance to the sum of the importances, the sampling probability corresponding to each dense three-dimensional Gaussian distribution is obtained, and each dense three-dimensional Gaussian distribution is sampled and selected based on the sampling probability to obtain the second retained Gaussian distribution, and a retained Gaussian distribution is obtained based on the first retained Gaussian distribution and the second retained Gaussian distribution, and a scene is reconstructed based on multiple retained Gaussian distributions to obtain a reconstructed scene matching the target scene.

[0153] In the embodiment of the present application, the key Gaussians that contribute significantly to the scene expression or may cause blur are first intelligently selected based on the two-dimensional projection weights of the three-dimensional Gaussian distribution. Then, these key Gaussians are subjected to targeted Gaussian splitting and reinitialized in combination with depth information to achieve effective densification. Finally, important retained Gaussians are selected based on the two-dimensional projection weights of the dense three-dimensional Gaussian distribution for simplification and final scene reconstruction. The density adjustment strategy driven by dynamic and data importance is used to effectively overcome the limitation of using a pre-set fixed density control strategy to perform appropriate density control when processing complex and numerous three-dimensional Gaussian distributions. This makes it possible to more accurately increase the Gaussian density in areas where details are required, while removing unnecessary Gaussians in redundant areas, avoiding computational waste and representation redundancy caused by blind adjustment, thereby significantly improving the overall efficiency of three-dimensional scene reconstruction and the compactness and quality of the final reconstructed scene. In addition, by selecting the three-dimensional Gaussian distribution with the largest two-dimensional projection weight at each image plane point as the important Gaussian, it is ensured that the Gaussian that contributes most to the main structure and color of the scene is identified, and then by comparing the effective projection area of ​​each three-dimensional Gaussian distribution with the predicted An area threshold is set to identify fuzzy Gaussians that may cause rendering blur. These two types of Gaussians are then merged to obtain a final set of key Gaussians. This method comprehensively identifies Gaussian distributions that have a significant impact on scene reconstruction quality and efficiency, including both those that contribute significantly to scene detail and those that may affect rendering clarity. This provides more accurate and targeted input for subsequent densification operations, avoids ineffective operations on unimportant or sufficiently fine areas, and thus helps improve the efficiency of overall 3D scene reconstruction and the quality of the final reconstructed scene. Furthermore, the method ensures the spatial inheritance of the split Gaussian distributions and sets a new opacity for the split Gaussian distribution through a specific mathematical transformation. Based on a covariance matrix calculation method based on the original parameters of the key Gaussian and the new opacity of the split Gaussian distribution, the method increases the number of 3D Gaussian basis elements in the identified key areas. By carefully setting the center position, opacity, and covariance matrix of the newly generated Gaussian, the method helps improve the representation of local details while maintaining the continuity of the scene structure, laying the foundation for the subsequent generation of high-quality, high-precision dense 3D Gaussian distributions and the final reconstructed scene.In addition, the Gaussian distribution depth is first extracted from the important Gaussian, and then a depth map is constructed based on this depth information. Finally, the depth map is used to reproject all existing three-dimensional Gaussian distributions (including the new Gaussian generated by splitting and the original Gaussian) into the coordinate system to optimize their parameters and generate a dense three-dimensional Gaussian distribution. The scene geometric prior (depth information) extracted from the Gaussian representation is effectively fed back into the optimization process of the Gaussian parameters, so that the Gaussian basis element can fit the real surface structure of the scene more closely, which helps to correct the geometric deviation that may be introduced by the initial sparse point cloud or splitting operation, and can also increase the density and coverage completeness of the Gaussian distribution while maintaining details, thereby providing high-quality three-dimensional scenes for subsequent use. This reconstruction provides a more solid and precise geometric foundation, resulting in multiple dense 3D Gaussian distributions that more accurately represent the target scene. Furthermore, the Gaussian distributions that contribute most visually to each pixel are retained. A probabilistic sampling mechanism based on global importance is then introduced to comprehensively evaluate and retain Gaussians that contribute significantly to the overall scene. This dual-guarantee selection mechanism effectively reduces the number of 3D Gaussian distributions, lowering model redundancy and computational complexity while maximizing the preservation of information critical to scene reconstruction quality. The resulting retained Gaussian distributions enable high-quality 3D scene reconstruction with a more compact representation, improving reconstruction efficiency and the practicality of the final model.

[0154] The present application also provides a three-dimensional Gaussian scene reconstruction device, which can implement the above-mentioned three-dimensional Gaussian scene reconstruction method, referring to Figure 11 , the apparatus 1100 comprises:

[0155] A Gaussian acquisition module 1110 is configured to acquire a three-dimensional Gaussian distribution of a target scene, where the three-dimensional Gaussian distribution includes multiple three-dimensional Gaussian distributions.

[0156] a key Gaussian screening module 1120 , configured to obtain a two-dimensional projection weight of each three-dimensional Gaussian distribution and select a key Gaussian from multiple three-dimensional Gaussian distributions based on the two-dimensional projection weight;

[0157] A Gaussian densification module 1130 is configured to perform Gaussian splitting on a key Gaussian to obtain multiple split Gaussian distributions, and perform deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain a dense three-dimensional Gaussian distribution, where the dense three-dimensional Gaussian distribution includes multiple dense three-dimensional Gaussian distributions.

[0158] The Gaussian sparsification module 1140 is used to select multiple retained Gaussian distributions from multiple dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the multiple dense three-dimensional Gaussian distributions, and reconstruct the scene based on the multiple retained Gaussian distributions to obtain a reconstructed scene that matches the target scene.

[0159] In some embodiments, the key Gaussian screening module 1120 is further configured to:

[0160] For each image plane point of each viewing angle, from the multiple three-dimensional Gaussian distributions corresponding to the image plane point, the three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight is selected as the important Gaussian corresponding to the image plane point;

[0161] Obtaining the effective projection area of ​​each three-dimensional Gaussian distribution, and selecting a three-dimensional Gaussian distribution whose effective projection area exceeds a preset area threshold from the multiple three-dimensional Gaussian distributions as a fuzzy Gaussian;

[0162] Get the key Gaussian based on all important Gaussians and all fuzzy Gaussians.

[0163] In some embodiments, the key Gaussian screening module 1120 is further configured to:

[0164] Get the two-dimensional projection result corresponding to each three-dimensional Gaussian distribution at each viewing angle;

[0165] When the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian, the effective projection value corresponding to each viewing angle is obtained;

[0166] The effective projection values ​​of all viewing angles are accumulated to obtain the effective projection area of ​​each three-dimensional Gaussian distribution.

[0167] In some embodiments, the Gaussian densification module 1130 is further configured to:

[0168] Based on the center position of the key Gaussian, the center position of the split Gaussian distribution is obtained;

[0169] Based on the difference between the opacity of one and the key Gaussian, a root opening process is performed to obtain a difference root, and based on the difference between one and the difference root, the opacity of the split Gaussian distribution is obtained;

[0170] Based on the difference between the opacity of twice the split Gaussian distribution and the square of the opacity of the split Gaussian distribution, the reciprocal of the split square is obtained by performing a quadratic reciprocal process;

[0171] The covariance matrix of the split Gaussian distribution is obtained based on the splitting square inverse, the covariance matrix of the key Gaussian, and the square of the opacity of the key Gaussian.

[0172] In some embodiments, the Gaussian densification module 1130 is further configured to:

[0173] Get the Gaussian distribution depth corresponding to the important Gaussian corresponding to each image plane point;

[0174] Get a depth map based on all Gaussian distribution depths;

[0175] Based on the depth map, multiple split Gaussian distributions and multiple three-dimensional Gaussian distributions are reprojected into a coordinate system to obtain multiple dense three-dimensional Gaussian distributions.

[0176] In some embodiments, the Gaussian sparsification module 1140 is further configured to:

[0177] Among multiple dense three-dimensional Gaussian distributions corresponding to each image plane point, the dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight is selected as the first retained Gaussian distribution corresponding to the image plane point;

[0178] The two-dimensional projection weights corresponding to each dense three-dimensional Gaussian distribution at each viewing angle are accumulated to obtain the Gaussian importance of each dense three-dimensional Gaussian distribution;

[0179] Accumulate all Gaussian importances to get the sum of importances, and based on the ratio of each Gaussian importance to the sum of importances, get the sampling probability corresponding to each dense three-dimensional Gaussian distribution;

[0180] Sampling each dense three-dimensional Gaussian distribution based on the sampling probability to obtain a second retained Gaussian distribution;

[0181] A preserved Gaussian distribution is obtained based on the first preserved Gaussian distribution and the second preserved Gaussian distribution.

[0182] In some embodiments, the 3D Gaussian scene reconstruction apparatus 1100 further includes a Gaussian optimization module 1150, which is configured to:

[0183] During each iteration, performing image rendering based on multiple three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a rendered image, and updating the opacity, center position, and covariance matrix of the multiple three-dimensional Gaussian distributions based on the difference between the rendered image and the real image of the target scene, until the current iteration number reaches a first preset iteration number;

[0184] During each iteration, image rendering is performed based on multiple dense three-dimensional Gaussian distributions corresponding to the current number of iterations to obtain a dense rendered image, and based on the difference between the dense rendered image and the real image, the opacity, center position and covariance matrix of the multiple dense three-dimensional Gaussian distributions are updated until the current number of iterations reaches the second preset number of iterations.

[0185] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, the specific implementation of the three-dimensional Gaussian scene reconstruction device is basically the same as the specific implementation of the above-mentioned three-dimensional Gaussian scene reconstruction method, and will not be repeated here.

[0186] In the embodiment of the present application, a three-dimensional Gaussian scene reconstruction device first intelligently selects key Gaussians that contribute significantly to the scene expression or may cause blurring based on the two-dimensional projection weights of the three-dimensional Gaussian distribution, then performs targeted Gaussian splitting on these key Gaussians and reinitializes them in combination with depth information to achieve effective densification, and finally selects important retained Gaussians based on the two-dimensional projection weights of the dense three-dimensional Gaussian distribution for simplification and final scene reconstruction. By using a dynamic and data-importance-driven density adjustment strategy, the limitation of using a pre-set fixed density control strategy to process complex and numerous three-dimensional Gaussian distributions is effectively overcome, so that the Gaussian density can be increased more accurately in areas where details are required, while removing unnecessary Gaussians in redundant areas, avoiding the computational waste and representation redundancy caused by blind adjustment, thereby significantly improving the overall efficiency of three-dimensional scene reconstruction and the compactness and quality of the final reconstructed scene; and by selecting the three-dimensional Gaussian distribution with the largest two-dimensional projection weight at each image plane point as the important Gaussian, it is ensured that the Gaussian that contributes most to the main structure and color of the scene is identified, and then by comparing the effective density of each three-dimensional Gaussian distribution. The effective projection area and the preset area threshold are combined to identify fuzzy Gaussians that may cause rendering blur, and these two types of Gaussians are merged to obtain the final key Gaussian set. This can comprehensively identify Gaussian distributions that have a significant impact on the quality and efficiency of scene reconstruction, including both Gaussians that contribute significantly to scene details and those that may affect rendering clarity. This provides more accurate and targeted input for subsequent densification operations, avoids invalid operations on unimportant or sufficiently fine areas, and thus helps improve the efficiency of overall 3D scene reconstruction and the quality of the final reconstructed scene. In addition, the spatial position inheritance of the split Gaussian is ensured, and a new opacity is set for the split Gaussian distribution through a specific mathematical transformation. Based on a covariance matrix calculation method of the original parameters of the key Gaussian and the new opacity of the split Gaussian distribution, the number of 3D Gaussian basis elements in the identified key areas can be increased. By carefully setting the center position, opacity, and covariance matrix of the newly generated Gaussian, it helps to improve the expression of local details while maintaining the continuity of the scene structure, laying the foundation for the subsequent generation of high-quality and high-precision dense 3D Gaussian distribution and the final reconstructed scene.In addition, the Gaussian distribution depth is first extracted from the important Gaussian, and then a depth map is constructed based on this depth information. Finally, the depth map is used to reproject all existing three-dimensional Gaussian distributions (including the new Gaussian generated by splitting and the original Gaussian) into the coordinate system to optimize their parameters and generate a dense three-dimensional Gaussian distribution. The scene geometric prior (depth information) extracted from the Gaussian representation is effectively fed back into the optimization process of the Gaussian parameters, so that the Gaussian basis element can fit the real surface structure of the scene more closely, which helps to correct the geometric deviation that may be introduced by the initial sparse point cloud or splitting operation, and can also increase the density and coverage completeness of the Gaussian distribution while maintaining details, thereby providing high-quality three-dimensional scenes for subsequent use. This reconstruction provides a more solid and precise geometric foundation, resulting in multiple dense 3D Gaussian distributions that more accurately represent the target scene. Furthermore, the Gaussian distributions that contribute most visually to each pixel are retained. A probabilistic sampling mechanism based on global importance is then introduced to comprehensively evaluate and retain Gaussians that contribute significantly to the overall scene. This dual-guarantee selection mechanism effectively reduces the number of 3D Gaussian distributions, lowering model redundancy and computational complexity while maximizing the preservation of information critical to scene reconstruction quality. The resulting retained Gaussian distributions enable high-quality 3D scene reconstruction with a more compact representation, improving reconstruction efficiency and the practicality of the final model.

[0187] An embodiment of the present application further provides an electronic device, including:

[0188] at least one memory;

[0189] at least one processor;

[0190] at least one program;

[0191] The program is stored in the memory, and the processor executes at least one of the programs to implement the three-dimensional Gaussian scene reconstruction method described above. The electronic device can be any smart terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0192] See also Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0193] The processor 1201 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0194] The memory 1202 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1202 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called by the processor 1201 to execute the three-dimensional Gaussian scene reconstruction method of the embodiments of this application;

[0195] Input / output interface 1203, used to implement information input and output;

[0196] Communication interface 1204, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0197] Bus 1205 , which transmits information between various components of the device (e.g., processor 1201 , memory 1202 , input / output interface 1203 , and communication interface 1204 );

[0198] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0199] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned three-dimensional Gaussian scene reconstruction method is implemented.

[0200] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0201] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0202] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0204] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0205] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0206] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0208] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0211] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A three-dimensional Gaussian scene reconstruction method, characterized in that: The method comprises: Obtain multiple three-dimensional Gaussian distributions of the target scene; Obtaining a two-dimensional projection weight of each of the three-dimensional Gaussian distributions, and selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weight; Performing Gaussian splitting on the key Gaussian to obtain multiple split Gaussian distributions, and performing deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions; Selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions, and performing scene reconstruction based on the plurality of retained Gaussian distributions to obtain a reconstructed scene matching the target scene; The three-dimensional Gaussian distribution includes at least a center position, opacity, and a covariance matrix. The Gaussian splitting is performed on the key Gaussian to obtain multiple split Gaussian distributions, including: Based on the center position of the key Gaussian, the center position of the split Gaussian distribution is obtained; Based on the difference between the opacity of the key Gaussian and the opacity of the key Gaussian, a root opening process is performed to obtain a difference root, and based on the difference between the opacity of the key Gaussian and the opacity of the split Gaussian distribution is obtained; Based on the difference between twice the opacity of the split Gaussian distribution and the square of the opacity of the split Gaussian distribution, a quadratic inverse process is performed to obtain a split square inverse; Obtaining a covariance matrix of the split Gaussian distribution based on the splitting square inverse, the covariance matrix of the key Gaussian, and the square of the opacity of the key Gaussian; The deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions includes: Get the Gaussian distribution depth corresponding to the important Gaussian corresponding to each image plane point; Obtaining a depth map based on all of the Gaussian distribution depths; Based on the depth map, reprojecting the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions into a coordinate system to obtain a plurality of the dense three-dimensional Gaussian distributions; The selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions comprises: Among the multiple dense three-dimensional Gaussian distributions corresponding to each image plane point, selecting the dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the first retained Gaussian distribution corresponding to the image plane point; Accumulating the two-dimensional projection weights corresponding to each of the dense three-dimensional Gaussian distributions at each viewing angle to obtain the Gaussian importance of each of the dense three-dimensional Gaussian distributions; Accumulating all the Gaussian importances to obtain a sum of importances, and obtaining a sampling probability corresponding to each of the dense three-dimensional Gaussian distributions based on a ratio of each Gaussian importance to the sum of importances; Sampling each of the dense three-dimensional Gaussian distributions based on the sampling probability to obtain a second retained Gaussian distribution; The retained Gaussian distribution is obtained based on the first retained Gaussian distribution and the second retained Gaussian distribution.

2. The three-dimensional Gaussian scene reconstruction method according to claim 1, characterized in that: The selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weights includes: For each image plane point of each viewing angle, from multiple three-dimensional Gaussian distributions corresponding to the image plane point, select the three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the important Gaussian corresponding to the image plane point; Obtaining an effective projected area of ​​each of the three-dimensional Gaussian distributions, and selecting, from the multiple three-dimensional Gaussian distributions, the three-dimensional Gaussian distribution whose effective projected area exceeds a preset area threshold as a fuzzy Gaussian; The key Gaussian is obtained based on all the important Gaussians and all the fuzzy Gaussians.

3. The three-dimensional Gaussian scene reconstruction method according to claim 2, characterized in that: The obtaining of the effective projection area of ​​each three-dimensional Gaussian distribution includes: Obtaining a two-dimensional projection result corresponding to each viewing angle of each three-dimensional Gaussian distribution; When the two-dimensional projection result of the three-dimensional Gaussian distribution is consistent with the two-dimensional projection result corresponding to the important Gaussian, obtaining a valid projection value corresponding to each of the viewing angles; The effective projection values ​​of all the viewing angles are accumulated to obtain the effective projection area of ​​each of the three-dimensional Gaussian distributions.

4. The three-dimensional Gaussian scene reconstruction method according to claim 1, characterized in that: Before selecting a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weights, the method further includes: During each iteration, image rendering is performed based on the multiple three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a rendered image, and the opacity, center position, and covariance matrix of the multiple three-dimensional Gaussian distributions are updated based on the difference between the rendered image and the real image of the target scene, until the current iteration number reaches a first preset iteration number; Before selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions, the method further includes: During each iteration, image rendering is performed based on the multiple dense three-dimensional Gaussian distributions corresponding to the current iteration number to obtain a dense rendered image, and based on the difference between the dense rendered image and the real image, the opacity, center position and covariance matrix of the multiple dense three-dimensional Gaussian distributions are updated until the current iteration number reaches a second preset iteration number.

5. A three-dimensional Gaussian scene reconstruction device, characterized in that: The device comprises: A Gaussian acquisition module, configured to acquire a three-dimensional Gaussian distribution of a target scene, wherein the three-dimensional Gaussian distribution includes multiple three-dimensional Gaussian distributions; a key Gaussian screening module, configured to obtain a two-dimensional projection weight of each of the three-dimensional Gaussian distributions, and select a key Gaussian from the plurality of three-dimensional Gaussian distributions based on the two-dimensional projection weight; a Gaussian densification module, configured to perform Gaussian splitting on the key Gaussian to obtain a plurality of split Gaussian distributions, and perform deep reinitialization based on the plurality of split Gaussian distributions and the plurality of three-dimensional Gaussian distributions to obtain a dense three-dimensional Gaussian distribution, wherein the dense three-dimensional Gaussian distribution includes a plurality of dense three-dimensional Gaussian distributions; a Gaussian sparsification module, configured to select a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions, and perform scene reconstruction based on the plurality of retained Gaussian distributions to obtain a reconstructed scene matching the target scene; The three-dimensional Gaussian distribution includes at least a center position, opacity, and a covariance matrix. The Gaussian splitting is performed on the key Gaussian to obtain multiple split Gaussian distributions, including: Based on the center position of the key Gaussian, the center position of the split Gaussian distribution is obtained; Based on the difference between the opacity of the key Gaussian and the opacity of the key Gaussian, a root opening process is performed to obtain a difference root, and based on the difference between the opacity of the key Gaussian and the opacity of the split Gaussian distribution is obtained; Based on the difference between twice the opacity of the split Gaussian distribution and the square of the opacity of the split Gaussian distribution, a quadratic inverse process is performed to obtain a split square inverse; Obtaining a covariance matrix of the split Gaussian distribution based on the splitting square inverse, the covariance matrix of the key Gaussian, and the square of the opacity of the key Gaussian; The deep reinitialization based on the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions to obtain multiple dense three-dimensional Gaussian distributions includes: Get the Gaussian distribution depth corresponding to the important Gaussian corresponding to each image plane point; Obtaining a depth map based on all of the Gaussian distribution depths; Based on the depth map, reprojecting the multiple split Gaussian distributions and the multiple three-dimensional Gaussian distributions into a coordinate system to obtain a plurality of the dense three-dimensional Gaussian distributions; The selecting a plurality of retained Gaussian distributions from the plurality of dense three-dimensional Gaussian distributions based on the two-dimensional projection weights of the plurality of dense three-dimensional Gaussian distributions comprises: Among the multiple dense three-dimensional Gaussian distributions corresponding to each image plane point, selecting the dense three-dimensional Gaussian distribution corresponding to the largest two-dimensional projection weight as the first retained Gaussian distribution corresponding to the image plane point; Accumulating the two-dimensional projection weights corresponding to each of the dense three-dimensional Gaussian distributions at each viewing angle to obtain the Gaussian importance of each of the dense three-dimensional Gaussian distributions; Accumulating all the Gaussian importances to obtain a sum of importances, and obtaining a sampling probability corresponding to each of the dense three-dimensional Gaussian distributions based on a ratio of each Gaussian importance to the sum of importances; Sampling each of the dense three-dimensional Gaussian distributions based on the sampling probability to obtain a second retained Gaussian distribution; The retained Gaussian distribution is obtained based on the first retained Gaussian distribution and the second retained Gaussian distribution.

6. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the three-dimensional Gaussian scene reconstruction method according to any one of claims 1 to 4 when executing the computer program.

7. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional Gaussian scene reconstruction method according to any one of claims 1 to 4 is implemented.

Images